Moving LoopBack 4 Example Project to the Monorepo

We often face a dilemma around example projects. They are useful learning resources that demonstrate individual features in a runnable application while making it easy to tweak and play with the code. But on the other hand, they can quickly become a maintenance burden. As the framework evolves, significant ongoing efforts are needed to keep these examples up to date with the latest APIs, conventions, and best practices.

Because our CLI tooling makes it so easy to quickly create a new project, we could not resist the temptation and ended up with about 18 example projects. To update them, one has to open 18 pull requests - that’s a lot of tedious work!

When work on LoopBack 4 (a.k.a Next) started, maintenance of example projects was off the radar. There was too much more important work in other areas, from figuring out Lerna monorepo with TypeScript builds, to finding the right design and APIs that will enable us to address pain points of LoopBack 3.x codebase.

Recently, we started to look into some cleanup tasks. For example, when we looked at dropping support for Node.js 6.x (see issue #611) we realized that with 8 example projects living in their own GitHub repositories, many changes that are easy to apply in the main monorepo are turning into long and tedious tasks.

Despite a decision we made at the beginning to keep everything in a single repository, we were now asking ourselves why example projects are living outside of our monorepo. The main reason was user experience, since we did not want our users to git clone the entire monorepo when they are interested only in a single example project! After some consideration, we fortunately found an easy solution: a CLI tool that downloads only files of the desired example project.

And so lb4 example was born.

$ lb4 example
? What example would you like to clone? (Use arrow keys)
❯ getting-started: An application and tutorial on how to build with LoopBack 4.
  hello-world: A simple hello-world Application using LoopBack 4
  log-extension: An example extension project for LoopBack 4
  rpc-server: A basic RPC server using a made-up protocol.

After you select the example to download, the tool makes an HTTPS request to download a tarball containing the latest code from our monorepo and extract only the files of the selected example project.

$ lb4 example
? What example would you like to clone? getting-started

The example was cloned to loopback4-example-getting-started.

Under the Hood

We use GitHub’s Get archive link API to download an archive containing the current master branch. It then extracts files from the relevant example package directory and places them in a newly created local directory, renaming file entries from packages/example-{name}/{path} to loopback4-example-{name}/{path} along the way. As a nice side effect, users don’t need git installed in order to obtain our example projects.

GitHub offers two archive formats: ZIP and tar+gzip, where the ZIP format is prominently offered on the website. Because I was not aware of the Archive API, my first implementation was downloading a ZIP archive. We were not very happy about this solution though.

Since yauzl, a popular implementation of zip/unzip, does not provide streaming mode, we had to save the downloaded archive to a temp file. Initially we thought this is a limitation of yauzl, but further examination revealed a limitation of the ZIP format. Quoting from yauzl’s No Streaming Unzip API:

Due to the design of the .zip file format, it’s impossible to interpret a .zip file from start to finish (such as from a readable stream) without sacrificing correctness. The Central Directory, which is the authority on the contents of the .zip file, is at the end of a .zip file, not the beginning. A streaming API would need to either buffer the entire .zip file to get to the Central Directory before interpreting anything (defeating the purpose of a streaming interface), or rely on the Local File Headers which are interspersed through the .zip file. However, the Local File Headers are explicitly denounced in the spec as being unreliable copies of the Central Directory, so trusting them would be a violation of the spec.

yauzl’s API was also too low-level. See how involved the ZIP version of extractExample() is:

packages/cli/example/clone-example.js@0359a262.

The module extract-zip provides a higher-level API that’s easier to use, but unfortunately it’s designed in a way that makes our task of filtering and renaming files impossible to implement.

After Raymond pointed out that GitHub offers tar+gzip archives too, I rewrote my code to handle tarballs instead of ZIP archives and the result is so much nicer! No temp files to deal with, just few lines of Stream piping:

return new Promise((resolve, reject) => {
  request(GITHUB_ARCHIVE_URL)
    .pipe(gunzip())
    .pipe(untar(outDir, exampleName))
    .on('error', reject)
    .on('finish', () => resolve(outDir));
});

The implementation of untar() is still more complex than I would like, because of the way how tar-fs applies “map” and “filter” operators, but at least that complexity does not leak out of our untar() method.

You can find the full source code of downloading and extracting example files here:

• packages/cli/example/clone-example.js.

Closing Thoughts

While working on the examples, I made few more related changes.

The doc page Examples and tutorials now contains an updated list of examples we are maintaining, together with brief instructions showing users how to obtain individual examples.

We have deprecated example projects that are not useful anymore:

• loopback4-extension-typeorm
• loopback-next-hello-world
• loopback4-extension-starter

The example project “loopback-next-example” is showcasing a monorepo with multiple microservices. We feel it does not belong to our monorepo. It’s a monorepo on its own and it will eventually depend on external tools like microservices orchestrator that we don’t want to bring into our main monorepo. I renamed the project to better tell its purpose:

loopback4-example-microservices.

Please note the codebase is rather outdated and doesn’t even compile against the latest alpha versions of LoopBack4. This is something we need to fix soon (see loopback4-example-microservices#76).