Context Propagation in LoopBack

LoopBack developers often need to propagate information (aka “context”) from the HTTP layer all the way down to data source connectors. We hear this regularly in the LoopBack discussion forums and GitHub issues.

Consider the following examples:

• When querying a database for a list of “todo” items, return only the items owned by the currently logged-in user.
• When making a REST or SOAP request to a backend service, include a transaction/correlation ID from the incoming HTTP request headers in the outgoing HTTP request headers, so that these two requests can be linked together by logging and tracing tools.
• When formatting an error message, translate it to the language understood by the user, as indicated by the HTTP request header “Accept-Language”.

These examples share a common pattern: the app needs to access information in the layer handling incoming HTTP requests, but it is not included in the arguments (parameters) of remote method APIs.

Background

In traditional synchronous languages/frameworks like J2EE, ASP.NET or Ruby on Rails, one can use thread-local storage to save a value at the beginning of request handling and later retrieve it deep inside business logic.  This works because each request has its own thread dedicated to serving it.

In this example, REQUEST1 uses Thread-Local Storage to set transaction id (TID) to value 1. Later, code in another module can retrieve TID and get the value 1 back again. Similarly for REQUEST2 which sets TID to value 2.

The situation is different in Node.js and similar async frameworks: a single thread is handling many requests at the same time, interleaving execution of bits of code for each request.

When the code handling requests are interleaved, the code storing TID is called in the same thread for both requests. How to know which value to return later, when the code asks for the stored TID?

When we were looking for alternatives for thread-local storage back in 2014, the concept of continuation-local storage (CLS) looked like a perfect fit: an API similar to that of thread-local storage, that works out of the box in all Node.js applications. (Or does it? Read more to find out!) So we went ahead and implemented loopback.getCurrentContext() API using continuation-local-storage module (see loopback#337).

Even before the pull request landed, the first problems started to pop up: the MongoDB connector did not work well with CLS and required some extra in LoopBack core to make it work again (see the initial comment and the fix). As more people started to use this new feature, more context-propagation bugs started to appear.

As we investigated these issues, we eventually came to the conclusion that Node.js does not provide necessary APIs for a reliable implementation of CLS, and therefore any solution based on current incomplete APIs will require cooperation from all modules to make CLS work correctly in all situations. Because that’s not happening, applications cannot rely on CLS to work correctly. What’s worse, there is no way how to tell in advance whether your application is susceptible to loosing current context (or even worse, seeing the context of another request as the current one). For a  great write-up of CLS problems see node-continuation-local-storage issue #59.

Once we accepted this fact and let go of the wish to have a solution as elegant as CLS (see loopback#1676), it was time to look for alternative ways of propagating extra context. Ritchie Martori (@ritch) proposed a reasonable workaround in loopback#1495 based on using an extra “options” argument. Since most built-in methods like “PersistedModel.create()” or “PersistedModel.find()” already have this argument, we decided to pursue this direction.

The solution

The current solution for context propagation has the following parts:

1. Any additional context is passed in the “options” argument. Built-in methods already accept this argument, and custom user methods must be modified to accept it too.
2. Whenever a method invokes another method, the “options” argument must be passed down the invocation chain.
3. To seed the “options” argument when a method is invoked via a REST call, the “options” argument must be annotated in remoting metadata with the “http” property set to the special string value “optionsFromRequest”. Under the hood, Model.remoteMethod converts this special string value to a function that will be called by strong-remoting for each incoming request to build the value for this parameter.

This way, the context is explicitly propagated through function calls, irrespective of sync/async flow. Because the initial value of the “options” argument is built by a server-side function, the client REST API remains unchanged. Sensitive context data like “currently logged-in user” remain safe from client-side manipulations.

Here is an example of a custom remote method propagating context via “options” argument:

// common/models/my-model.js
module.exports = function(MyModel) {
  MyModel.log = function(messageId, options) {
    const Message = this.app.models.Message;
    // IMPORTANT: forward the options arg
    return Message.findById(messageId, options)
      .then(msg => {
        // NOTE: code similar to this can be placed into
        // Operation hooks too as long as options are forwarded above
        const userId = options
          && options.accessToken
          && options.accessToken.userId;
        const user = userId ? 'user#' + userId : '<anonymous>';
        console.log('(%s) %s', user, msg.text));
      });
  };
};

// common/models/my-model.json
{
  "name": "MyModel",
  // ...
  "methods": {
    "log": {
      "accepts": [
        {"arg": "messageId", "type": "number", "required": true},
        {
          "arg": "options", "type": "object",
          "http": "optionsFromRequest"
        }
      ],
      "http": {"verb": "POST", "path": "/log/:messageId"}
    }
  }
}

Please refer to the documentation to learn how to access the context from Operations hooks and customize the value of “options” argument provided by the remoting layer.

The future of loopback-context

Even though we decided to remove all CLS-based context APIs from LoopBack core, we still wanted to allow existing users to keep using getCurrentContext() API if it worked well for their app, thus we moved all code to a new project loopback-context. Soon after that move was done, a LoopBack user opened a pull request loopback-context#2 to upgrade the module to use a different implementation of CLS based on a newer Node.js API. This work was eventually landed via loopback-context#11 and released as loopback-context@3.0.0.

While the new Node.js API (AsyncWrap via async-hook) promises to work more reliably than the old one (async-listener), the AsyncWrap API is still a work in progress that’s far from being complete. Most importantly, there is no official and standard way for modules to tell AsyncWrap/CLS when and how to correctly restore the continuation context. As a result, any module that implements a custom task queue or a connection pool is prone to break context storage. See loopback-context’s issue tracker for the list of known problems.

We are keeping an eye on the development in Node.js land and when the AsyncWrap API becomes stable and widely supported by user-land modules, we will be happy to bring loopback-context back as an official solution for context propagation again.