Learning by fixing: Node.js, modules and packages

Everything is a skill and everything can be learned. Even learning itself is something that can be trained and improved. And in software development, constant learning is at the core of a successful career as a developer. I believe that one of the most efficient ways to gain really deep knowledge fast is not to go through online courses, or not even build something yourself, but to fix a problem.

In this article, I want to share one of the problems that I encountered recently, which turned out a perfect use case to practice deep learning of how and why Node resolves modules and how it deals with packages. I will walk you through the process of finding a solution step by step and explain relevant knowledge and discoveries along the way.

To get the most of the article some experience with modern frontend ecosystem and basic knowledge of tools like npm and webpack would be helpful.

The mystery

The project I was working on has a pretty standard frontend setup: React, Next.js, css-in-js, UI components from an external library to build the interface. One day I added a new component, and although everything worked locally, my CI, while attempting to build the website, greeted me with this:

After a bit of “errr, wat?”, scratching my head furiously, meddling with CI config and doing the usual clean-the-cache, nuke-node-modules cargo-culting activities I managed to reproduce the problem locally. Turned out that:

• it only happens with the Lozenge component, which uses Compiled - a new css-in-js library
• it was working locally because the version of Node in the CI was newer than on my local machine.

So clearly the problem was either with Lozenge, or with the Compiled library itself, and clearly, there was something in their code that prevented it from working with the latest Node. So the solution to the problem seemed simple: downgrade version of Node in the CI to unblock my builds and raise an issue with the library in the hope that maintainers, who know their source code, can figure it out.

Only…

What can possibly a Lozenge component, that just renders a few divs, or a css-in-js library, that converts js-written styles into <style> tags can have, that depends on a version of Node? Especially on an old version of Node? It’s usually the other way around…

It’s a proper mystery! Time to put my Sherlock hat on and solve it.

ESM or CJS

The very first step of playing a detective would be to take a closer look at the crime scene and extract all available clues from there.

In this case, we can see that a file from the path /dist/cjs/Lozenge/Container.js is trying to require a file with the path /dist/esm/runtime.js. This is the very first case of digging deep in this investigation: what is CJS, ESM, and why the error is there?

First of all, due to various historic reasons, javascript at the beginning didn’t have any concept of a module, you could not from one file import content of another file. Various attempts to introduce modularity in javascript resulted in a bunch of different formats, with CJS and ESM being the most commonly used ones.

Multiple articles and deep dives are dedicated to the topic. The key differences that are relevant here:

ESM — “modern” format, uses “import” syntax, familiar to anyone who ever wrote anything with Typescript or React.

import React from 'react';

CJS — “old” format, uses “require” syntax, mostly could be found in Node apps and in the results of bundlers/compilers since it’s the only format that Node can understand (until very recently).

const React = require('react').default;

CJS, since it’s older, can’t deal with ESM, any attempts to use “import”, or require a file in ESM format, will fail. And this is exactly what is happening: for some reason, Lozenge in CJS format attempts to require runtime.js file in ESM format.

It is not unusual to see both of those formats to be distributed in modern frontend libraries. One of the reasons for this is that ESM code makes it easier for tools like Webpack to do “tree-shaking” (elimination of unused code), which positively affects the final javascript size.

In our case, if we look inside project’s node_modules at what is installed in Lozenge and Compiled, we will see that this is exactly the case for both of them: their respective dist folders contain folders with both CJS and ESM code.

If you’re not familiar with how node_modules work, this article might be an interesting read: https://medium.com/@adevnadia/webpack-and-yarn-magic-against-duplicates-in-bundles-52b5e1a5e2e2

Modules resolution and packages

Now, that we have an idea of where the problem occurs (CJS module in Lozenge tries to require ESM module from Compiled), it’s time to dig deeper and understand why exactly this happens. First of all, let’s check out what is happening in the problematic file /dist/cjs/Lozenge/Container.js:

...
var _react = _interopRequireDefault(require("react"));

var _runtime = require("@compiled/react/runtime");

var _constants = require("@atlaskit/theme/constants");
...

This looks like would you would typically expect to see in CJS code that was transpiled from something like Typescript, and it gives us a few clues for the further investigation:

• Compiled’s runtime is indeed required by the Lozenge, as expected
• It is required as a deep import from @compiled/react package

If Lozenge for some reason required ESM module directly as a file path, then the problem would have been a bug somewhere in Lozenge’s compilation process. In this case, however, everything looks fine from the Lozenge perspective, which allows us to eliminate it from the investigation and focus on Compiled only.

Another interesting thing in this code is the deep import from Compiled. Normal packages can only have names composed from 2 parts — scope and package name itself, i.e. @compiled/react is the name you’d see on npm. Additional /runtime suggests that some sort of custom multi-entry strategy is implemented here, likely for the purpose of reducing bundle sizes.

Let's take a closer look at Compiled itself now. If we open /node_modules/@compiled/react/ folder we’ll see:

• package.json file at the root, with all the usual fields that you’d expect from an npm package, like name, version or dependencies;
• dist folder, with CJS and ESM code for the entire package
• runtime folder, also with package.json file. This one looks weird, it only has a few fields, and all of them are relative links to files inside dist folder. No name, version, etc

{
  "main": "../dist/cjs/runtime.js",
  "module": "../dist/esm/runtime.js",
  "browser": "../dist/browser/runtime.js",
  "types": "../dist/esm/runtime.d.ts"
}

And good news! We see a direct link to the file that Lozenge somehow was trying to require during the build process — “../dist/esm/runtime.js” 🥳. Can it be that something in the build process got confused and mixed up links in main and module? What is the purpose of those by the way, how it works exactly? Time to read the docs again, this time digging deep into how Node resolves modules and deals with package.json.

First of all, Node looks for required paths in node_modules folders, starting from the file where require was called, and going up the ancestor tree until it finds something useful (or throws). In our case, it will start from /node_modules/@atlaskit/lozenge/dist/cjs/Lozenge/Container.js, and will attempt to find node_modules in:

• /node_modules/@atlaskit/lozenge/dist/cjs/Lozenge
• /node_modules/@atlaskit/lozenge/dist/cjs
• /node_modules/@atlaskit/lozenge/dist
• /node_modules/@atlaskit/lozenge
• …

until it finally reaches the root node_modules where it finds /node_modules/@compiled/react/runtime folder with the weird package.json inside. If I had a version of Compiled in my root package.json different from the one used in Lozenge, then npm would install an additional copy of it in /node_modules/@atlaskit/lozenge/node_modules/@compiled/react, and the search above would have stopped on the fourth step instead of going up to the very root. This is how and why we can use different versions of one package in the same application.

On every iteration node will try to find package.json file, and when it does, it will grab main field from it and try to resolve it as a file. Node itself doesn’t know or care about anything else other than main field, which explains why in our case package.json in @compiled/react/runtime folder looks so weird — it’s not an actual “package”, but just a way to trick Node into resolving that file via the simplified path. If we wanted to, we could’ve reached this exact file via require(‘@compiled/react/dist/cjs/runtime/index.js’).

In theory, if in main field of that package.json was a link to a ESM file, that would explain our mystery. Unfortunately, the field is absolutely correct and points to CJS file as expected. There is, however, “module” field there, which points to what we need! And there is nothing about it in node’s docs, so it’s not something that is native to node.

{
  "main": "../dist/cjs/runtime.js",
  "module": "../dist/esm/runtime.js",
  "browser": "../dist/browser/runtime.js",
  "types": "../dist/esm/runtime.d.ts"
}

After some googling, the investigation revealed the answer: this field is used by bundling tools (Webpack, Rollup, etc) to bypass node’s standard resolution algorithm, avoid CJS and bundle ESM code directly. This field is de-facto standard, although not supported by node, and widely used by libraries that distribute both CJS and ESM code.

Great, looks like we’re really close and the answer is right there! Can it be just some really weird edge-casy bug in webpack itself, that somehow confused Lozenge CJS code into using file from module field instead of main? 🥁 Really easy to verify: just replace ESM link in module with CJS, so that it points to the code Lozenge can deal with, run the build, and 🤞🏽…

and…

And it didn’t work: the build still fails in the exact same place 🤦🏽‍♀️. Looks like this field was not used at all and we’re clearly missing something. Time to step back a bit, re-group and think some more. What do we know?

• all the fields are correct and on their places
• both CJS and ESM code is there
• all the packages are on their right places and according to node resolution algorithm correct code should be used
• the build fails on the new version of Node but works on the older one

What we don’t know yet, is what changed? Something was introduced to Node that somehow ruins this perfectly designed investigation. A bit of triaging narrowed down the version where everything starts to fail to 12.17.x., and its changelog gives another clue: this is the version where ESM support was enabled by default, without an additional flag.

Package exports

Although “ESM support by default” sounds like good news, it’s not that helpful in reality: that means that the change that breaks the build could be introduced in any previous version and was just hidden until now. On the other hand, it proves that:

• whatever happens is related to ESM specifically
• and since we verified that all the fields and code are correct, then there has to be something explicit in Compiled, related to ESM, that overrides the default node behaviour

At this point I did something that, arguably, I should’ve done at the very beginning of the investigation (but then there wouldn’t be a good mystery and that article): I searched through any mention of /esm/runtime.js in its source code. And finally, some luck! In the package.json at the root of @compiled/react package, hidden among other fields, there was this:

"exports": {
  ".": "./dist/esm/index.js",
  "./babel-plugin": "./dist/esm/babel-plugin.js",
  "./runtime": "./dist/esm/runtime.js"
},

“Exports” is a relatively new addition to Node that allows developers to specify multiple entries to the package instead of a single “main” entry like it was in the past. And if we look at the node modules resolution algorithm again, we will see that introduction of “exports” turns what we discovered earlier in a bit more complicated process: now on every iteration of looking for the right folder Node will try to:

• parse the required path (@compiled/react/runtime), extract from it package scope (@compiled), package name (react) and subpath (everything else from the path, i.e. /runtime)
• If there is a package.json available in the path that is a combination of scope + name (@compiled/react), then it extracts exports from it and tries to match it with the required path

And this is finally the end of the mystery! Since Compiled at the root had package.json with exports, node never even reached the weird package.json with the reference to the correct CJS runtime.js, and that’s why meddling with it didn’t have any effect on the build. And in the older version of Node exports is not supported, that field was ignored, and the build would use the weird package.json with the correct link and therefore would work. And ./runtime entry of exports references ESM module, which is picked up by the Lozenge’s CJS file and causes the build to fail. Replacing this link with CJS one finally fixes the build locally 🥳.

Only one thing left to do — fix it for real. Just replacing ESM with CJS in the actual library doesn’t seem like a good idea, that would prevent consumers of the library from using ESM modules completely. Luckily, there is an answer to this: conditional exports. This is a way to map different paths and subpaths depending on certain conditions, for example, require or import type of modules. And the actual solution, that works both for consumers that want to use CJS or ESM version of Compiled, would look like this:

"exports": {
  ...
  "./runtime": {
    "import": "./dist/esm/runtime.js",
    "require": "./dist/cjs/runtime.js"
  }
}

Now, when CJS Lozenge requires @compiled/react/runtime with Node version more than 12.17.x it will:

• find @compiled/react folder in project’s node_modules
• parse node_modules/@compiled/react/package.json file
• extract exports field from it
• match ./runtime entry with our requested path
• detect that the request comes from CJS file and resolve the correct ./dist/cjs/runtime.js file

And older Node will:

• find @compiled/react/runtime folder in project’s node_modules
• parse node_modules/@compiled/react/runtime/package.json file
• extract main field from it
• resolve the correct ../dist/cjs/runtime.js file

And everything now Just Works™!