Learning by fixing: Node.js, modules and packages
Everything is a skill and everything can be learned. Even learning itself is something that can be trained and improved. And in software development, constant learning is at the core of a successful career as a developer. I believe that one of the most efficient ways to gain really deep knowledge fast is not to go through online courses, or not even build something yourself, but to fix a problem.
In this article, I want to share one of the problems that I encountered recently, which turned out a perfect use case to practice deep learning of how and why Node resolves modules and how it deals with packages. I will walk you through the process of finding a solution step by step and explain relevant knowledge and discoveries along the way.
To get the most of the article some experience with modern frontend ecosystem and basic knowledge of tools like npm and webpack would be helpful.
The project I was working on has a pretty standard frontend setup: React, Next.js, css-in-js, UI components from an external library to build the interface. One day I added a new component, and although everything worked locally, my CI, while attempting to build the website, greeted me with this:
After a bit of “errr, wat?”, scratching my head furiously, meddling with CI config and doing the usual clean-the-cache, nuke-node-modules cargo-culting activities I managed to reproduce the problem locally. Turned out that:
- it only happens with the Lozenge component, which uses Compiled - a new css-in-js library
- it was working locally because the version of Node in the CI was newer than on my local machine.
So clearly the problem was either with Lozenge, or with the Compiled library itself, and clearly, there was something in their code that prevented it from working with the latest Node. So the solution to the problem seemed simple: downgrade version of Node in the CI to unblock my builds and raise an issue with the library in the hope that maintainers, who know their source code, can figure it out.
What can possibly a Lozenge component, that just renders a few divs, or a css-in-js library, that converts js-written styles into <style> tags can have, that depends on a version of Node? Especially on an old version of Node? It’s usually the other way around…
It’s a proper mystery! Time to put my Sherlock hat on and solve it.
ESM or CJS
The very first step of playing a detective would be to take a closer look at the crime scene and extract all available clues from there.
In this case, we can see that a file from the path
/dist/cjs/Lozenge/Container.js is trying to require a file with the path
/dist/esm/runtime.js. This is the very first case of digging deep in this investigation: what is
ESM, and why the error is there?
ESM being the most commonly used ones.
ESM — “modern” format, uses “import” syntax, familiar to anyone who ever wrote anything with Typescript or React.
import React from 'react';
CJS — “old” format, uses “require” syntax, mostly could be found in Node apps and in the results of bundlers/compilers since it’s the only format that Node can understand (until very recently).
const React = require('react').default;
CJS, since it’s older, can’t deal with
ESM, any attempts to use “import”, or require a file in
ESM format, will fail. And this is exactly what is happening: for some reason, Lozenge in
CJS format attempts to require
runtime.js file in
It is not unusual to see both of those formats to be distributed in modern frontend libraries. One of the reasons for this is that
In our case, if we look inside project’s
node_modules at what is installed in Lozenge and Compiled, we will see that this is exactly the case for both of them: their respective
dist folders contain folders with both
If you’re not familiar with how
node_modules work, this article might be an interesting read: https://medium.com/@adevnadia/webpack-and-yarn-magic-against-duplicates-in-bundles-52b5e1a5e2e2
Modules resolution and packages
Now, that we have an idea of where the problem occurs (
CJS module in Lozenge tries to require
ESM module from Compiled), it’s time to dig deeper and understand why exactly this happens. First of all, let’s check out what is happening in the problematic file
...var _react = _interopRequireDefault(require("react"));
var _runtime = require("@compiled/react/runtime");
var _constants = require("@atlaskit/theme/constants");
This looks like would you would typically expect to see in
CJS code that was transpiled from something like Typescript, and it gives us a few clues for the further investigation:
- Compiled’s runtime is indeed required by the Lozenge, as expected
- It is required as a deep import from
If Lozenge for some reason required
ESM module directly as a file path, then the problem would have been a bug somewhere in Lozenge’s compilation process. In this case, however, everything looks fine from the Lozenge perspective, which allows us to eliminate it from the investigation and focus on Compiled only.
Another interesting thing in this code is the deep import from Compiled. Normal packages can only have names composed from 2 parts — scope and package name itself, i.e.
@compiled/react is the name you’d see on npm. Additional
/runtime suggests that some sort of custom multi-entry strategy is implemented here, likely for the purpose of reducing bundle sizes.
Let's take a closer look at Compiled itself now. If we open
/node_modules/@compiled/react/ folder we’ll see:
package.jsonfile at the root, with all the usual fields that you’d expect from an npm package, like
ESMcode for the entire package
runtimefolder, also with
package.jsonfile. This one looks weird, it only has a few fields, and all of them are relative links to files inside
And good news! We see a direct link to the file that Lozenge somehow was trying to require during the build process —
“../dist/esm/runtime.js” 🥳. Can it be that something in the build process got confused and mixed up links in
module? What is the purpose of those by the way, how it works exactly? Time to read the docs again, this time digging deep into how Node resolves modules and deals with
First of all, Node looks for required paths in
node_modules folders, starting from the file where
require was called, and going up the ancestor tree until it finds something useful (or throws). In our case, it will start from
/node_modules/@atlaskit/lozenge/dist/cjs/Lozenge/Container.js, and will attempt to find
until it finally reaches the root
node_modules where it finds
/node_modules/@compiled/react/runtime folder with the weird
package.json inside. If I had a version of Compiled in my root
package.json different from the one used in Lozenge, then npm would install an additional copy of it in
/node_modules/@atlaskit/lozenge/node_modules/@compiled/react, and the search above would have stopped on the fourth step instead of going up to the very root. This is how and why we can use different versions of one package in the same application.
On every iteration node will try to find
package.json file, and when it does, it will grab
main field from it and try to resolve it as a file. Node itself doesn’t know or care about anything else other than
main field, which explains why in our case
@compiled/react/runtime folder looks so weird — it’s not an actual “package”, but just a way to trick Node into resolving that file via the simplified path. If we wanted to, we could’ve reached this exact file via
In theory, if in
main field of that
package.json was a link to a
ESM file, that would explain our mystery. Unfortunately, the field is absolutely correct and points to
CJS file as expected. There is, however, “module” field there, which points to what we need! And there is nothing about it in node’s docs, so it’s not something that is native to node.
After some googling, the investigation revealed the answer: this field is used by bundling tools (Webpack, Rollup, etc) to bypass node’s standard resolution algorithm, avoid
CJS and bundle
ESM code directly. This field is de-facto standard, although not supported by node, and widely used by libraries that distribute both
Great, looks like we’re really close and the answer is right there! Can it be just some really weird edge-casy bug in webpack itself, that somehow confused Lozenge
CJS code into using file from
module field instead of
main? 🥁 Really easy to verify: just replace
ESM link in
CJS, so that it points to the code Lozenge can deal with, run the build, and 🤞🏽…
And it didn’t work: the build still fails in the exact same place 🤦🏽♀️. Looks like this field was not used at all and we’re clearly missing something. Time to step back a bit, re-group and think some more. What do we know?
- all the fields are correct and on their places
ESMcode is there
- all the packages are on their right places and according to node resolution algorithm correct code should be used
- the build fails on the new version of Node but works on the older one
What we don’t know yet, is what changed? Something was introduced to Node that somehow ruins this perfectly designed investigation. A bit of triaging narrowed down the version where everything starts to fail to 12.17.x., and its changelog gives another clue: this is the version where
ESM support was enabled by default, without an additional flag.
Although “ESM support by default” sounds like good news, it’s not that helpful in reality: that means that the change that breaks the build could be introduced in any previous version and was just hidden until now. On the other hand, it proves that:
- whatever happens is related to
- and since we verified that all the fields and code are correct, then there has to be something explicit in Compiled, related to
ESM, that overrides the default node behaviour
At this point I did something that, arguably, I should’ve done at the very beginning of the investigation (but then there wouldn’t be a good mystery and that article): I searched through any mention of
/esm/runtime.js in its source code. And finally, some luck! In the
package.json at the root of
@compiled/react package, hidden among other fields, there was this:
“Exports” is a relatively new addition to Node that allows developers to specify multiple entries to the package instead of a single “main” entry like it was in the past. And if we look at the node modules resolution algorithm again, we will see that introduction of “exports” turns what we discovered earlier in a bit more complicated process: now on every iteration of looking for the right folder Node will try to:
- parse the required path (
@compiled/react/runtime), extract from it package scope (
@compiled), package name (
react) and subpath (everything else from the path, i.e.
- If there is a
package.jsonavailable in the path that is a combination of scope + name (
@compiled/react), then it extracts
exportsfrom it and tries to match it with the required
And this is finally the end of the mystery! Since Compiled at the root had
package.json with exports, node never even reached the weird
package.json with the reference to the correct
runtime.js, and that’s why meddling with it didn’t have any effect on the build. And in the older version of Node
exports is not supported, that field was ignored, and the build would use the weird
package.json with the correct link and therefore would work. And
./runtime entry of
ESM module, which is picked up by the Lozenge’s
CJS file and causes the build to fail. Replacing this link with
CJS one finally fixes the build locally 🥳.
Only one thing left to do — fix it for real. Just replacing
CJS in the actual library doesn’t seem like a good idea, that would prevent consumers of the library from using
ESM modules completely. Luckily, there is an answer to this: conditional exports. This is a way to map different paths and subpaths depending on certain conditions, for example, require or import type of modules. And the actual solution, that works both for consumers that want to use
ESM version of Compiled, would look like this:
CJS Lozenge requires
@compiled/react/runtime with Node version more than 12.17.x it will:
@compiled/reactfolder in project’s
exportsfield from it
./runtimeentry with our requested path
- detect that the request comes from
CJSfile and resolve the correct
And older Node will:
@compiled/react/runtimefolder in project’s
mainfield from it
- resolve the correct
And everything now Just Works™!