The JavaScript Modules Limbo

screenshot from Playdead ApS’ Limbo game

During a refactoring with the goal to migrate a project to ES2015 standard modules, I’ve encountered so many wasting-time gotchas and vomited a rant on twitter out of patience, mostly complaining about the infamous decision of forcing NodeJS to use the .mjs file extension convention.

… and that’s just the tip of the iceberg …

NodeJS is Not Standard

I know there’s no surprise here, CommonJS is a great convention that made developers able to publish, share, and reuse, npm packages for many years and it worked well.

Indeed, I am not talking about the good old require, I am talking about non standard behaviors that is causing more troubles than it solves: NodeJS is incapable of following ECMAScript specification about loading ECMAScript modules. A simple, standard, statement like the following one:

import realESM from "./realesm.js";

would fail to load a real ESM module that hasn’t been transpiled.

Even if the importing file is named .mjs, which apparently should be the mandatory, unambiguous flag, instructing we want to opt in explicitly to standard JavaScript as meant by specifications, the result will be an error.

Browsers, JSC, and SpiderMonkey follow the Standard

It’s apparently obviously easy for browsers, JSC, and SpiderMonkey to simply follow the bloody standard.

<script type=module src=index.js><!-- browser --></script>

js52 -m index.js # SpiderMonkey

jsc index.js # modules enabled by default

TL;DR is that you have an ESM entry point, you run your app as ESM, easy-peasy. And how come that’s so easy?

But the signal in NodeJS is not respected! If you want to write today a new project, library, module, fully based on standards, you cannot do so without forcing every other environment that simply follows the specification to use an annoying, undesired, and unwelcomed, .mjs file extension.

Diverging even more

The introduction of unnecessary magic in NodeJS environment is actually over-complicating the situation, instead of helping to solve it.

If you write the following statement, which is invalid in every other standard engine, you’d be surprised by how many things could go wrong:

import module from "./module";

Here a quick summary of the fully ambiguous situation you just put yourself into:

• if that’s loaded via an .mjs file, it will look for an esm module named module.mjs
• if that’s loaded after transpilation, it will look for a module.js file

… and what could possibly go wrong ?

The Babel Boom Shakalaka

During last few years we’ve fully locked ourselves behind toolings so that today we’re unable to use standard JavaScript and make it work: congratulations JavaScript community, this was a hell of an achievement!

If Dante’s hell had 9 circles, we’ve reached already 4 of them:

• the polluted builtins’ hell we created by ourselves
• the callback’s hell we’ve created due our laziness
• the Promise’s hell we’ve inherited from the previous point ’cause we just moved the problem instead of understanding it
• the transpilation’s hell we’ve all early jumped in solving short terms relative issues causing long term disasters

While we wait for the next circle to happen, let me explain the current one.

A Basic NodeJS Module

export default function test() { console.log('ok'); };

Now, that file is named module.mjs, because we understood NodeJS has identity crisis and we want to help it coming out of it.

If we use that module via a simple index.mjs that does only the following:

import test from "./module.mjs"; test();

we’ll succeed! To test that, start node with the current experimental flag:

node --experimental-modules index.mjs

The Babel Output

We have a universal, perfectly working, module that in an ideal world, where every engine is updated, would not require extra steps.

Then reality kicks in and Babel comes to help publishing the module.

Bear in mind this post is not against Babel or tooling in general, it’s just a demonstration of the problems our choice to early transpile modules caused in the long term. Mostly every tooling that early solved standards have caveats, we need to understand and be aware of these instead of keep ignoring them.

babel ./*.mjs --out-dir ./package/ --presets=es2015

which will produce the following package/{index,module}.js files:

package/index.js

package/module.js

Problem #1

Have you noticed these files have been exported as .js instead of .mjs ?

So, you’ve also noticed our code is already broken due that require("./module.mjs") that points to the wrong file, right?

The ideal solution

To solve the first problem following standards, we should simply call files .js so that Babel won’t generate broken code, Browsers will automatically be compatible with the full file name and source code, and so will be every other JS engine.

But this is so simple that it has to break in NodeJS, because you cannot write code that imports .js, remember? ESM in NodeJS can only load .mjs

The current *NodeJS only* solution

We clearly have a conflict already between what standards would allow and what .mjs decision wouldn’t, so that NodeJS own implementation of the standard is that you can omit the file extension and magic happens.

import test from "./module";

We now have a non standard behavior that works only in NodeJS but cannot ever be tested on browsers, JSC, or SpiderMonkey, unless we add some alchemy upfront to make it happen, penalizing for no reasons portability and production performances.

That import statement will be transpiled as such, moving the problem instead of really solving the issue:

require("./module");

Problem #2

We’ve solved the file extension issue, letting some non standard mechanism condition our code that is now not portable anymore across different environments, including our possible main target: browsers.

And while this solution was needed to solve transpiled code issues, we still have fully broken code. How?

Imagine instead of the index.mjs file, all we exported was just the module.mjs one. Now imagine there is a project that installs module , and use it within its ESM wannabe file. A third parts index.mjs file we don’t control, but that does exactly the same thing, this time relying on the installed package:

import test from "module"; test();

… and BOOOM! That breaks because our transpiled module.mjs uses a CommonJS+Babel convention with an exports.default property that will not be handled by NodeJS magic importer while importing.

How hilarious …

The current solution without transpiling

Now that we import an npm package instead of an .mjs file, assuming we know that package was published transpiled to be as compatible as possible, we need to reach the .default export.

import module from "module";

const test = module.default; test();

But wasn’t the whole .mjs point to avoid these kind of situations?

Not only we need to know if we have to reach the ugly .default property upfront now, our code changed from not portable in terms of standards, to fully ambiguous in terms of usage. How, you ask again?

If you transpile that file via Babel, the initial import would work through CommonJS by default, meaning we need to know how the module was published, and if we need Babel to use it because it exported a default.

So, our initial one line module is now not portable, environment hostile, and fully dependent on transpilation, which in turns it means that two teams working on the same JS source code across the company are unable to decide themselves if they want to use tranpilers or not, or if they want to use a module in the browser or not: interoperability within JS itslef is lost.

This is a full-circle lock-in behind Babel instrumentation to load CommonJS through ESM syntax, and a whole new hell for modules authors to consider.

We’re developing upside-down

Most npm development is done for browsers, yet we are dumbing down modern browsers native features to use NodeJS, non browser, module system, and re-publish bloated non standard code in production.

Read above sentence again because most likely that’s what you are doing.

We locked the Web and browsers behind a module system that never belonged to them, and we’re now trapped behind this early transpiling choice that somehow forced NodeJS to use .mjs , a solution that today would never scale for portability like standard .js file would do, a solution lost ASAP if you have transpilers instead of standard code.

As consequence, this is the current JS development situation, which is entirely our own fault and it’s absolutely not a good thing: we’ve been fighting against the very same standards we were supposed to early adopt.

Can we solve this mess?

To be honest, who already set up an environment behind Babel, which I believe is the majority of early ESM adopters, should probably do nothing … like, literally, nothing at all.

You know that already worked for you, and that’s at the end the less ambiguous or messed up situation, compared to those like me that wanted to follow standards and also the new file extension.

It’s now clear to me that .mjs doesn’t really scale, doesn’t grant any developer or environment signal, and it doesn’t solve current real-world JavaScript’s development status: it’s practically annoying without any real-world benefit.

The only signal to opt in for ESM should be the usage of the import statement as defined by specifications.

I fell like I’ve just wasted a lot of time trying to make things work the way standards were telling me on the client side, combined with the non standard but apparently official direction taken by NodeJS environment.

I cannot publish a module that would’t break once included and transpiled.

I cannot trust the very same standard JS code I’m writing: WTF, seriously!

A possible pattern to move forward

Forget about .mjs, keep .js and code happily ever after via @std/esm configuring it via package.json as such:

"@std/esm": {"esm": "js"}

or simply

"@std/esm": "js"

Once you’ve done that, everything problematic mentioned in this post will disappear. You now rename both index.mjs and module.mjs files into index.js and module.js and launch your program like this:

node -r @std/esm index.js

At this point you have .js extensions that will work with every one-off server side spinner you have, you can finally import test from "./module.js"; specifying the entire file name and that will work out of the box in browsers, JSC, SpiderMonkey, and whatever other environment you want.

Start publishing non transpiled modules!

It’s not your responsibility, as package/module author, to force developers using your module to use Babel to import it properly.

The real-world situation here is that mostly everyone is using NodeJS as Web development environment and NodeJS, from 4 to current one, can use @std/esm without issues.

Do a major update to packages you want to publish fully @std/esm compatible so that migration can be painless.

Once most famous modules will be available as ES2015 standard, NodeJS would have no choice but offer a way to fully opt-in into ESM without needing de-facto useless conventions like .mjs unfortunately is.

That day we will change -r @std/esm into -esm and everyone will live happily ever after!

… any possible extra hint?

Using require to signal CJS import intent and import to signal ESM could be another option. The issue here is that nobody is probably going to refactor that, specially because require in ESM is not available (it’s not part of any standard so that’s OK) and including another dependency upfront, beside @std/esm , would not be developer friendly.

… and what about browsers?

This is most likely a non issue you are trying to address, because production code targeting browsers inevitably needs to be bundled.

You don’t want users to download unminified code; you don’t want users to load 700 files at runtime; you want to test as it is on your modern browser then you want to publish production code through bundlers.

ESM is there to keep the RAD aspect of JavaScript available to everyone, but it’s not really the best way to ship entire PWAs.

We still need tree-shaking, transpiling accordingly with targets, minifications, etcetera, and these are also the file we should offer via any CDN: the original entry point, if needed, as well as eventually the esm and cjs file.

Of course what I’m talking about requires a bit of community effort, but it can be done because bundlers such Webpack or Rollup have been created for that.

After all, tooling should be there to help us, not be on our way so … let’s step back a tiny bit and put ourselves where that’s still the case, shall we?