vmx

2019-08-01 22:35

There are cases where linking local packages don't produce the same result as if you would've installed all packages from the registry. Here I'd like to tell the story about one of those real world cases and conclude with a solution to those problems.

The problem

When you do an npm install heavy module deduplication and hoisting, which doesn't always behave the same way in all cases. For example if you npm link a package, the resulting node_modules tree is different. This may lead to unexpected runtime errors.

It happened to me recently and I thought I use exactly this real world example to illustrate that problem and a possible solution to it.

Real world example

Preparations

Start with cloning the js-ipfs-mfs and js-ipfs-unixfs-importer repository:

$ git clone https://github.com/ipfs/js-ipfs-mfs --branch v0.12.0 --depth 1
$ git clone https://github.com/ipfs/js-ipfs-unixfs-importer --branch v0.39.11 --depth 1

Our main module is js-ipfs-mfs and let's say you want to make local changes to js-ipfs-unix-importer, which is a direct dependency of js-ipfs-mfs.

First of all you of course make sure that currently the tests pass (we just run a subset, to get to the actual issue faster). I'm sorry that the installation takes so long and so much space, the dev dependencies are quite heavy.

$ cd js-ipfs-mfs
$ npm install
$ npx mocha test/write.spec.js
…
  53 passing (4s)
  1 pending

Ok, all tests passed.

Reproducing the issue

Before we even start modifying js-ipfs-unix-importer, we link it and check that the tests still pass.

$ cd js-ipfs-unixfs-importer
$ npm link
$ cd ../js-ipfs-mfs
$ npm link ipfs-unixfs-importer
$ npx mocha test/write.spec.js
…
  37 passing (2s)
  1 pending
  16 failing
…

Oh, no. The tests failed. But why? The reason is deep down in the code. The root cause is in the [hamt-sharding] module and it's not even a bug. It just checks if something is a Bucket:

static isBucket (o) {
  return o instanceof Bucket
}

instanceof only works if both instances we check on came from the exact same module. Let's see who is importing the hamt-sharding module:

$ npm ls hamt-sharding
ipfs-mfs@0.12.0 /home/vmx/misc/protocollabs/blog/when-npm-link-fails/js-ipfs-mfs
├── hamt-sharding@0.0.2
├─┬ ipfs-unixfs-exporter@0.37.7
│ └── hamt-sharding@0.0.2  deduped
└─┬ UNMET DEPENDENCY ipfs-unixfs-importer@0.39.11
  └── hamt-sharding@0.0.2  deduped

npm ERR! missing: ipfs-unixfs-importer@0.39.11, required by ipfs-mfs@0.12.0

Here we see that ipfs-mfs has a direct dependency on it, and an indirect dependency through ipfs-unixfs-exporter and ipfs-unixfs-importer. All of them use the same version (0.0.2), hence it's deduped and the instanceof call should work. But there's also an error about an UNMET DEPENDENCY, the ipfs-unixfs-importer module we linked to.

To make it clear what's happening inside Node.js. When you require('hamt-sharding') from the ipfs-mfs code base, it will load the module from the physical location js-ipfs-mfs/node_modules/hamt-sharding. When you require it from ipfs-unixfs-importer it will be loaded from js-ipfs-mfs/node_modules/ipfs-unixfs-importer/node_modules/hamt-sharding resp. from ipfs-unixfs-importer/node_modules/hamt-sharding, as js-ipfs-mfs/node_modules/ipfs-unixfs-importer is just a symlink to a symlink to that directory.

When you do a normal installation without linking, you won't have this issue as hamt-sharding will be properly deduplicated and only loaded once from js-ipfs-mfs/node_modules/hamt-sharding.

Possible workarounds that do not work

Though you still like to change ipfs-unixfs-importer locally and test those changes with ipfs-mfs without breaking anything. I had several ideas on how to workaround this. I start with the ones that didn't work:

1. Just delete the js-ipfs-unixfs-importer/node_modules/hamt-sharding directory. The module should still be found in the resolve paths of ipfs-mfs. No it doesn't. Tests fail because hamt-sharding can't be found.
2. Global linking runs an npm install when you run the initial npm link. What if we remove the js-ipfs-unixfs-importer/node_modules completely and symlink to the module manually. That also doesn't work, the hamt-sharding module also can't be found.
3. Install ipfs-unixfs-importer directly with a relative path (npm install ../js-ipfs-unixfs-importer). No, that doesn't work either, it will still have its own node_modules/hamt-sharding, it won't be properly deduplicated.

There must be a way to make local changes to a module and testing them without publishing it each time. Luckily there really is.

Working workaround

I'd like to thank my colleague Hugo Dias for this workaround that he has been using for a while already.

You can just replicate what a normal npm install <package> would be doing. You pack the module and then install that packed package. In our case that means:

$ cd js-ipfs-mfs
$ npm pack ../js-ipfs-unixfs-importer
…
ipfs-unixfs-importer-0.39.11.tgz
$ npm install ipfs-unixfs-importer-0.39.11.tgz
…
+ ipfs-unixfs-importer@0.39.11
added 59 packages from 76 contributors and updated 1 package in 31.698

Now all tests pass.

This is quite a manual process. Luckily Hugo created a module to automate exactly that workflow. It's called connect-deps.

Conclusion

Sometimes linking packages doesn't create the same structure of modules and you need to use packing instead. To automate this you can use connect-deps.

Categories: en, JavaScript, npm