How do you define node_modules as externs in Closure Compiler?

Question

I have a Node.js project that I want to compile with Closure Compiler. I do not want it to run in the browser/use browserify. I mainly want the utility of type checking. I originally got the compiler to work correctly using the following:

java -jar compiler.jar -W VERBOSE 
                       --language_in ECMASCRIPT5_STRICT 
                       --externs closure-externs.js 
                       --js="lib/**.js"

Where closure-externs.js manually defined variables and functions which I was using from Node.js in a rather crude way:

// closure-externs.js

/** @constructor */function Buffer(something){}
function require(path){}
var process = {};
[...]

It turns out that this worked only through sheer luck. There is no dependency tracking between files, so you can have cases where you return a type {Foo} and the compiler will complain that it doesn't exist (depending on the machine, depending on the compile order). I then found out I was doing it all wrong and should be using --process_common_js_modules so the compiler will do dependency tracking where I require("foo"). I am currently invoking the compiler like so:

java -jar compiler.jar -W VERBOSE 
                       --language_in ECMASCRIPT5_STRICT 
                       --externs externs/fs.js 
                       --js="lib/**.js"
                       --process_common_js_modules 
                       --common_js_entry_module app.js

But this is failing with:

 ERROR - required entry point "module$crypto" never provided
 ERROR - required entry point "module$dgram" never provided
 ERROR - required entry point "module$extend" never provided
 ERROR - required entry point "module$fs" never provided
 ERROR - required entry point "module$net" never provided
 ERROR - required entry point "module$q" never provided

Some of these modules are native to Node.js (e.g. fs) whereas others are contained in node_modules like q. I don't want to run these external modules through the compiler, so I know I need to set up externs file(s) for them. I know there is https://github.com/dcodeIO/node.js-closure-compiler-externs for common Node.js externs, and I know how to invoke them on the compiler, but for some reason when I do something like --externs externs/fs.js the error for module$fs remains. What am I doing wrong?

I know there's other flags like --module and --common_js_module_path_prefix but I'm not sure if I need to use them to get this to work or not. My Google-fu has failed to come up with any answers on the correct incantation here. :(

Answer 1

The issue is that you wish for the compiler to somehow recognize that certain require calls are internal, namely that the required module should be processed by the compiler as source, and others are external so should be left alone. There isn't a good way to handle this situation currently.

Workarounds

Use Post-processing to Add External Require Statements

In this scenario you would completely omit any require statements to external modules. The compiler would only process code with internal require statements and modules. After compilation, you would prepend the external require statements:

Header JS To Be Prepended

var crypto = require('crypto');

Source To Be Compiled

console.log(crypto);

Because crypto is declared in an extern, the compiler will correctly recognize the type and symbol name.

Alias Require Calls

When the --process_common_js_modules is specified, the compiler recognizes require statements and expands them in a similar fashion to the way macros work in other languages. By aliasing the require statements that should remain external, the compiler will not recognize them and thus not expand them.

Source To Be Compiled

var externalRequire = require;
/** @suppress {duplicate} this is already defined in externs */
var crypto = externalRequire('crypto');
console.log(crypto)

Answer 2

If you're using the Closure Compiler only for type-checking—i.e., with the --checks-only option—there is another workaround which has the advantage (over the ones mentioned in Chad's answer) of working correctly with unmodified third-party NPM modules which in turn import built-in modules.

Using Stubs

The trick is to create stub NPM modules to stand in for the built-in ones. These can be minimal; they only need to declare the parts of the API you're actually using.

Here's an example for the path built-in module.

In externs/path/path.js I have the "externs" declarations (not actually externs, so e.g. you can't use @nosideeffects) for the part of path that I need:

/** @const */
var path = {};

/**
 * @param {string} path
 * @return {string}
 */
path.dirname = function(path) {};

/**
 * @param {string} path
 * @return {string}
 */
path.extname = function(path) {};

/**
 * @param {...string} var_args
 * @return {string}
 */
path.join = function(var_args) {};

module.exports = path;

In externs/path/package.json, I have a minimal NPM package config:

{
  "description": "Fake package.json for require('path')",
  "main": "path.js",
  "name": "path",
}

Then create a symlink from node_modules/path to externs/path, and add the following to my compiler flags:

node_modules/path/package.json
node_modules/path/path.js

(You could put the stub implementation directly in to node_modules but I prefer to keep my stubs separate from the real modules managed by npm. I just need to remember to manually add the symlinks to my Git repo, because it's otherwise configured to ignore node_modules.)