Accessors Wiki

Require

This page contains notes about require(). See Module Specification for a short summary of how to use require().

Use npm to install modules at runtime

Some hosts, such as the Node Host can install packages at run time using a package manager such as NPM. See accessors/web/hosts/node/node_modules/@accessors-modules/serial/serial.js

It turns out that node/lib/modules.js caches the paths for which it has searched. See https://github.com/nodejs/node/blob/master/lib/module.js for the node 7.x version.

Thus, invoking require() for a missing package adds the paths where it searched to the cache.

If we then use npm install to install the package, invoking require() again will look at its cached values and not find our newly installed package.

I verified this by building node 6.10.2 from sources. https://github.com/nodejs/node/releases lists the tags for the releases. node/lib/modules.js has several caches, I think stat.cache is the problem. I don't see how I can get access to stat.cache.

My solution is to search for the module before invoking require(). If the module is not found, then we run npm install. Then we require() the module.

Happily, module.paths is an array of the node_modules directories that will be searched. My sample code is below.

The problem with invoking npm install is that it takes time and if the value of the timeout parameter is short, then execution may end before the installation has completed.

I'm not sure of a solution for the above.

I don't think we should redefine require() because there is quite a bit of work in modules.js .

If we define our own require(), then we need to have a perfect implementation of what is in modules.js that evolves as modules.js evolves.

We don't want to override require() for all packages because this method always searches for the package and runs npm install if it is not found. Thus, this is not a good fit for built in packages.

accessors/web/hosts/node/node_modules/@accessors-modules/serial/serial.js contains:

var nodeHost = require('@accessors-hosts/node');
var SerialPort = nodeHost.installIfMissingThenRequire('serialport');

accessors/web/hosts/node/nodeHost.js contains:

/** Check the require path for a module and if it is not found, invoke
* npm install.
*
* In node, the module loading system caches the return values of stat()
* calls. Thus, if we require() a missing package, then installing it
* will not invalidate the cache and calling require() again will not
* find our newly installed cache. As a workaround, for possibly missing
* packages, we search the array of paths contained by module.paths.
*
* @param npmPackage the package to be possibly installed using npm
* and then required.
* @return the value returned by requiring the package
*/
function installIfMissingThenRequire(npmPackage) {
// console.log('nodeHost.js: installIfMissingThenRequire(' + npmPackage);
const paths = module.paths;
var foundIt = false;
for (var i = 0; i < paths.length; i++) {
if (fs.existsSync(paths[i] + '/' + npmPackage)) {
// console.log('nodeHost.js: installIfMissingThenRequire(' + npmPackage + '): found ' + paths[i] + '/' + npmPackage);
foundIt = true;
break;
}
}
if (!foundIt) {
var execSync = require('child_process').execSync;
var npmOutput;
try {
npmOutput = execSync('npm install ' + npmPackage);
} catch (error) {
console.log('npm install ' + npmPackage + ' failed: ' + error + '. A return code of 1 can typically be ignored because package.json is present.');
}
}
return require(npmPackage);
}

util

All of the accessor hosts should be able to load the util module
util.js should not be duplicated. In April, 2016, we had:
- accessors/web/hosts/browser/modules/util.js - Used by the Browser Host To Do: Merge changes in to accessors/web/hosts/common/modules/util.js and use accessors/web/hosts/common/modules/util.js instead of accessors/web/hosts/browser/modules/util.js
  - Checked in as r376 on 12/20/2015. Comment: "Support for loading modules in browser host, and very beginning of a Node.js host using the same infrastructure as the browser host"
- accessors/web/hosts/common/modules/util.js See comm/modules/util.js below
- accessors/web/hosts/node/node_modules/@accessors-modules/util/util.js - Supposed to be used by Node Host, but node does not load this file, so it was removed. See In Node, util is core, below.
- $PTII/ptolemy/actor/lib/jjs/node/util.js See $PTII/ptolemy/actor/lib/jjs/node/util.js below

Edward wrote:

"We could create a directory accessors/common/modules and pure pure JavaScript modules there. Every host would need to modified to look there if it doesn't find a local module. It would be worth looking at whether the CommonJS standard says anything about the search path. Then look at the docs for Node.js require() to figure out what it uses for a search path."

"As you've said before, this is like a classpath. My guess is that the standard is agnostic and that Node.js has a mechanism for specifying a require search path."

common/modules/util.js

accessors/web/hosts/common/modules/util.js Loaded by commonHost.js under the Duktape Host.
- Checked in as r694 on 4/14/2016. Comment: "...util.js from node_modules/util for use with Duktape.".
  - Note that node_modules/util/util.js was removed, but is based on https://github.com/defunctzombie/node-util/blob/master/util.js

Below are the diffs on 4/18/16 between the github version and our version:

To get the GitHub version, use:

  wget -O /tmp/util.js https://raw.githubusercontent.com/defunctzombie/node-util/master/util.js

Then run diff:

bash-3.2$ diff -rwb /tmp/util.js ./{$HOSTS_COMMON}/modules/util.js
210a211
>
525a527,530
> // See {$ACCESSORS_HOME}/wiki/Main/ResourcesForHostAuthors#Differentiating
> var commonHost = require('common/commonHost');
>
> if (commonHost.accessorHost === commonHost.accessorHostsEnum.NODE) {
526a532,538
> } else {
> // Browser and Duktape
> function isBuffer(arg) {
> return arg instanceof Buffer;
> }
> exports.isBuffer = isBuffer;
> }
569a582
> if (commonHost.accessorHost === commonHost.accessorHostsEnum.NODE) {
570a584,597
> } else {
> // Browser and Duktape
> exports.inherits = function(ctor, superCtor) {
> ctor.super_ = superCtor;
> ctor.prototype = Object.create(superCtor.prototype, {
> constructor: {
> value: ctor,
> enumerable: false,
> writable: true,
> configurable: true
> }
> });
> };
> }
bash-3.2$

We could leave this or use catch blocks to try the default action and then try the Duktape specific action.

$PTII/ptolemy/actor/lib/jjs/node/util.js

Used by the Cape Code Host

r70707 2014-11-21: "Added util module from node (slightly modified) and a new console module with an API compatible with Node."

The changes between the version in the GitHub repo and r70707 are fairly extensive. Note that after r70707, a number of other changes were also made.

bash-3.2$ wget -O /tmp/util.js https://raw.githubusercontent.com/defunctzombie/node-util/master/util.js
bash-3.2$ svn update -r 70707
Updating '/Users/cxh/ptII/ptolemy/actor/lib/jjs/node/util.js':
U /Users/cxh/ptII/ptolemy/actor/lib/jjs/node/util.js
Updated to revision 70707.
bash-3.2$ diff -rwb /tmp/util.js $PTII/ptolemy/actor/lib/jjs/node/util.js

bash-3.2$ diff -rwb /tmp/util.js $PTII/ptolemy/actor/lib/jjs/node/util.js
21a22,26
> // --- Modified by Edward A. Lee, eal@eecs.berkeley.edu to remove Node.js dependence.
> // Changes made:
> // --- Removed deprecated functions and the deprecate function that deprecates them.
> // FIXME: Test all functions.
>
62,94d66
< // Mark that a method should not be used.
< // Returns a modified function which warns once by default.
< // If --no-deprecation is set, then it is a no-op.
< exports.deprecate = function(fn, msg) {
< // Allow for deprecating things in the process of starting up.
< if (isUndefined(global.process)) {
< return function() {
< return exports.deprecate(fn, msg).apply(this, arguments);
< };
< }
<
< if (process.noDeprecation === true) {
< return fn;
< }
<
< var warned = false;
< function deprecated() {
< if (!warned) {
< if (process.throwDeprecation) {
< throw new Error(msg);
< } else if (process.traceDeprecation) {
< console.trace(msg);
< } else {
< console.error(msg);
< }
< warned = true;
< }
< return fn.apply(this, arguments);
< }
<
< return deprecated;
< };
<

Since this is diverging from util.js, maybe we should call it util-accessor.js or accessor-util.js?

In formatValues(), we have these changes.

242,246c214,233
< // IE doesn't make error fields non-enumerable
< // http://msdn.microsoft.com/en-us/library/ie/dww52sbt(v=vs.94).aspx
< if (isError(value)
< && (keys.indexOf('message') >= 0 || keys.indexOf('description') >= 0)) {
< return formatError(value);
---
> // This could be a boxed primitive (new String(), etc.), check valueOf()
> // NOTE: Avoid calling `valueOf` on `Date` instance because it will return
> // a number which, when object has some additional user-stored `keys`,
> // will be printed out.
> var formatted;
> var raw = value;
> try {
> // the .valueOf() call can fail for a multitude of reasons
> if (!isDate(value))
> raw = value.valueOf();
> } catch (e) {
> // ignore...
> }
>
> if (isString(raw)) {
> // for boxed Strings, we have to remove the 0-n indexed entries,
> // since they just noisey up the output and are redundant
> keys = keys.filter(function(key) {
> return !(key >= 0 && key < raw.length);
> });
263a251,263
> // now check the `raw` value to handle boxed primitives
> if (isString(raw)) {
> formatted = formatPrimitiveNoColor(ctx, raw);
> return ctx.stylize('[String: ' + formatted + ']', 'string');
> }
> if (isNumber(raw)) {
> formatted = formatPrimitiveNoColor(ctx, raw);
> return ctx.stylize('[Number: ' + formatted + ']', 'number');
}
> if (isBoolean(raw)) {
> formatted = formatPrimitiveNoColor(ctx, raw);
> return ctx.stylize('[Boolean: ' + formatted + ']', 'boolean');
> }
295c295,313
< if (keys.length === 0 && (!array || value.length == 0)) {
---
> // Make boxed primitive Strings look like such
> if (isString(raw)) {
> formatted = formatPrimitiveNoColor(ctx, raw);
> base = ' ' + '[String: ' + formatted + ']';
> }
>
> // Make boxed primitive Numbers look like such
> if (isNumber(raw)) {
> formatted = formatPrimitiveNoColor(ctx, raw);
> base = ' ' + '[Number: ' + formatted + ']';
> }
>
> // Make boxed primitive Booleans look like such
> if (isBoolean(raw)) {
> formatted = formatPrimitiveNoColor(ctx, raw);
> base = ' ' + '[Boolean: ' + formatted + ']';
> }
>
> if (keys.length === 0 && (!array || value.length === 0)) {
333c351,355
< if (isNumber(value))
---
> if (isNumber(value)) {
> // Format -0 as '-0'. Strict equality won't distinguish 0 from -0,
> // so instead we use the fact that 1 / -0 < 0 whereas 1 / 0 > 0 .
> if (value === 0 && 1 / value < 0)
> return ctx.stylize('-0', 'number');
334a357
> }
342a366,374
> function formatPrimitiveNoColor(ctx, value) {
> var stylize = ctx.stylize;
> ctx.stylize = stylizeNoColor;
> var str = formatPrimitive(ctx, value);
> ctx.stylize = stylize;
> return str;
> }
>
>
418c450,451
< .replace(/(^"|"$)/g, "'");
---
> .replace(/(^"|"$)/g, "'")
> .replace(/\\\\/g, '\\');
428d460
< var numLinesEst = 0;
430,431d461
< numLinesEst++;
< if (cur.indexOf('\n') >= 0) numLinesEst++;

Not sure about the above changes, we probably should include them

450,453c480
< function isArray(ar) {
< return Array.isArray(ar);
< }
< exports.isArray = isArray;
---
> var isArray = exports.isArray = Array.isArray;
526c553,556
< exports.isBuffer = require('./support/isBuffer');
---
> function isBuffer(arg) {
> return arg instanceof Buffer;
> }
> exports.isBuffer = isBuffer;

Two different ways of defining isArray() and isBuffer(). Either one would work

570c600,610
< exports.inherits = require('inherits');
---
> exports.inherits = function(ctor, superCtor) {
> ctor.super_ = superCtor;
> ctor.prototype = Object.create(superCtor.prototype, {
> constructor: {
> value: ctor,
> enumerable: false,
> writable: true,
> configurable: true
> }
> });
> };
bash-3.2$

Note that Browser and Duktape want ot use the function()

In Node, util is core

It looks to me that when the node host encounters require('util');, then it is loading a core module and not accessing the file system. Thus hosts/node/node_modules/@accessors-modules/util/util.js is not loaded. Maybe we should remove this directory?

bash-3.2$ cd $PTII/org/terraswarm/accessor/accessors/web/{$HOSTS_NODE}
bash-3.2$ head -2 node_modules/util/util.js
error("This is util.js");
// Copyright Joyent, Inc. and other Node contributors.
bash-3.2$ node
> var util = require('util');
var util = require('util');
undefined
>

The same is true of node_modules/events/events.js

bash-3.2$ head -2 node_modules/events/events.js
error("This is events.js");

bash-3.2$ node
> var events=require('events');
var events=require('events');
undefined
>

Maybe these packages are loaded in some other way, but I doubt it.

npm install ws creates options, ultron and ws:

bash-3.2$ cd /tmp
bash-3.2$ mkdir test
bash-3.2$ cd test
bash-3.2$ rm -rf node_modules/
bash-3.2$ npm install ws
/private/tmp/test
|- ws@1.1.0
|- options@0.0.6
|- ultron@1.0.2

WARN enoent ENOENT: no such file or directory, open '/private/tmp/test/package.json'
WARN test No description
WARN test No repository field.
WARN test No README data
WARN test No license field.
bash-3.2$ ls node_modules/
options ultron ws
bash-3.2$

events: From npm install events. Core module, appears in lib, could be deleted.
inherits: From npm install util
options: From npm install ws
ultron: From npm install ws
util: From npm install util. Core module, appears in lib, could be deleted.
webSocketClient: Created by Christopher, from effort to support the WebSocket accessors.

Note that webSocketClient is not a legitmate npm name. npm install webSocketClient says

npm ERR! 404 Your package name is not valid, because
npm ERR! 404 1. name can no longer contain capital letters

https://nodejs.org/api/modules.html#modules_core_modules says:

Core Modules

Node.js has several modules compiled into the binary. These modules are described in greater detail elsewhere in this documentation.

The core modules are defined within Node.js's source and are located in the lib/ folder.

Core modules are always preferentially loaded if their identifier is passed to require(). For instance, require('http') will always return the built in HTTP module, even if there is a file by that name

The lib directory is at https://github.com/nodejs/node/tree/master/lib and includes events.js and util.js

As a test, under Mac OS X, I ran node under dtruss to see if it accessed the node_modules/util/util.js file and it did not.

ealmac23:test root# ls node_modules/
inherits options ultron util ws
ealmac23:test root# sudo dtruss -a -f node

The output is voluminous, but I ran var util=require('util'); and util.js was no loaded.

When I ran var ws=require('ws');, dtruss showed that node_modules/ws/package.json was downloaded:

   6167/0x1083e9:     80265    1408     35 open("/private/tmp/test/node_modules/ws/package.json\0", 0x1000000, 0x0)                = 21 0

Should we remove both the events and util directories?

require()

In Module Specification, details about require() are given.

The CommonJS Modules v1.1 specification states:

"6. The "require" function may have a "paths" attribute, that is a prioritized Array of path Strings, from high to low, of paths to top-level module directories."

"The "paths" property must not exist in "sandbox" (a secured module system)."
"The "paths" attribute must be referentially identical in all modules."
"Replacing the "paths" object with an alternate object may have no effect."
"If the "paths" attribute exists, in-place modification of the contents of "paths" must be reflected by corresponding module search behavior."
"If the "paths" attribute exists, it may not be an exhaustive list of search paths, as the loader may internally look in other locations before or after the mentioned paths."
"If the "paths" attribute exists, it is the loader's prorogative to resolve, normalize, or canonicalize the paths provided."

Google Groups: Kris Kowal wrote:

"Some implementations use a global "require", while others have distinct "require" functions. If you are writing portable code, you cannot guarantee that monkey patching a property on require will be reflected in other modules. If you are writing an implementation, you can encourage folks to write portable code by guaranteeing a unique "require" function in every module."

""require.paths" was always stop-gap today is an anti-pattern. Narwhal discouraged its use at run-time and would initialize it using a topological sort of the transitive dependencies. Node.js does not have it at all, and goes on to mandate that top-level identifiers must reach into dependencies, requiring relative identifiers for modules inside the same package."

"If you do support "require.paths", I recommend copying it as a property of each module�s "require" function. That will satisfy all of the requirements of the spec. That is, replacing "require.paths" on one module�s "require" will have no effect, but changes to the content of any "require.paths" will, as in "require.paths.push(path)"."

Sam L'ecuyer wrote:
"1. The "paths" attribute must be referentially identical in all modules."

"I would read that to mean that if require.paths exists, all updates to the path list will be available to all modules in the runtime. The implementations I've seen (and written myself) wind up wrapping the module body in a function declaration with a parameter called "require" and invoking it."

Node

Node.js modules documentation
Node.js 0.4.3: Old versions of Node.js had access to require.paths
- nodeJS require.paths resolve problem (Stackoverflow)

Node's module algorithm is clearly documented. If the module is not a core module (like http) and does not start with ./ or / or ../, then the module is searched for in the node_modules/ directory. If the module is not found, then ../node_modules is checked.

Node can read a NODE_PATH environment variable, but this is deprecated.

Interestingly, when node.js sees require('util'), it loads in a core module, not web/hosts/node/node_modules/@accessors-modules/util/util.js. To prove this, edit web/hosts/node/node_modules/@accessors-modules/util/util.js and insert garbage text, then run node nodeHostShell.js

Solution: web/hosts/node/node_modules/@accessors-modules/util/ and events/ were removed.

Possible solution

common/commonHost.js should be modified so that

var util = require('util');

becomes

try {
// The node host will load the core util module here and not access the file system.
var util = require('util');
} catch (error) {
// The duktape host will load this module.
var util = require('../common/modules/util/util.js');
}

Final Solution

In commonHost.js, we defined accessorHost, which is an enum that names the type of Accessor Host. See Differentiating the hosts

In commonHost.js, we do:

if (accessorHost === accessorHostsEnum.DUKTAPE) {
var util = require('../common/modules/util.js');
var EventEmitter = require('../common/modules/events.js').EventEmitter;
} else {
// The node host will load the core util module here and not access the file system.
var util = require('util');
var EventEmitter = require('events').EventEmitter;
}

and check in copies of util.js and events.js.

FIXME: We have multiple util.js files and need to combine them.

See Also

Module Specification: Information about how to use require()
npm: Page of links to wiki pages about npm
NPM: Notes about npm, including possible moves.

Edit - History - Print - Recent Changes - Search

Page last modified on May 01, 2017, at 07:30 pm