• View
• Edit
• History
• Print

GDPJSNotes

This page contains old and obsolete notes about the GDP JavaScript Interface.

See GDP JS for more recent information.

Overview

This page is about using a C-based, Node.js interface between the GDP and JavaScript.

An alternative would be to write a GDP JavaScript Interface that uses the GDP wire protocol.

The C-based code is in gdp/lang/js

• gdp/lang/js/apps
  • reader-test.js A partial port of the C-based reader-test example
  • simple_reader_01.js A simple reader that does not use the gdpjs code. See Invoke simple_reader_V01.js.txt
  • writer-test.js A partial port of the C-based writer-test example
• gdp/lang/js/gdpjs/ C code that provides an easier to interface-to set of functions for C code that is cpp macros
• gdp/lang/js/libs/ The libgdpjs shared library used by reader-test.js and writer-test.js
• gdp/lang/js/test/ Node mocha tests, see Testing
• gdp/lang/js/tests/ Older script-based tests.

See Also

• How to Use the GDP Node.js/JavaScript RESTful Interface
• Swig And The GDP
• Node.js
  • Node.js
  • Node.js Accessors
  • Node-FFI-Tutorial
  • Node ref documentation

Notes about getting it to work

Get access to the gdp repo

Ask Eric Allman for access to the gdp repo.

Build the gdp

If make does not work, then read the gdp README file. In particular, on the Mac, you may need install Mac Ports and do

Start up gdp_router and gdplogd

To run the gdp_router.py and gdplogd on your local machine, while in the gdp/ directory, invoke test/setupAndRun.sh, let it run and then put it in the background. FIXME: The issue here is that the script may prompt the user for their git credentials to update the router directory.

Optionally, check that the gdp processes are running:

Install Node

Download from http://nodejs.org/ . "Node.js® is a platform built on Chrome's JavaScript runtime [V8] for easily building...."

Mac package node 6.0.2 pkg installs into:

• /usr/local/bin/node // Node.js executable
• /usr/local/bin/npm // Node package manager executable (implemented as a JS script - "#!/usr/bin/env node")
• /usr/local/include/node/ // Also contains vc8.h which might be useful
• /usr/local/lib/node_modules/ // Location for npm's "global" modules
• /usr/local/share/man/man1/node.1 // Meager man page

and who knows what other locations. Good luck with uninstallation.

Node.js docs at http://nodejs.org/api/

Node package manager, npm, docs at https://www.npmjs.org/doc/ .

Install the node packages

Running (cd gdp/lang/js/; make) does the npm installs for you.

Below is some possibly out of date documentation about what packages are installed by running make.

This creates node_modules in the current directory (which should be gdp/ as the JavaScript below expects to find libraries in libs/)

Create a log

If gdplogd is running, locally, then use:

See also Create A Remote Log

Invoke writer-test

cd to the gdp/lang/js/apps directory

Append a record to the log:

Invoke reader-test

Read from the log:

Berkeley GDP Servers

Below are instructions that should work without running local gdplogd and gdp_router processes.

Create A Remote Log

To create a log on a remote server, try

We use

  export LOGNAME=edu.berkeley.eecs.$logname.`date +d.M.%S`

so that a new log is created on the server for the following steps

Append a record to the log:

Read from the log:

Testing

We use mocha to test

Error messages and solutions

Configuration

See gdp/README.md for how libgdp looks for configuration information.

In summary, what happens is that the following path is searched:

  .ep_adm_params:~/.ep_adm_params:/usr/local/etc/ep_adm_params:/etc/ep_adm_params

For example, if .ep_adm_params/gdp is found, then the values of that file are used to configure the gdp. If that file contains

  swarm.gdp.routers=gdp-01.eecs.berkeley.edu; gdp-02.eecs.berkeley.edu; gdp-03.eecs.berkeley.edu

Then each host is queried (in order) to see if it is a gdp router. If it is not, then the next host is checked. This parameter is consulted only if zero-conf fails.

Invoke simple_reader_V01.js.txt

You are better off using reader-test.js, simple_reader_v01.js is a bit of a mess.

Edit gdp/lang/js/apps/simple_reader_v01.js and update var gclname_arg to match the output of gcl-create above:

In the gdp/lang/js/apps/ directory, invoke

Compare this with reader-test

Debug Not Working

In simple_reader_01.js, debugging was being set with:

Analysis showed that in gdp/ep/ep_dbg.c, FlagList was being set in ep_dbg_setto()

But in ep_dbg_flaglevel(), FlagList was 0!

The problem was that when libgdp was being created, we were linking with the static libep.a library.

To see this, edit gdp/gdp/Makefile and add -Wl,-t to the line that creates the shared library.

Rebuilding the shared library will show:

This is because under Mac OS X, ../ep contains libep.a and libep.so.3.0. However the linker is looking for libep.dylib, does not find it and uses libep.a instead. Probably in static libraries, the visibility of the static FlagList is somehow different.

The solution is to edit gdp/gdp/Makefile and add -L${LOCALLIBSDIR} \ so that .dylib files in gdp/gdp/libs are found and used.

Debug File

As a small example, we try to set the debug file in C and then in JavaScript

Debug File C Code

Here, in epTest.c calls ep_dbg_setfile() and ep_dbg_getfile() returns the value as expected

For details about why we need to do the cp ..., see Darwin: shared libraries have the wrong internal name

Debug File JavaScript Code

To get access to these methods, we have some edits to make:

• $PTII/node_modules/@terraswarm/gdp/gdpjs/gdpjs.js: add declarations for ep_dbg_getfile() and ep_dbg_setfile() inside @@
  var libep = ffi.Library(GDP_DIR + '/libs/libep.3.0', {
  ...
      'ep_dbg_getfile': ['pointer', []],
      'ep_dbg_setfile': ['void', ['pointer']],

  [$[Get Code]]
• $PTII/node_modules/@terraswarm/gdp/index.js: add:
  exports.ep_dbg_getfile = gdpjsSupport.ep_dbg_getfile;
  exports.ep_dbg_setfile = gdpjsSupport.ep_dbg_setfile;

  [$[Get Code]]
• In $PTII/vendors/gdp/gdp/test:
  bash-3.2$ cat epTest.js
  var gdpjs = require('@terraswarm/gdp');
  
  gdpjs.ep_dbg_setfile(new Buffer(42));
  
  var f = gdpjs.ep_dbg_getfile();
  
  console.log("f: " + f);
  console.log(f);
  bash-3.2$ node epTest.js
  f:
  <Buffer@0x104018df8 >
  bash-3.2$

  [$[Get Code]]

We get the buffer back?

Debug Analysis

In ep_dbg.c: DebugFile is declared as a file static variable:

The library that is loaded is $PTII/node_modules/@terraswarm/gdp/libs/libep.3.0.dylib. To see this, move that file to a temporary location and run.

The nm man page says: "S (symbol in a section other than those above)"

Using -m gives us more information:

Another possibility is that the libgdp shared library has libep statically linked to it, but when we call the ep_dbg_ methods from the libgdp functions, we are getting different library and symbol.

Debug Solution

The libgdp shared library is linked statically to libep.a, so instead of looking for the ep_* symbols in libep, we look in libgdp. If we get the ep_* symbols from the libep shared library, then they will be different than the ep_* symbols that are used by calls in the libgdp shared library, which means that debugging will appear not to work.

So, in gdpjs.js:

Segfault

Under Mac OS X, the gdb binary is named ggdb and is a PITA to install. After installation, running ggdb node and then:

The problem here is that libs/libgdp.dylib can't find libep.so.3.0

We used to be able to use DYLD_LIBRARY_PATH, but that does not seem to help:

The solution is to use the Mac OS X install_name_tool program:

after install_name_tool is run, then ggdb node, r simple_reader_01.js will find the shared library.

The following change to gdp/Makefile will run @@install_name_tool under Mac OS X:

GDB Stacktrace has no line numbers

Under Mac OS X, in ggdb node, r simple_reader_01.js does not print any line numbers in the stack trace:

The workaround is to update gdp/gdp/Makefile and use -ggdb instead of -g.

This gets a bit more information:

These changes:

1. Updating gdp/ep/Makefile and using -ggdb instead of -b
2. Updating gdp/gdp/Makefile and gdp/ep/Makefile and commenting out the O= line
   #O=             -O  

   [$[Get Code]]
3. Running
   (cd gdp/ep; make clean; make; cd ../gdp; make clean; make)

   [$[Get Code]]
4. Give us better results:
   rogram received signal SIGABRT, Aborted.
   0x00007fff92d20f06 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
   (gdb) where
   #0 0x00007fff92d20f06 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
   #1 0x00007fff954664ec in pthread_kill () from /usr/lib/system/libsystem_pthread.dylib
   #2 0x00007fff9130a6e7 in abort () from /usr/lib/system/libsystem_c.dylib
   #3 0x0000000101fc6500 in ep_assert_failure (type=0x10319d920 "require",
   file=0x10319e6e7 "gdp_gcl_ops.c", line=646, msg=0x10319d8a0 "%s") at ep_assert.c:88
   #4 0x000000010318f96a in ?? () from /Users/cxh/ptII/vendors/gdp/gdp/gdp/libgdp.so.0.5
   #5 0x00007fff5fbfe940 in ?? ()
   #6 0x00001355191a9a40 in ?? ()
   #7 0x0000000102804000 in ?? ()
   #8 0x00000000a40c040d in ?? ()
   #9 0x0000000103305e20 in ?? ()
   #10 0x00000001033071e0 in ?? ()
   #11 0x0000000000000000 in ?? ()
   (gdb)
   

   [$[Get Code]]
5. In the above, we can see that line 646 in gdp_gcl_ops.c is throwing an assertion, but we might as well recompile the GDP JavaScript interface without optimization and with -ggdb
6. Editing gdp/lang/js/Makefile and commenting out the O= line and using -ggdb instead of -g, then make clean; make yields:
   (gdb) where
   #0 0x00007fff92d20f06 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
   #1 0x00007fff954664ec in pthread_kill () from /usr/lib/system/libsystem_pthread.dylib
   #2 0x00007fff9130a6e7 in abort () from /usr/lib/system/libsystem_c.dylib
   #3 0x0000000101fc6500 in ep_assert_failure (type=0x10319d920 "require",
   file=0x10319e6e7 "gdp_gcl_ops.c", line=646, msg=0x10319d8a0 "%s") at ep_assert.c:88
   #4 0x000000010318f96a in ?? () from /Users/cxh/ptII/vendors/gdp/gdp/gdp/libgdp.so.0.5
   #5 0x00007fff5fbfe940 in ?? ()
   #6 0x00000d743b1a9a40 in ?? ()
   #7 0x0000000102804000 in ?? ()
   #8 0x00000000a40c040d in ?? ()
   #9 0x0000000103305e80 in ?? ()
   #10 0x0000000103201ab0 in ?? ()
   #11 0x0000000103803da0 in ?? ()
   #12 0x000000010283ee98 in ?? ()
   #13 0x00007fff5fbfe980 in ?? ()
   #14 0x0000000103189172 in gdp_gcl_read (gcl=0x103803da0, recno=4347402928,
   datum=0x103305e80) at gdp_api.c:456
   Backtrace stopped: frame did not save the PC
   (gdb)
   

   [$[Get Code]]

GDP_ASSERT_GOOD_GCL assertion

In gdp_gcl_ops.c, line 646 is marked:

Summary about how to reproduce

1. Edit gdp/ep/Makefile, gdp/gdp/Makefile and gdp/lang/js/Makefile and make these changes
   1. Comment out the O= line:
      #O=     -O

      [$[Get Code]]
   2. Replace -g with -ggdb:
      COPTS=  ${PG} -ggdbçç ${WALL} $O ${STD}

      [$[Get Code]]
2. In gdp/gdp/Makefile, add lines to run install_name_tool:
   libgdp.so.${GDPLIBVER}: ${OBJS}
           ${CC} ${SHARED} -o $@ ${OBJS} ${LDFLAGS}
           if [ `uname` = 'Darwin' ]; then \
                   echo "Mac OS X: Running install_name_tool to add full path to libep"; \
                   install_name_tool -change libep.so.3.0 `(cd ../ep; pwd)`/libep.so.3.0 $@; \
                   echo "otool -L should list the full path to libep:"; \
                   otool -L $@; \
           fi

   [$[Get Code]]
3. Build:
   cd gdp; make

   [$[Get Code]]
4. Create a log, note the name
   bash-3.2$ ./apps/gcl-create -k none -s ealmac23.local myLog3
   Created new GCL cHL47Jlv6t9sc74PjV1Gist_PcTT1HnibulAXA8OocI
           on log server ealmac23.local
   exiting with status OK
   bash-3.2$

   [$[Get Code]]
5. Build the GDP JavaScript interface:
   cd lang/js; make
   

   [$[Get Code]]
6. cd to gdp/lang/js/apps and edit simple_reader_01.js and update gclname_arg with the value from the gcl-create command above:
   var gclname_arg = "cHL47Jlv6t9sc74PjV1Gist_PcTT1HnibulAXA8OocI"

   [$[Get Code]]
7. Start up gdb (See How to install gdb debugger under Mac OS X):
   ggdb node
   

   [$[Get Code]]
8. Run the script:
   r simple_reader_01.js
   

   [$[Get Code]]
9. It might take a few runs to get a stack trace, but eventually running where will show:
   (gdb) where
   #0 0x00007fff92d20f06 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
   #1 0x00007fff954664ec in pthread_kill () from /usr/lib/system/libsystem_pthread.dylib
   #2 0x00007fff9130a6e7 in abort () from /usr/lib/system/libsystem_c.dylib
   #3 0x0000000101fc6500 in ep_assert_failure (type=0x10329d920 "require",
   file=0x10329e6e7 "gdp_gcl_ops.c", line=646, msg=0x10329d8a0 "%s") at ep_assert.c:88
   #4 0x000000010328f96a in ?? () from /Users/cxh/ptII/vendors/gdp/gdp/gdp/libgdp.so.0.5
   #5 0x00007fff5fbfe940 in ?? ()
   #6 0x00002b4d331ab000 in ?? ()
   #7 0x0000000102804000 in ?? ()
   #8 0x00000000a40c040d in ?? ()
   #9 0x0000000103306310 in ?? ()
   #10 0x0000000103003cd0 in ?? ()
   #11 0x0000000000000000 in ?? ()
   (gdb)
   

   [$[Get Code]]
10. In gdp_gcl_ops.c, line 646 is marked:

Solution

It turns out that the problem was that the declaration for gdp_gcl_open was incorrect.

The symptoms were that if, in gdp/gdp_api.c, gdp_gcl_open() was modified to print the pointer that was being assigned:

Then in simple_reader_01.js, calling deref() on gclPtrPtr was returning a different value

The solution was to declare the third argument as a void *, not a void:

Debugging

Debugging gdpd

Use

Error: "Dynamic Symbol Retrieval Error"

The first problem here is that gdp.js had

 var libgdp = ffi.Library('./libgdp.1.0', {
  'gdp_init': [ 'ulong', [ 'string' ] ],
  'log_view_ls': [ 'int', [ ] ]
 })

The problem is with the ./. The fix is to do run node gdp.js in gdp/libs after running make

The second problem is that log_view_ls is not in the current version of gdp.

Using find, I see that log_view_ls only appears in the gdp.js files that I'm working on:

The solution is to turn on debugging. Change gdp.js so that we define the ep_dbg_set function and don't define log_view_js

Error: running gdp_init("2468") loops endlessly

Now that we have debugging turned on we see that we get a loop:

The key line is

  ^[[0m^[[33mgdp_init: contacting host 2468 port (null)

host 2468 makes no sense.

gdp_proto.c has

BTW - curly brackets would really help the readability and maintainability of this code.

So, try 127.0.0.1:2468

Results:

The current gdp.js:

4. writer-test.js can write a GCL item

writer-test.js and reader-test.js work and have a test script

gdp/lang/js/apps/{ writer-test.js and reader-test.js } are working like gdp/apps/{ writer-test.c and reader-test.c } and gdp/lang/js/(Makefile test) will run a bash testing script in js/tests/.

update gdp inside ptII: Pull gdp into ptII/org/terraswarm/gdp/

Re-Make ptII/org/terraswarm/gdp/src/gdp :

Make gdp/lang/js/ -- makes C support library, gdpjs/libgdpjs.1.0.dylib, for JS support routines, gdpjs/gdpjs_supt.js .

Note, gdp/lang/js/Makefile is not yet driven by gdp/Makefile.

Finally, run the test script from lang/js/ using the "make test" target in the js/Makefile. Results of test script go to stdout.

Note, there's a false positive hit shown by the diff: ">>>>>>>> Checking reader-test.js -s output" when we're running a reader "-s" process that's subscribed to a test gcl that is concurrently being appended to by a writer "-a" process. The diff seems to arise because of differing order of test output depending on whether the destination is a Terminal or a file (like we are doing in this scripted test situation. We captured the corresponding known good output file manually using a Terminal invoked reader -s. This seems to still happen though the JS code for both reader and writer is explicitly calling fflush() after writes. This issue is noted in the README and in the test script. It's a testing artifact (the subscribe and the appends do work) that I've got on my TBD list.

Node.js JavaScript Testing Frameworks

Found some possible Node-compatible (server-side -- non-browser-based) test frameworks for JS. Will be looking for the best developed / widely used one.

Review of Mocha & Jasmine -- Mocha newer but most Pros

Mocha-Related Information:

Looks like Mocha is a place to start for Node.js JavaScript testing frameworks.

Subscription

As of 8/17/16, Subscription using Cape Code works, see $PTII/ptolemy/actor/lib/jjs/modules/gdp/test/auto/GDPLogSubscribe.xml

However, Subscription using the Node Accessor host does not work.

Subscription Overview

Cape Code's GDPHelper.subscribe() spawns a thread to do the dirty work. It looks like the C version of the gdp subscriber in $PTII/vendors/gdp/gdp/apps/gdp-reader.c does something similar. In the Node host, we should be using callbacks. Whether or not we have register a callback function, we are not receiving events.

Run the Node Accessor Host Subscription Composite Accessor

Run without Subscription

To test that the code other than GDPLogSubscribe works, we run a Node Composite Accessor that does not subscribe: ((cd $PTII/org/terraswarm/accessor/accessors/web/gdp/test/auto; node ../../../hosts/node/nodeHostInvoke.js -timeout 6000 gdp/test/auto/GDPLogCreateAppendReadJS)

The end of the output should look like:

The Ptolemy model for the above Composite Accessor is at $PTII/ptolemy/actor/lib/jjs/modules/gdp/test/auto/GDPLogCreateAppendReadJS.xml

Run With Subscription

Next, run the Node Composite Accessor that does subscription: (cd $PTII/org/terraswarm/accessor/accessors/web/gdp/test/auto; node ../../../hosts/node/nodeHostInvoke.js -timeout 6000 gdp/test/auto/GDPLogSubscribeJS)

The model hangs, below is an analysis of the end of the run output.

The above is good, it means that getNextData() was called.

In the above, the call to gdp_gcl_open() is occurring. A log is not being created because it already exists.

In the above, debugging is not being enabled. There is a bug with setting debugging in the GDP/JS interface, see above.

In the above, we have entered the subscribe() method in gdp.js. The problem here is that we don't return from read_gcl_records()

In gdp.js, the subscribe() function looks like:

Continuing with the output:

In the above, read_gcl_records() is checking for the handle, which is fine

In the above, we call gdp_event_next_js() with a 5 second timeout. The output stops for 5 seconds.

Here, I would expect that we would get an event from the append, but we are probably not in a separate thread.

In any case, we don't want to be in a separate thread here, we want to use callbacks.

We did not get an event, so we are not updating recarray_out.

I'm not sure about the above.

We returned from read_gcl_records()

BTW - gdp.js contains this:

In our hack of calling read_gcl_records() and returning the data, perhaps getNextData could retrieve the next data?

However, what we really want is to use callbacks.

Continuing with the output:

In the above, the Test accessor fails because the input is "null" instead of "datum was null"

Subscription Files

A review of files is in order:

$PTII/ptolemy/actor/lib/jjs/modules/gdp/test/auto/GDPLogSubscribe.xml

The Cape Code Model, which runs in vergil

$PTII/org/terraswarm/accessor/accessors/web/gdp/test/auto/GDPLogSubscribeJS.js

Code generated from GDPLogSubscribe.xml. To run it, use: (cd /Users/cxh/ptII/org/terraswarm/accessor/accessors/web/gdp/test/auto; node ../../../hosts/node/nodeHostInvoke.js -timeout 6000 gdp/test/auto/GDPLogSubscribeJS)

$PTII/org/terraswarm/accessor/accessors/web/gdp/GDPLogSubscribe.js

The accessor definition shared between the Cape Code and Node accessor hosts.

$PTII/org/terraswarm/accessor/accessors/web/hosts/node/node_modules/@accessors-modules/gdp/gdp.js

The gdp module that is found by the Node Accessor Host.

$PTII/node_modules/@terraswarm/gdp/gdpjs/gdpjs.js

The version of gdpjs.js that is found by Node when Node reads var gdpjs = require('@terraswarm/gdp'); in gdp.js. Important: gdpjs must be installed with:

cd $PTII
npm install vendors/gdp/gdp/lang/js

[$[Get Code]]

$PTII/vendors/gdp/gdp/lang/js/gdpjs/gdpjs.js

The master copy of gdpjs.js that is read by npm install .... For development and debugging, edit $PTII/node_modules/@terraswarm/gdp/gdpjs/gdpjs.js, and then before running npm install ..., update the master copy with

cp $PTII/node_modules/@terraswarm/gdp/gdpjs/gdpjs.js $PTII/vendors/gdp/gdp/lang/js/gdpjs/gdpjs.js

[$[Get Code]]

$PTII/node_modules/@terraswarm/gdp/libs

Shared libraries loaded by gdpjs.js. libgdpjs contains some glue code for the interface.

Subscription Analysis

See GDP JavaScript Callbacks -> Subscription Analysis?.

Subscription Callbacks

See GDP JavaScript Callbacks -> Subscription Callbacks?.

Calling the C-based subscribe() function

It could be that we need to call the C-based subscribe() function.

gdpjs.js:

There are two functions defined in gdpjs.js that call gdp_gcl_subscribe() and one that calls gdp_gcl_multiread():

do_multiread() calls the last one:

Subscription as a separate process

Interestingly, running a Node Host Composite Accessor that appends to a log in one process and a Node Host Composite Accessor that subscribes to a log works somewhat.

1. Edit org/terraswarm/accessor/accessors/web/gdp/test/auto/GDPLogCreateAppendReadJS.js and modify JavaScriptGDPLogName so that it does not call Math.random(). Instead, have it create a log with a fixed name, such as ptolemy.actor.lib.jjs.modules.gdp.test.auto.GDPLogSubscribeJS.yourlogin7.
2. Edit org/terraswarm/accessor/accessors/web/gdp/test/GDPLogSubscribeOnlyJS.js and set the logName parameter to the name of the log set above.
3. Invoke the Append Composite Accessor: (cd $PTII/org/terraswarm/accessor/accessors/web/gdp/test/auto; node ../../../hosts/node/nodeHostInvoke.js -timeout 600000 gdp/test/auto/GDPLogCreateAppendReadJS) >& /tmp/append7.txt &
4. Invoke the Subscribe Composite Accessor: (cd $PTII/org/terraswarm/accessor/accessors/web/gdp/test/auto; node ../../../hosts/node/nodeHostInvoke.js -timeout 600000 gdp/test/GDPLogSubscribeOnlyJS)
5. The Subscribe Composite Accessor will print out the values that are appended by the Append Composite Accessor.

Problems:

• The append and subscribe accessors are not in the same accessor.
• The output is being printed, not sent to the output port

Getting the Next Data

The issue here is that GDPLogSubscribe.js calls log.subscribe(), which could return some data, and then loops and calls log.getNextData():

The problem is that log.getNextData() is unimplemented. log.subscribe() will get some data, but then we need to get data forever from log.getNextData().