• View
• Edit
• History
• Print

Testing

This page describes the goals of the Accessor regression testing framework, the testing status of different hosts, and philosophies for creating test doubles for outside resources (physical things or web services).

Regression Test Summary

To run all the tests, run

To see what tests are available:

Note that each of the accessor hosts has a separate ant target. So, to run the tests for just one target:

A good policy is to run the tests before you do any work, then make your changes, then run the tests again.

Test Results

Jenkins

• accessors Jenkins site
  • Code Coverage (Notes)
• ptIIci Jenkins site - runs the Cape Code tests
• CapeCodeInstaller - Created a Cape Code Host installer and runs the tests.
• Library Status (Notes) (Uses the output of the accessor Jenkins page and the ptII Jenkins page.

Coverity

• See Coverity Scan for static analysis indicating possible errors.
• The $PTII/adm/bin/coverityAccessors script is invoked as part of the ptII Jenkins build. In particular, $PTII/adm/bin/Coverity.tcl invokes make coverity in $PTII/adm/gen-11.0. The makefile has this rule:
  coverity:
          (cd $(PTIIHOME); adm/bin/coverity)
          (cd $(PTIIHOME); adm/bin/coverityAccessors)

  [$[Get Code]]

Note that Coverity reports EXPLICIT_THIS_EXPECTED because the setTimeout implementation for the Duktape Host uses this. However, we can't seem to exclude this using the Coverity exclusion settings. See To Do List -> General

Ports

When writing network demos and tests, avoid using port 8080 because on the nightly build machine, that port on tis used by the Jenkins web server. Instead, use 8089, or 8091 and above. It is best if Cape Code demos avoid 8080 because one of the tests exports HTML, which by default runs the model.

To see what ports are occupied on terra, use nmap:

In the above, the ports 8080, 8088 and 8089 have servers running on them and should probably not be used in demos.

Directory Structure

• accessors/web/test/ contains test accessors
  • accessors/web/test/auto/ contains test Composite Accessors that should be run by each of the accessor hosts
• Each of the accessors should contain a test/auto directory that contains Composite Accessors for that accessor.
  • See Cape Code Host -> Code Generation for directory information.

Goals

• When possible, we use Composite Accessors for testing. The reason is because if an accessor host implements a module, then it should be able to execute an CompositeAccessor
  • These Composite Accessors should probably reside in test/auto/ directories in the Accessor declaration directories. For example accessors/web/net/test/auto/WebSocketClientJS.js
• There are generic Composite Accessor tests located in accessors/web/test/auto. All accessor hosts should be able to run these.
• Functionality that does not require a module can be implemented as JavaScript that are not accessors.
  • A simple Ramp function that compares inputs with known good inputs is such an actor
  • However, accessor/web/test/TrainableTest.js is an accessor because it is complex.
  • Composite Accessors can include all the code for facilities like the Test function
  • It might be nice if we had one common shared location for facilities like the Test function
  • We would like to be able to use the Coverity Scan static analyzer on this JavaScript. If the functionality is in .js files, then it is easy. If the functionality is in MoML files, then we need to have a step that exports the functionality to .js files.
• It should be possible to run the tests using ant from the accessors/web/ directory.
  • Cape Code tests are run from the ptII tree.
    • In the accessors repo, to run just the Cape Code tests, use ant tests.capecode
    • To run a test by hand using the Nashorn Host, use
        cd $PTII/org/terraswarm/accessor/accessors/web/hosts/nashorn
        ./nashornAccessorHost test/auto/RampJSTestDisplay

      [$[Get Code]]
  • Browser host ant tests.browser invokes the tests on the Browser Host.
  • We have a prototype of this for duktape and node
• The output should be in the JUnit format so that we can view the results in Jenkins
  • Browser: already does this
  • Cape Code: already does this for Ptolemy models. Could easily do it for Composite Accessors
  • Duktape: ToDo: Does not generate JUnit output.
  • Node: already does this

Browser

• Browser Host
• https://ptolemy.berkeley.edu/accessors/hosts/browser/test/index.html - web pages with tests
• Regression Testing is available.

Browser Host To Do

• Seems to be working

Cape Code

• Cape Code Host
• Cape Code uses Ptolemy II. To test the accessors, we create Ptolemy II models that are placed in test/auto/ directories. When the nightly build runs, an ant rules in $PTII/build.xml invoke the make, which run the models.
• CapeCode has the following auto/ directories:
  bash-3.2$ find $PTII/ptolemy/actor/lib/jjs -name auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/audio/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/contextAware/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/discovery/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/eventbus/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/gdp/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/httpClient/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/mapManager/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/objectAssign/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/serial/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/shell/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/socket/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/udpSocket/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/webSocket/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/modules/xbee/test/auto
  /Users/cxh/ptII/ptolemy/actor/lib/jjs/test/auto
  bash-3.2$ find $PTII/org/terraswarm/accessor -name auto
  /Users/cxh/ptII/org/terraswarm/accessor/test/auto
  bash-3.2$
  

  [$[Get Code]]
  • Most of the Cape Code test models are Ptolemy models that include Ptolemy actors that are neither JavaScript nor JSAccessor actors, so these models are not composite accessors that can be run in other hosts
  • See Cape Code Host -> Code Generation for how to create a composite accessor for testing with Node and where to put it.
• $PTII/ptolemy/cg/kernel/generic/accessor/test/auto contains Ptolemy models can be used to create composite accessors

To run the Cape Code tests:

• ptIIci Jenkins site - runs the Cape Code tests

Cape Code Host To Do

• It would probably be helpful to use Cape Code to create Ptolemy models from which we can generate accessors.
  • The accessors should probably be placed in the subdirectory of the accessor declaration in the accessors repo. In this way, we can build up a set of acceptance tests.
    • The different hosts should be able to run these tests automatically. The directory could be named auto/
      • Question: Should we include the Cape Code model in the auto/ directory in the accessors repo?
        • Should we include the JUnit infrastructure also in the accessor repo?
      • Or, should the Cape Code model be in the $PTII/ptolemy/cg/...?
        • If it is in cg, we should probably mimic the accessor layout

Duktape Host

• Duktape Host
  • accessors/web/hosts/duktape/test/ contains .js files that can be invoked from the command line or with (cd accessors/web; ant test.duktape)

Duktape Host To Do

• Convert the output into JUnit compatible output

Node Host

• Node Host
• Node Host Testing
• web/hosts/node/test/ - A couple of .js tests
  • web/hosts/node/test/mocha/ - Mocha tests that can be run using (cd accessors/web; ant tests.mocha).
    • These tests are run as part of the nightly build and generate JUnit-compatible output in http://terra.eecs.berkeley.edu:8080/job/ptII/lastCompletedBuild/testReport/%28root%29/

Node Host To Do

• See Todo List -> Node

Node Host Code Coverage

http://terra.eecs.berkeley.edu:8080/job/accessors/Code_Coverage/ displays the code coverage of the Node Host tests.

To generate the coverage data and view it on your machine:

How we generate code coverage data for the Node Host is that we use the Istanbul nyc node module.

One tricky thing is that as we want code coverage for the packages that we wrote that are in node_modules, we need to tell Istanbul nyc to not exclude node_modules. Unfortunately, Istanbul nyc seems slightly broken, but if we add --exclude='!**/node_modules/**' to the command line arguments, the we can get coverage for the node_modules directories. Unfortunately, we also get coverage for subdirectories that we don't really care about.

Test Doubles

Gerard Meszaros and Martin Fowler define a test double as "the generic term for any kind of pretend object used in place of a real object for testing purposes". Three relevant kinds of test doubles are:

• Fake objects "actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example)." A fake is used in place of an accessor, simulating the behavior of a resource without requiring a resource API. Fake and are useful if the resource's API is still changing or unknown.
• Mocks are "...objects pre-programmed with expectations which form a specification of the calls they are expected to receive." Mocks are used in conjunction with an existing accessor. The mock object intercepts API calls to the resource and returns responses.
• Stubs "provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'." Stubs provide record / playback testing. They may be unsuitable for stateful resources.

Actors That Support Test Doubles.

• Camera
• SerialPort

See Also

• Accessor Demo Policy - demo naming and placement conventions
• Directory Structure - description of the directory structure
• Software Configuration Management - lots of Accessor links
• To Do List