• View
• Edit
• History
• Print

Regression Testing in the Browser Host

Overview

The browser host offers automated regression testing.

The process is as follows:

• A node.js test server is started to serve accessors.
• A node.js script invokes selenium-webdriver.
• selenium-webdriver opens a browser window.
• For each composite test accessor in accessors/web/test/auto and accessors/web/net/test/auto, a test page is generated.
• selenium-webdriver simulates a click on the 'react' button if present.
• The test is observed for a timeout period. If no error occurs, the test passes.
• For each Mocha test file in hosts/browser/test/auto/mocha, a test page is loaded.
• selenium-webdriver simulates a click on the 'react' button.
• The Test accessor loads and executes a Mocha test file, which may instantiate other accessors.
• All results are saved to the file /accessors/web/reports/junit/browserTestResults.xml

Setup

These instructions assume that node.js is already installed.

Install the selenium-webdriver package for node:

   npm install -g selenium-webdriver

Firefox is supported by default. Firefox versions 47+ are NOT supported. Additional drivers are required for other browsers; please see the selenium-webdriver page.

Execution

Run using Node

Open a command prompt, navigate to /accessors/web/hosts/browser/test , and enter:

   node regressionTestScript.js

Test results are written to /accessors/web/reports/junit/browserTestResults.xml .
Output is also echoed to the console window:

   <?xml version="1.0" encoding="UTF-8"?>
   <testsuites>
   <testsuite name="BrowserHost" tests="7" failed="3">
   <testsuite name="Composite Accessor Tests">
   <testcase name="test/auto/RampJSDisplay.js>
   </testcase>
   <testcase name="test/auto/RampJSTest.js>
   </testcase>
   <testcase name="test/auto/RampJSTestDisplay.js>
   </testcase>
   <testcase name="test/auto/TestComposite.js>
   </testcase>
   <testcase name="net/test/auto/WebSocketClient2JS.js>
   <failure message="Error from accessor at net/test/auto/WebSocketClient2JS.js: TypeError: WebSocket.supportedReceiveTypes is not a function"/>
   </testcase>
   <testcase name="net/test/auto/WebSocketClient3JS.js>
   <failure message="Error from accessor at net/test/auto/WebSocketClient3JS.js: TypeError: WebSocket.supportedReceiveTypes is not a function"/>
   </testcase>
   <testcase name="net/test/auto/WebSocketClientJS.js>
   <failure message="Error from accessor at net/test/auto/WebSocketClientJS.js: TypeError: WebSocket.supportedReceiveTypes is not a function"/>
   </testcase>
   </testsuite>
   </testsuite>
   </testsuites>

The script will automatically search for an open port, starting at 8089 and incrementing. If a port is occupied, you will see an error message echoed to the console, then the script will try the next port. For example, if ports 8089 and 8090 are occupied:

   Starting server on port 8089.
   { [Error: listen EADDRINUSE :::8089]
     code: 'EADDRINUSE',
     errno: 'EADDRINUSE',
     syscall: 'listen',
     address: '::',
     port: 8089 }
   Starting server on port 8090.
   { [Error: listen EADDRINUSE :::8090]
     code: 'EADDRINUSE',
     errno: 'EADDRINUSE',
     syscall: 'listen',
     address: '::',
     port: 8090 }
   Starting server on port 8091.
   Server listening.

Run using ant

To run using ant

1. If necessary, start up X11
2. Run:
   cd accesors/web
   ant tests.browser

   [$[Get Code]]

Creating tests

The browser regression tester supports two kinds of tests: Composite accessor tests and Mocha tests.

Composite accessor tests can be created using the CapeCode code generator. Place your file in accessors/web/test/auto.

For Mocha test files, please see the Mocha instructions. Place your file in accessors/web/hosts/browser/test/auto/mocha.

Integration with the nightly build

Manually running script as Jenkins

This requires permissions to run scripts as the Jenkins user on terra.eecs.berkeley.edu.

• Log in to terra.eecs.berkeley.edu
• Log in as jenkins.
  sudo -i -u jenkins

  [$[Get Code]]
• Start up an X server to intercept screen output.
  Xvfb :4 -screen 0 1024x768x24 &
  export DISPLAY=localhost:4.0

  [$[Get Code]]
• You may see this message. However, the X server appears to start successfully.
  _XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root

  [$[Get Code]]
• Change to the browser test directory
  cd workspace/accessors/web/hosts/browser/test

  [$[Get Code]]
• Run the script. Note that there are two installations of node, /usr/bin/node (v0.10.42) and /usr/local/bin/node (v5.9.0). v0.10.42 will throw a strict mode error due to the const keyword in the selenium libraries. Therefore, use v5.9.0.
  /usr/local/bin/node regressionTestScript.js

  [$[Get Code]]
• The script will find an open port for the web server, start the web server, run the tests, and echo the results to the screen. The results are saved to a file.
  /home/jenkins/workspace/accessors/web/reports/junit/browserTestResults.xml

  [$[Get Code]]
• Note that if the script terminates improperly, the web server may remain running. To kill it, run
  pkill node

  [$[Get Code]]
  The pkill command will kill all instances of the node process owned by the current user (jenkins)

Note that the Jenkins nightly build runs ant tests.browser, see http://terra.eecs.berkeley.edu:8080/job/accessors/configure (accessible only to people with admin access on the accessors build).

Firefox 47.0

Firefox under Mac OS X crashes when running the tests.

Yes, it turns out Firefox 47 is incompatible with the latest version of Selenium. This drove me nuts! My FF also auto-updated and I had no idea why the script quit working. Turns out Selenium is migrating to a new driver for Firefox, which some people are still having problems with, but unfortunately FF47 doesn't work with the old driver. I have a workaround script using the new driver if we need it and can post instructions. https://github.com/SeleniumHQ/selenium/issues/2110

terra is running:

It would be great to keep it that way for a while.

---

Back to testing module.
Back to Testing page.