• View
• Edit
• History
• Print

Travis

Resources

• https://travis-ci.org/icyphy - The accessors, kepler and ptII Travis instances
• https://icyphy.github.io/ptII/ - GitHub Pages that contains static pages that are not often updated
• https://icyphy.github.io/ptII-test - GitHub Pages that has links to logs, reports and other things updated by Travis
• https://github.com/icyphy/ptII-test/issues/1 - The ptII-test GitHub issue that is updated with JUnit Summary Email

Overview

Travis-CI is a free continuous integration system that runs jobs when a GitHub repository is updated. We are using Travis-ci to build Accessors, Kepler and Ptolemy II.

FAQ

Why can't we use Jenkins?

Jenkins has much better JUnit summary facilities, but Jenkins requires that we support a machine for building, whereas Travis is free.

How do I get email about the summary of tests?

Go to https://github.com/icyphy/ptII-test/issues/1 and subscribe to the notifications about this issue. See Summary Email below.

We have lots of tests, how do we get around the 50 minute limit?

$PTII/build.xml.in is read by ./configure and $PTII/build.xml is created. $PTII/build.xml.in has three sets of targets, one for CapeCode (test.capecode1.xml etc.), and one for core (test.core1.xml etc.), and one for exports (test.export1.xml etc.) These jobs use filesets to include and exclude tests. Travis runs each target as a separate job and the output is checked in to the gh-pages branch of the ptII-test repo. Note that we use the ptII-test repo instead of the ptII repo so as to avoid cluttering the ptII repo with commit messages each time a target runs and updates the repo.

How come for each Travis build, I get multiple commit messages?

This is because Travis-ci jobs have a 50 minute time limit, so we are running Multiple Jobs to invoke JUnit and update GitHub Pages. If the volume of email is annoying, then try filtering email that has the subject "Travis Build gh-branch". UCB Bmail users could consider using the Bmail web interface to add a filter so that their email is filtered before an iPhone or Thunderbird client sees the message. In general, we run the tests only once a day using a Travis cron job, which reduces the volume of email. To run the tests anyway, set RUN_TESTS to true in https://travis-ci.org/icyphy/ptII/settings.

Why does Travis indicate that the PT_TRAVIS_TEST_INSTALLER job has failed?

This was because of a Travis-ci Deployment bug, that seems to be fixed now.

It seems like a job is building OpenCV and then timing out after 50 minutes, what do I do?

There was probably a problem with the Travis-CI cache for that job. See Caching for what to do. See also My builds are timing out.

How do I get a shell on the Travis-CI build machine?

See Debugging

Why does the JUnit Test Results show only a few tests?

The way the tests work is that Travis invokes ant targets in parallel that write .xml files to the reports/junit/ directory of the gh-pages branch of the ptII-test repo. One of the first jobs that Travis runs cleans the .xml files from that branch so that subsequent targets can populate the branch. The problem is that if Travis runs more than one master job, then the .xml files generated by the first master job will be removed by the clean job of the second master job. Travis can have multiple master jobs running if there is a git pushes to the ptII repo which starts one master job and then a second push before the first master job completes. One workaround for this is to change the travis-ci ptII settings so that RUN_TESTS is set to false. If this is done, then only the Travis-ci cron job will run the tests, and that cron job happens only once a day.

How do I get more details about failures?

Try clicking on the individual test and then click on the class, which is typically JUnitTclTest, then go to the bottom and click on the System.out and System.err links. Travis runs multiple targets, which are listed in $PTII/.travis.yml and defined in $PTII/build.xml. The output of each target ends up in *.txt files in the reports/junit/ directory of the gh-pages branch of the ptII-test repo. You might find it easier to check out the branch with git clone -b gh-pages https://github.com/icyphy/ptII-test

Ptolemy II (ptII) repository setup

$PTII/.travis.yml

Controls the Travis-ci build

$PTII/bin/ptIITravisBuild.sh

The primary script invoked by Travis to invoke various builds

$PTII/org/ptolemy/opencv/travis_build_opencv.sh

A script that builds OpenCV, see OpenCV Below for details.

Limits and Problems

Multiple Jobs

Jobs on the free https://travis-ci.org system are limited to 50 minutes in run time. However, we can run multiple jobs in parallel, where each job is separate and cannot communicate with the other jobs.

In Travis, we use a Build Matrix to specify multiple jobs. In the $PTII/.travis.yml file, we specify environment variables in the matrix section:

Then, in $PTII/bin/ptIITravisBuild.sh, we check to see if each of the variables is set to true and behave accordingly.

Because these jobs are separate there are various issues

Caching

Each job does not have access to the other jobs, so we use Caching

$PTII/.travis.yml contains:

One key thing is that each of the separate jobs has a separate cache. Different public environment variables cause different caches. The cache is only updated after a successful build.

Building OpenCV can take 20 minutes. If the build is not able to download the OpenCV cache, then building OpenCV could mean that the job will exceed the 50 minute Travis-ci time limit.

$PTII/bin/ptIITravisBuild.sh will detect if an attempt is made to start the tests after 1200 seconds. If this condition is detected, then one test ($PTII/ptolemy/util/test/travis/TravisTimeout.tcl) will be run and will fail. This should allow the Travis run to complete and the cache to be updated.

The next run should work fine.

GitHub Pages

The ptII Travis-ci deploys files to GitHub Pages so that https://icyphy.github.io/ptII-test/ is updated.

Note that Ptolemy II has two sets of GitHub Pages.

For the Travis output, we use the gh-pages branch of the ptII-test repo instead of the gh-pages branch of the ptII repo so as to avoid cluttering the ptII repo with commit messages each time a target runs and updates the repo.

• https://icyphy.github.io/ptII-test/ :
  • gh-pages branch of the ptII-test repository for JavaDoc, JSDoc and JUnit output generated by the Travis-ci ptII job.
  • How to edit the gh-pages branch of the ptII-test repository
• https://icyphy.github.io/ptII/ :
  • gh-pages branch of the ptII repository for static pages about downloads etc. that include links to the JavaDoc, JSDoc and JUnit output in the branch above.
  • How to edit the gh-pages branch of the ptII repository

The ptII Travis-ci build commits the following to the gh-pages branch of the ptII-test repository

• JavaDoc output for Java files
  • Generated by the PT_TRAVIS_DOCS section of $PTII/bin/ptIITravisBuild.sh
• JsDoc output for Javascript
  • Generated by the PT_TRAVIS_DOCS section of $PTII/bin/ptIITravisBuild.sh
• Logs
  • The output generated by a travis-ci job cannot be more than 10,000 lines, so some of the individual jobs in $PTII/bin/ptIITravisBuild.sh generate their output to log files
• Reports
  • Some of the sections in $PTII/bin/ptIITravisBuild.sh generate JUnit output. See JUnit below for details.

The way this works is that in $PTII/bin/ptIITravisBuild.sh, each job that updates the gh-pages branch of the ptII-test repository checks out a separate copy of the branch, copies in any files and does a commit. This means that we get multiple commits each time Travis runs. Each commit is from a separate job that is updating the gh-pages branch of the ptII-test repository. One possible workaround is to set up an email filter to ignore these messages.

Having each Travis job commit to the gh-pages branch can cause version control conflicts if two separate Travis jobs are both trying to update the same pages at the same time. To avoid this, we use different timeouts in $PTII/bin/ptIITravisBuild.sh

Note that the PT_TRAVIS_DOCS job cleans the .xml and html reports out of reports/junit so that the output from the subsequent tests goes in to an empty directory. If we don't do that, then we will get out of date test results from previous runs.

Deployment

We use the Travis-ci GitHub Releases facility to deploy JavaDoc jar files and installers to the GitHub ptII releases page.

$PTII/.travis.yml contains:

The above has two conditionals represented by the on clause that are run for different jobs (Recall from Multiple Jobs above, that Travis uses the variables defined in the matrix section to launch multiple jobs.):

1. If $PTII_TRAVIS_TEST_INSTALLERS = true, then update the installers
2. If $PT_TRAVIS_DOCS = true, then update the codeDoc jar files, that are used to create the installers

Note that there is a travis-ci/dpl issue: #796: Timeout while uploading to GitHub Releases: Net::ReadTimeout (Faraday::TimeoutError). This issue is intermediate, but it means that the job that builds the installers seems to fail during the daytime, but it sometimes succeeds at night.

We are using GitHub releases because of the size of the installers. GitHub limits file sizes to 100mb unless GitHub Large File Storage (LFS) is used. In reality, we don't really want to commit the installers to version control each time we create them because it is a waste of resources. Instead, we want to save the latest build and save installers at major milestones.

Our hope is that the above issue will be fixed so that we can deploy large installers.

OpenCV

We want to have OpenCV available so as to test the OpenCV accessors and reduce the number of error messages.

Unfortunately, as of March, 2018, Travis only Precise and Trusty (Ubuntu 14.04). Trusty has rpms for OpenCV 2.x, which does not have Imgproc.LINE_8, which is used by org/ptolemy/opencv/ComputerVision.java

Unfortunately, travis-ci does not yet have support for Ubuntu 16 or later:

• Ubuntu 16.04 LTS - Xenial Xerus
• Ubuntu 16.04 LTS Support
  • dist: xenial worked for a bit in December, 2017, but probably is not working now.

OpenCV Solution

As a workaround, we are building OpenCV-3.4.1 as part of the travis-ci ptII build using $PTII/bin/ptIITravisBuild.sh and then using the Travis cache to cache the results.

JUnit

https://github.com/softwaresaved/build_and_test_examples/blob/master/travis/README.md says:

• Handle Ant's JUnit XML formatted output from test/spec runs (travis-ci issue)
• Extract number of tests passed from build output (travis-ci issue)
• junit2html Read a single .xml file, produce a .html file. Can handle matrices.
• junit-email-report Merges junit xml from reports path and creates a static email html file for reporting

JUnit Solution

We are using ant junitreport target to generate https://icyphy.github.io/ptII/reports/junit/html/index.html

The way this works is that we have multiple travis-ci jobs that run JUnit and check in the output of the tests in to the gh-pages branch of the ptII-test repo so that the results appear on GitHub pages. The reason we do this is because jobs are limited to 50 minutes and each job has a separate directory. The jobs also run the junitreport target, so the last job that is run includes the test results of all of the jobs. The $PTII/bin/ptIITravisBuild.sh script does this for us.

Summary Email

What I really want is a summary email that includes the number of test failures. Even just:

  Summary
  Tests	Failures	Errors	Skipped	Success rate	Time
  958	3	36	0	95.93%	4276.927

There is a long standing open issue about this Handle Ant's JUnit XML formatted output from test/spec runs (https://github.com/travis-ci/travis-ci/issues/239), and Travis-ci has this in their list of possible features.

It turns out that we can invoke a script at the end of a run (https://docs.travis-ci.com/user/deployment/script/). The script could send email. The travis-ci interface that sends email does not seem to be available to us, though I could look in to this. To send email would almost certainly require a user name and password. The tricky part is that we need to keep secret any sort of password. Travis does have some capabilities to keep secrets. Initially, I was thinking we could use npm to do this, but we could also use Ptolemy's email actor to do so. The main thing is keeping the secret.

One issue with using Ptolemy is that if Ptolemy fails to build, then we won't get test results. However, we can have the build send email if there is a failure using the Travis-ci notification facility.

Summary Solution

The npm module @icphy/github-issue-joint is based on https://github.com/vitalets/github-trending-repos/ and a suggestion by @nebgnahz.

The module reads JUnit HTML output and adds a comment to an open issue on GitHub.

This module hacks around the problem of running a Travis-CI job and wanting to notify users of failed tests.

This is a start on solving Handle Ant's JUnit XML formatted output from test/spec runs.

See https://github.com/icyphy/ptII-test/issues/1 for sample output.

A separate script clean-issues.js is run that updates the issue and removes results that are older than 30 days.

How this works is that $PTII/.travis.yml defines a report stage, that is invoked after all the test are run:

Then $PTII/bin/ptIITravisBuild.sh checks for the value of the PT_TRAVIS_JUNITREPORT environment variable and invokes the node module:

Private ptII Repo And Travis

''Note that it is possible to run a public repo in debug mode, the instructions below are for how to set up a private repo, which was thought to be necessary to ssh in to Travis-ci for debugging. See Debugging below."

https://github.com/icyphy/ptII-private is a private copy of the ptII repo that was made in April, 2018 by creating a new empty private repo named ptII-private and then importing the ptII repo.

See https://education.travis-ci.com/ for how to get access to travis-ci.com for two years via an educational account.

See https://travis-ci.com/profile/icyphy for how to add the ptII-privi repo.

Debugging

It is possible to run the public ptII repo in debug mode.

https://docs.travis-ci.com/user/running-build-in-debug-mode/ says to email support to enable debugging in a public repo, which we did.

See https://docs.travis-ci.com/user/running-build-in-debug-mode/ for how to log in using ssh.

1. Log in to https://travis-ci.org using GitHub.
2. Go to the job to run, in this case I'm using one of the PT_TRAVIS_BUILD_ALL jobs. I think any job works, though recent is better than older so that the correct setup is done. For example: https://travis-ci.org/icyphy/ptII/jobs/366197043 and note the ID is 366197043
3. Get the API Authentication token from https://app.travis-ci.com/account/preferences
4. Invoke the curl command with the token and Id
   
     curl -s -X POST \
      -H "Content-Type: application/json"   \
      -H "Accept: application/json"  \
      -H "Travis-API-Version: 3"  \
      -H "Authorization: token YourToken "   \
      -d "{\"quiet\": true}"  \
      https://api.travis-ci.com/job/366197043/debug
    

   [$[Get Code]]
5. Then look at https://travis-ci.org/icyphy/ptII/jobs/366197043 and look for the ssh command

Timeouts

I believe that the reason that a bunch of tests are failing under Travis is because Travis is slow, so the DE Director stopTIme is reached. For example:

My workaround is to add a JavaScriptBarrier accessor connected to a Stop accessor.

One key thing is that the timeout of the JavaScriptBarrier actor needs to be adjusted. I've been setting it to the value of the DE Director stopTime parameter. JavaScriptBarrier timeout is in ms. and DE Director stopTime is in seconds when synchronizeToRealTime is selected, so I multiply by 1000.

This has the added advantage of ending the model execution as soon as the last TrainableTest finishes.

ptolemy/actor/lib/jjs/modules/httpClient/test/auto/RESTGet.xml is below.

"Protocol family unavailable"

Under Travis, ptolemy/actor/lib/jjs/modules/httpClient/test/auto/RESTGet.xml was failing with

  Caused by: java.lang.RuntimeException: REST2: Protocol family unavailable: www.example.com/2606:2800:220:1:248:1893:25c8:1946:443   

The solution is to invoke Java with -Djava.net.preferIPv4Stack=true

See https://docs.oracle.com/javase/7/docs/api/java/net/doc-files/net-properties.html

For the JUnit tests invoked by ant, we invoke a script at ptolemy/util/test/junit/javachdir, so that script was modified to include -Djava.net.preferIPv4Stack=true

You can see how javachdir is invoked in the jvm value of the junit command in $PTII/build.xml:

Private Keys

Various demos use private keys. The solution is to use Travis-ci encrypted environment variables.

To do this requires installing the travis command line tool.

Then add the secure line to $PTII/.travis.yml

Note that the above is on one line, with no backslashes.

Then, edit $PTII/bin/ptIITravisBuild.sh:

Important': Also, update $PTII/bin/ptIITravisBuild.sh so that WEATHER_TOKEN is removed:

There are several occurrences of egrep statements like the one above.