Recent Changes - Search:

edit SideBar

Matlab

Notes about the Ptolemy Matlab interface.

GLIBXCXX_3.4.15 not found

1/12/15: While running the nightly build under RHEL 6 on sisyphus using hudson

  In addition, loadLibrary("ptmatlab") was called, the exception for the loadLibrary() call was: 
  java.lang.UnsatisfiedLinkError: /home/hudson/jobs/ptII/workspace/lib/libptmatlab.so:
  /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/local/matlab/recent/bin/glnxa64/libut.so)

1/25/2015 Solution

The problem is triggered when I run Java 3D and the load a shared library that uses Matlab.

A possible workaround:

 # Add /usr/local/matlab/recent/sys/os/glnxa64 and avoid problems with Matlab after Java3D
 export LD_LIBRARY_PATH=/usr/local/matlab/recent/sys/os/glnxa64:${LD_LIBRARY_PATH} 

What's happening is that something is loading /usr/lib64/libstdc++.so.6 which is not up to date enough for Matlab.

Running lsof -p pid on the Java pid after the Java3D is loaded, and then running ldd indicates that /usr/lib64/dri/swrast_dri.so requires /usr/lib64/libstdc++.so.6.

I ran:

 lsof -p 24726 >& /tmp/o
 awk '{print $NF}' /tmp/o | egrep '\.so' | xargs ldd >& /tmp/l

/tmp/l says:

 /usr/lib64/dri/swrast_dri.so: 
        linux-vdso.so.1 =>  (0x00007fffa2fff000)
        libdrm_intel.so.1 => /usr/lib64/libdrm_intel.so.1 (0x00007fcdc52db000)
        libdrm_radeon.so.1 => /usr/lib64/libdrm_radeon.so.1 (0x00007fcdc50ce000)
        libdrm.so.2 => /usr/lib64/libdrm.so.2 (0x00007fcdc4ec3000)
        libexpat.so.1 => /usr/local/matlab/recent/bin/glnxa64/libexpat.so.1 (0x00007fcdc4c9b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcdc4a5d000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fcdc4859000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fcdc463a000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fcdc4333000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fcdc40af000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fcdc3d1b000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fcdc3b04000)
        libpciaccess.so.0 => /usr/lib64/libpciaccess.so.0 (0x00007fcdc38fb000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fcdc36f3000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003236a00000)

http://dri.freedesktop.org/wiki/glxinfo/ indicates that swrast_dri is part of OpenGL.

1/23/2015 Yet another attempt

  • It is unclear when this started to fail. It could be with the update to R2014b on Nov. 26, things started failing, or with the changes for OpenModelica
  • The MatlabWirelessSoundDetector test started failing around #1063 on 12/24. Before then, it could be that the tests were not running until that test. Below is a list of runs where ExportModelJUnit Test was run and the last error that was reported.
    • Nov 11, 00:40am 1011: [85] ptolemy/actor/lib/jmf/demo/JMFJAI/JMFJAI.xml - Note that this run ended early.
    • Oct 24, 3:00am 991: [204] ptolemy/demo/Robot/RobotChase.xml - Not clear if the MatlabWirelessSoundDetector ran here.
    • Oct 6, 3:02am 971: [211] ptolemy/demo/Robot/SmartIntruder.xml
  • The problem can be reproduced on another machine running the same OS (RHEL 6.1) but running Jenkins, so the problem is not machine-specific nor is it hudson-specific.

The idea now is to narrow the problem down. As running the one model by hand does not work, now we try running all the export tests as hudson on sisyphus:

  Xvfb :3 -screen 0 1024x768x24 &
  export DISPLAY=localhost:3.0
  cd $PTII
  nohup ant test.longest >& /tmp/m1.out &

The above takes about 90 minutes and reproduces the bug!

Narrowing down the problem: Line 4

Placing the test early in the file:

  $CLASSPATH/lbnl/demo/CRoom/CRoom.xml
  $CLASSPATH/lbnl/demo/MatlabRoom/MatlabRoom.xml
  $CLASSPATH/lbnl/demo/SystemCommand/SystemCommand.xml
  $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
  $CLASSPATH/org/ptolemy/machineImprovisation/demo/JazzImprovisation/JazzImprovisation.xml
  $CLASSPATH/org/ptolemy/machineLearning/hmm/demo/CommunicationAnomalyDetectionAOM/CommunicationAnomalyDetectionAOM.xml

Does not replicate the problem.

Ok, so perhaps something is messing with the environment? Or the process is getting large?

Narrowing: Line 249:

Trying it at the middle of the file.

  $CLASSPATH/ptolemy/domains/continuous/demo/NewtonsCradle/NewtonsCradleAnimated.xml
  $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
  $CLASSPATH/ptolemy/domains/continuous/demo/NewtonsCradle/NewtonsCradleComplicated.xml

Line 249 Failed.

Line 64 failed

Line 32 passed

 $CLASSPATH/org/ptolemy/ptango/demo/UCBPower/UCBPower.xml
 $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
 $CLASSPATH/org/ptolemy/ptango/demo/UCBPower/UCBPowerAnalyzerServer.xml

Line 48 failed

 $CLASSPATH/ptolemy/actor/gt/demo/ModelGeneration/ModelGenerationDE.xml
 $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
 $CLASSPATH/ptolemy/actor/gt/demo/PublisherTest/PublisherTest.xml

Line 40 failed

 $CLASSPATH/ptolemy/actor/gt/demo/ConstOptimization/ConstOptimizationDE.xml
 $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
 $CLASSPATH/ptolemy/actor/gt/demo/DiningPhilosophers/DiningPhilosophers.xml

Line 36 passed

 $CLASSPATH/org/ptolemy/qss/demo/RLC/RLC.xml
 $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
 $CLASSPATH/ptolemy/actor/gt/demo/BouncingBallX2/BouncingBallX2.xml

Line 38 failed

 $CLASSPATH/ptolemy/actor/gt/demo/ConstOptimization/ConstOptimization.xml
 $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
 $CLASSPATH/ptolemy/actor/gt/demo/ConstOptimization/ConstOptimizationDDF.xml

Line 37 failed:

 $CLASSPATH/ptolemy/actor/gt/demo/BouncingBallX2/BouncingBallX2.xml
 $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
 $CLASSPATH/ptolemy/actor/gt/demo/ConstOptimization/ConstOptimization.xml

Success!

The issue here is that running a model that loads Java3D results in an error when subsequently loading a Matlab model.

Searching the web

Search for matlab GLIBCXX_3.4.15 not found

Steps to reproduce

  1. Log on to sisyphus as hudson
  2. Edit $PTII/ptolemy/configs/doc/models.txt and replace the contents with:
    $CLASSPATH/ptolemy/domains/wireless/demo/MatlabWirelessSoundDetection/MatlabWirelessSoundDetection.xml
  3. Start up Xvfb and set the DISPLAY:
    Xvfb :3 -screen 0 1024x768x24 &
    export DISPLAY=localhost:3.0
  4. cd to the Ptolemy tree and run the test:
    cd $PTII
    ant test.longest

Unfortunately, this does not reproduce the problem.

Setting

  export LD_LIBRARY_PATH=/usr/lib64:/usr/local/matlab/recent/bin/glnxa64/:/home/hudson/jobs/ptII/workspace/lib

does not reproduce the problem.

ldd

Where is the message coming from?

Running ldd does not reproduce the problem

bash-4.1$ /usr/bin/ldd /usr/local/matlab/recent/bin/glnxa64/libut.so | grep stdc++
libstdc++.so.6 => /usr/local/matlab/recent/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6 (0x00007fc2c96ff000)
bash-4.1$

/usr/lib64/libstdc++.so.6

The message is about GLIBCXX_3.4.15 missing. Indeed, it is missing from that library:

bash-4.1$ strings /usr/lib64/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_FORCE_NEW
GLIBCXX_DEBUG_MESSAGE_LENGTH
bash-4.1$

RHEL does not keep up with GLib, so this is not that surprising.

LD_LIBRARY_PATH

At the start of the hudson run, we echo LD_LIBRARY_PATH:

echo 'LD_LIBRARY_PATH: /usr/local/certi/lib:/home/hudson/jobs/ptII/workspace/lbnl/lib/util:/home/hudson/jobs/ptII/workspace/lib:/usr/local/matlab/recent/bin/glnxa64:/home/hudson/jobs/ptII/workspace/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/local/certi/lib:/home/hudson/jobs/ptII/workspace/lbnl/lib/util:/home/hudson/jobs/ptII/workspace/lib:/usr/local/matlab/recent/bin/glnxa64:/home/hudson/jobs/ptII/workspace/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/local/certi/lib:${WORKSPACE}/lbnl/lib/util:${WORKSPACE}/lib:/usr/local/matlab/recent/bin/glnxa64:${WORKSPACE}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/local/certi/lib:${WORKSPACE}/lbnl/lib/util:${WORKSPACE}/lib:/usr/local/matlab/recent/bin/glnxa64:${WORKSPACE}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/../lib/amd64'

Update I modified the Build Environment plugin and now LD_LIBRARY_PATH is /usr/local/certi/lib:/home/hudson/jobs/ptII/workspace/lbnl/lib/util:/home/hudson/jobs/ptII/workspace/lib:/usr/local/matlab/recent/bin/glnxa64:/home/hudson/jobs/ptII/workspace/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64

java.library.path

The error message includes the value of the java.library.path property, which is set from LD_LIBRARY_PATH and has other properties added.

[junit] The java.library.path property was: /usr/lib/jvm/jdk1.8.0_25/jre/lib/amd64/server:/usr/lib/jvm/jdk1.8.0_25/jre/lib/amd64:/usr/lib/jvm/jdk1.8.0_25/jre/../lib/amd64:/usr/local/certi/lib:/home/hudson/jobs/ptII/workspace/lbnl/lib/util:/home/hudson/jobs/ptII/workspace/lib:/usr/local/matlab/recent/bin/glnxa64:/home/hudson/jobs/ptII/workspace/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/local/certi/lib:/home/hudson/jobs/ptII/workspace/lbnl/lib/util:/home/hudson/jobs/ptII/workspace/lib:/usr/local/matlab/recent/bin/glnxa64:/home/hudson/jobs/ptII/workspace/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/local/certi/lib:${WORKSPACE}/lbnl/lib/util:${WORKSPACE}/lib:/usr/local/matlab/recent/bin/glnxa64:${WORKSPACE}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/local/certi/lib:${WORKSPACE}/lbnl/lib/util:${WORKSPACE}/lib:/usr/local/matlab/recent/bin/glnxa64:${WORKSPACE}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Update Now the java.library.path is

[junit] The java.library.path property was: /usr/local/certi/lib:/home/hudson/jobs/ptII/workspace/lbnl/lib/util:/home/hudson/jobs/ptII/workspace/lib:/usr/local/matlab/recent/bin/glnxa64:/home/hudson/jobs/ptII/workspace/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

  • In the configuration for the Hudson job, the Build Environment plugin has:
    LD_LIBRARY_PATH=${CERTI_HOME}/lib:${WORKSPACE}/lbnl/lib/util:${WORKSPACE}/lib:/usr/local/matlab/recent/bin/glnxa64:${WORKSPACE}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:${LD_LIBRARY_PATH}

The above might account for the ${WORKSPACE}, so for the next run, I've substituted PTII_DIR and set PTII_DIR to a path.

  • ~hudson/.bashrc has:
    export LD_LIBRARY_PATH=${CERTI_HOME}/lib:${PTII}/lbnl/lib/util:${PTII}/lib:/usr/local/matlab/recent/bin/glnxa64:${PTII}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64:${LD_LIBRARY_PATH}
  • Why are there ${WORKSPACE} values?
    • These are coming from the "Build Environment" plugin, which does not do double substitution. The fix was to change the lines to
      • PTIIDIR=/home/hudson/jobs/ptII/workspace
      • LD_LIBRARY_PATH=${CERTI_HOME}/lib:${PTIIDIR}/lbnl/lib/util:${PTIIDIR}/lib:/usr/local/matlab/recent/bin/glnxa64:${PTIIDIR}/vendors/misc/dbxml-2.5.16/install/lib:/usr/local/lib:/usr/lib64
  • The java-1.6.0-openjdk directories at the end are probably because Hudson is running using Java 1.6.
    • The System Properties window for my installation of Hudson says that java.library.path has the value:
    /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    • The System Properties window says that LD_LIBRARY_PATH has the value:
    /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.33.x86_64/jre/../lib/amd64
  • Explicitly setting LD_LIBRARY_PATH to the above and running ant by hand does not reproduce the problem.

A hint from Stack Overflow:

http://stackoverflow.com/questions/9959306/how-to-tell-mex-to-link-with-the-libstdc-so-6-in-usr-lib-instead-of-the-one-i

Not much there

https://fantasticzr.wordpress.com/2013/05/29/matlab-error-libstdc-so-version-glibcxx_3-4-15-not-found/

suggests

  sudo ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 /usr/local/MATLAB/R2012a/bin/glnxa64/libstdc++.so.6

R2014b has different files:

bash-4.1$ ls -l /usr/local/matlab/recent/bin/glnxa64/../../sys/os/glnxa64/ | grep stdc++
lrwxrwxrwx 1 cxh gopher 19 Nov 26 08:15 libstdc++.so.6 -> libstdc++.so.6.0.17
-r-xr-xr-x 1 cxh gopher 6345639 Jul 26 01:52 libstdc++.so.6.0.17
-r--r--r-- 1 cxh gopher 291 Jul 26 01:52 README.libstdc++
bash-4.1$ cat /usr/local/matlab/recent/bin/glnxa64/../../sys/os/glnxa64/README.libstdc++

The GCC runtime libraries included here:

    libstdc++.so.6.0.17 libgcc_s.so.1 libgfortran.so.3.0.0 libquadmath.so.0.0.0

and associated symlinks are part of gcc-4.7.2, available from ftp.gnu.org. They are included with MATLAB in the event that your distribution does not provide them.

I'm hesitant to do the link because I can't reproduce the problem from the command line.

/usr/lib64

I believe I added /usr/lib64 to support OpenModelica. So, I'll try removing it and see what happens.

Edit - History - Print - Recent Changes - Search
Page last modified on January 27, 2015, at 03:53 AM