Recent Changes - Search:

edit SideBar

Subversion

Resources

Installing Subversion

Checking out Ptolemy II

SVN vs CVS

Below is a comparison between SVN and CVS. The first two colums were taken from the The Subversion Features List

FeatureSVNCVS
Most CVS featuresSubversion was originally designed to be a better CVS, so it has most of CVS's features. Generally, Subversion's interface to a particular feature is similar to CVS's, except where there's a compelling reason to do otherwise.Subversion has since expanded beyond its original goal of replacing CVS, but its history influenced its feature and interface choices; Subversion today should still feel very familiar to CVS users.CVS has CVS's features (CVS Wins)
Directories are versioned.Subversion versions directories as first-class objects, just like files.(SVN Wins)In CVS, directories are not first-class objects
Copying, deleting, and renaming are versionedCopying and deleting are versioned operations. Renaming is also a versioned operation, albeit with some quirks.(SVN wins because of renaming)Renaming in CVS is difficult
Free-form versioned metadata ("properties").Subversion allows arbitrary metadata ("properties") to be attached to any file or directory. These properties are key/value pairs, and are versioned just like the objects they are attached to. Subversion also provides a way to attach arbitrary key/value properties to a revision (that is, to a committed changeset). These properties are not versioned, since they attach metadata to the version-space itself, but they can be changed at any time. (SVN Wins, though it is vendor lock-in)CVS does not have this, but do we need it? Actually, it turns out that we can use these properties in Kepler to place a dependency on the ptII svn tree so that when a user checks out the kepler svn tree, they can automatically check out the ptII svn tree. This is a case of vendor lock-in, but does give SVN an advantage.
Atomic commits.No part of a commit takes effect until the entire commit has succeeded. Revision numbers are per-commit, not per-file, and commit's log message is attached to its revision, not stored redundantly in all the files affected by that commit.This is a UI issue, In Eclipse, it is possible to commit many changes at once. In 50k changes over 12 years, we have not had a problem. I could see in a very busy repository, this might help. but if the repository is that busy, then there are going to be plenty of conflict problems.
Branching and tagging are cheap (constant time) operations.There is no reason for these operations to be expensive, so they aren't. Branches and tags are both implemented in terms of an underlying "copy" operation. A copy takes up a small, constant amount of space. Any copy is a tag; and if you start committing on a copy, then it's a branch as well. (This does away with CVS's "branch-point tagging", by removing the distinction that made branch-point tags necessary in the first place.)Computers are fast, I've not noticed this as a problem.
Merge tracking.Subversion 1.5 introduces merge tracking: automated assistance with managing the flow of changes between lines of development, and with the merging of branches back into their sources. The 1.5 release of merge tracking has basic support for common scenarios; we will be extending the feature in upcoming releases.Again, Eclipse or diff help here.
File locking.Subversion supports (but does not require) locking files so that users can be warned when multiple people try to edit the same file. A file can be marked as requiring a lock before being edited, in which case Subversion will present the file in read-only mode until a lock is acquired.CVS can support file locking, but file locking is bad.
Symbolic links can be versioned.Unix users can place symbolic links under version control. The links are recreated in Unix working copies, but not in win32 working copies.(SVN wins)CVS does not create symbolic links. It would be nice if svn would create cygwin compatible links under win32.
Executable flag is preserved.Subversion notices when a file is executable, and if that file is placed into version control, its executability will be preserved when it it checked out to other locations. (The mechanism Subversion uses to remember this is simply versioned properties, so executability can be manually edited when necessary, even from a client that does not acknowledge the file's executability, e.g., when having the wrong extension under Microsoft Windows).CVS has something similar.
Apache network server option, with WebDAV/DeltaV protocol.Subversion can use the HTTP-based WebDAV/DeltaV protocol for network communications, and the Apache web server to provide repository-side network service. This gives Subversion an advantage over CVS in interoperability, and allows certain features (such as authentication, wire compression) to be provided in a way that is already familiar to administrators (SVN Wins, if you are willing to have a significantly less secure and compartmentalized source server)CVS does not have WebDAV access, so offsite developers behind firewalls have to jump through hoops. However, CVS is much more secure than SVN.
Standalone server option (svnserve).Subversion offers a standalone server option using a custom protocol, since not everyone wants to run an Apache HTTPD server. The standalone server can run as an inetd service or in daemon mode, and offers the same level of authentication and authorization functionality as the HTTPD-based server. The standalone server can also be tunnelled over ssh.CVS has pserver
Parseable output.All output of the Subversion command-line client is carefully designed to be both human readable and automatically parseable; scriptability is a high priority.CVS is similar
Localized messages.Subversion uses gettext() to display translated error, informational, and help messages, based on current locale settings.(SVN Wins, but this is not that important)CVS does not particularly have localization.
Interactive conflict resolution.The Subversion command-line client (svn) offers various ways to resolve conflicting changes, include interactive resolution prompting. This mechanism is also made available via APIs, so that other clients (such as graphical clients) can offer interactive conflict resolution appropriate to their interfaces. (A slight win for SVN)CVS indicates conflicts and expects the user to use their IDE.
Repository read-only mirroring.Subversion supplies a utility, svnsync for synchronizing (via either push or pull) a read-only slave repository with a master repository. (A big win for SVN)CVS does not have mirroring out of the box.
Write-through proxy over WebDAV.Subversion 1.5 introduces a write-through proxy feature that allows slave repositories (see read-only mirroring) to handle all read operations themselves while passing write operations through to the master. This feature is only available with the Apache HTTPD (WebDAV) server option. (A win for SVN, if security is not a problem)CVS does not have this feature.
Natively client/server, layered library design with clean APIs.Subversion is designed to be client/server from the beginning; thus avoiding some of the maintenance problems which have plagued CVS. The code is structured as a set of modules with well-defined interfaces, designed to be called by other applications.CVS is mature software with a idiosyncratic interface. This is neither an advantage nor a disadvantage.
Binary files handled efficiently.Subversion is equally efficient on binary as on text files, because it uses a binary diffing algorithm to transmit and store successive revisions. (SVN wins, but size is not a huge problem these days and SVN repositories are almost 2x CVS repositories)CVS handles binary files, but not efficiently.
Costs are proportional to change size, not data size.In general, the time required for a Subversion operation is proportional to the size of the changes resulting from that operation, not to the absolute size of the project in which the changes are taking place.CVS and SVN seem to have about the same speed over ssh. SVN over WebDAV will be slower than SVN over SSH.
Bindings to programming languages.The Subversion APIs come with bindings for many programming languages, such as Python, Perl, Java, and Ruby. (Subversion itself is written in C.)CVS is callable from the command line. I don't see this as important.
Changelists.Subversion 1.5 introduces changelists, which allows a user to put modified files into named groups on the client side, and then commit by specifying a particular group. For those who work on logically separate changesets simultaneously in the same directory tree, changelists can help keep things organized.Is this important?

SVN benefits to Kepler

  • WebDAV
    • Access from behind firewalls works.
    • LDAP access is easier than named Unix accounts
    • WebDAV has finer grained access control
  • Kepler found cvs2svn conversion to result in smaller modules, which is different than the ptII experience (see below).
  • Easier to move and rename directories in svn and retain logs
  • svnsync will make it easier because Kepler depends on ptII.

Building SVN under Solaris

See http://subversion.apache.org/source-code.html

We use FSFS for the backend, not Berkeley DB. See http://mercury.nceas.ucsb.edu/kepler/pipermail/kepler-dev/2008-June/013097.html

Below are the steps to build subversion-1.7.2:

I was having various problems installing subversion-1.7.2 under Solaris so I tried autogen.sh

wget http://apache.osuosl.org/subversion/subversion-1.7.2.tar.gz
gtar -zxf subversion-1.7.2.tar.gz
cd subversion-1.7.2
export LD_LIBRARY_PATH=/usr/local/lib
bash get-deps.sh
bash autogen.sh
./configure --prefix=/usr/local/subversion-1.7.2 --with-ssl --with-libs=/usr/local/ssh/lib --enable-shared="yes" --enable-static="no" --disable-nls
gmake >& make.out &
export PATH=/usr/bin:${PATH}
gmake install >& install.out &

Problem: SSL negotiation failed: SSL disabled due to library version mismatch

source.EECS.Berkeley.EDU 9# svn co https://bambi.millennium.berkeley.edu/projects/chess/ptII/trunk ptII
svn: E175002: Unable to connect to a repository at URL 'https://bambi.millennium.berkeley.edu/projects/chess/ptII/trunk'
svn: E175002: OPTIONS of 'https://bambi.millennium.berkeley.edu/projects/chess/ptII/trunk': SSL negotiation failed: SSL disabled due to library version mismatch (https://bambi.millennium.berkeley.edu)

http://www.svnforum.org/threads/35175-SSL-negotiation-failed-Secure-connection-truncated suggests: "So Mozilla, RapidSVN, and TortoiseSVN were all breaking because of the SSLCipherSuite setting (both the original Apache-supplied value, and another value recommended by QualysGuard)."

When I upgraded from httpd-2.2.20 to 2.2.22, I noticed that httpd-ssl.conf changed:

< SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
---
> SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5

However, it looks like the problem is that /usr/local/lib had links to older versions of the openssl libraries.

Below are the steps to build Subversion-1.6.17:

wget http://subversion.tigris.org/downloads/subversion-deps-1.6.17.tar.gz
wget http://subversion.tigris.org/downloads/subversion-1.6.17.tar.gz
gtar -zxf subversion-1.6.17.tar.gz
gtar -zxf subversion-deps-1.6.17.tar.gz
cd subversion-1.6.17
export LDFLAGS="-L/usr/local/lib -R/usr/local/lib -L/usr/sfw/lib -R/usr/sfw/lib -L/usr/lib -R/usr/lib"
export LIBS="$LIBS -lintl"
export LD_FLAGS=-L/usr/local/lib
./configure --prefix=/usr/local/subversion-1.6.17 --with-ssl --with-libs=/usr/local/ssh/lib --enable-shared="yes" --enable-static="no" --disable-nls
cd serf
./configure --with-ssl --with-libs=/usr/local/ssh/lib
cd ..
gmake
export PATH=/usr/bin:${PATH}
gmake install

See http://bill.sirinek.com/?p=83

"1. Multiple inclusion of file errors while building neon libraries
/build/directory/.libs/libsvn_subr-1.so:
attempted multiple inclusion of file
Undefined first referenced
symbol in file
libintl_dgettext main.o (symbol belongs to implicit
dependency /usr/local/lib/libintl.so.8)
ld: fatal: Symbol referencing errors. No output written to .libs/svnadmin
collect2: ld returned 1 exit status
I fixed this by doing a two things:
- Make sure the $LIBS variable has “-lintl” included.
- Don’t build static libraries. I’m not sure why you would want to do this anyway. When running the top-level configure, include the options --enable-shared="yes" and --enable-static="no"

Note that during the install step, I got:

/bin/bash /export/home1/root/tools/subversion-1.6.12/apr/libtool --mode=install
build/install.sh -c -m 755 libaprutil-1.la /usr/local/subversion-1.6.12/lib
libtool: install: error: cannot install `libaprutil-1.la' to a directory not ending in /usr/local/apr/lib
gmake[1]: *** [install] Error 1
gmake[1]: Leaving directory `/export/home1/root/tools/subversion-1.6.12/apr-util'
gmake: *** [external-install] Error 1

However, I already have /usr/local/apr/lib installed, so I don't think this is an issue.

We need --with-ssl so that we avoid svnsync: SSL is not supported.

I tried --disable-nls so as to avoid

bash-3.00# /etc/init.d/apachectl configtest
httpd: Syntax error on line 430 of /usr/local/apache-2.2.16/conf/httpd.conf:
Cannot load /usr/local/apache-2.2.16/modules/mod_dav_svn.so into server:
ld.so.1: httpd: fatal: relocation error: file /usr/local/subversion-1.6.12/lib/libsvn_repos-1.so.0:
symbol libintl_dgettext: referenced symbol not found

but that did not work

The link was failing in subversion-1.6.12:

Undefined first referenced
symbol in file
BIO_set_flags ./.libs/libserf-0.a(ssl_buckets.o)
BIO_clear_flags ./.libs/libserf-0.a(ssl_buckets.o)
SSL_CTX_set_client_cert_cb ./.libs/libserf-0.a(ssl_buckets.o)
BIO_test_flags ./.libs/libserf-0.a(ssl_buckets.o)

One fix is to edit Makefile:

LIBS = -L/usr/local/lib -lssl -lcrypto -liconv -lintl -lsocket -lz

Another fix is to run ./configure --with-ssl --with-libs=/usr/local/ssh/lib in the serf directory.

Another fix is to add -L/usr/local/lib to the link lines before -lssl -lcrypto:

/export/home1/root/tools/subversion-1.6.12/apr/libtool --silent --mode=link \
gcc -static -o test/serf_get libserf-0.la test/serf_get.lo \
/export/home1/root/tools/subversion-1.6.12/apr-util/libaprutil-1.la \
-lexpat /export/home1/root/tools/subversion-1.6.12/apr/libapr-1.la -luuid \
-lsendfile -lrt -lsocket -lnsl -lpthread -lm -lz -L/usr/local/lib -lssl -lcrypto

The real fix is to use export LDFLAGS=-L/usr/local/lib before running ./configure.

Updating OpenSSL from 0.9.8.g to 0.9.8.o:

./config \
--prefix=/usr/local/openssl-0.9.8o \
--openssldir=/usr/local/openssl-0.9.8o \
-fPIC shared

The link was failing in subversion-1.4.5:

Undefined first referenced
symbol in file
libintl_bind_textdomain_codeset /export/home1/root/tools/subversion-1.4.5/subversion/libsvn_subr/.libs/libsvn_subr-1.so
libintl_bindtextdomain /export/home1/root/tools/subversion-1.4.5/subversion/libsvn_subr/.libs/libsvn_subr-1.so
libintl_dgettext blame-cmd.o
ld: fatal: Symbol referencing errors. No output written to .libs/svn
collect2: ld returned 1 exit status

The fix was to edit Makefile and add the following to SVN_APR_LIBS:

-L/usr/local/lib -liconv -lintl

and set LIBS:

LIBS = -L/usr/local/lib -liconv -lintl -lsocket -lz

Complaints against Subversion

  • Is svn that much better than cvs? http://subversion.tigris.org says "Subversion is meant to be a better CVS, so it has most of CVS's features". Typical arguments used for svn over cvs:
    • Merging in svn is better than in cvs. How many people actually use branches?
    • It is easier to move files in svn than in cvs. This has some merit, but is it worth the effort?
  • If svn is a better CVS, then why is the svn status command different than cvs status.
  • It turns out that svn info provides some of the info of cvs status, but how do I determine if keyword substitution is occurring? Under cvs, the cvs status command says -kb. What about svn?
  • Building the client requires way too many other packages. How can svn possibly stay secure if it depends on so many packages?
  • Subversion can optionally use Apache for access. Enabling a web server on a machine that does not already have one makes the machine less secure.
  • There is no decent svn Unix style man page. This is deliberate, see Bug 1508. This is not good. I want to know exactly what commands will work with a specific installation of SVN, not what the latest documentation for the latest version is.
  • On the server, svn is a disk hog.
    • A gzipped tar file of the Ptolemy II cvs tree is 372.9 MB.
    • A gzipped tar file of the same tree after running the conversion from cvs to svn is 570MB.
  • On the client, svn is a disk hog.
    • Under Linux, the checked out ptII cvs repository is 333Mb.
    • Under Windows, the checked out ptII ssh repository is 726Mb
-bash-3.1$ du -sk ptII
332880  ptII
-bash-3.1$ du -sk trunk
725648  trunk
-bash-3.1$
  • Problems with Eclipse clients Subversive and Subclipse.

A user wrote: "I found both Subversive and Subclipse painful; I had to perform significant repairs externally using TortoiseSVN. I even managed to get Subversive to delete a 10MB directory tree by mistake!"

"Both Subversive and Subclipse are very dangerous for CVS users. In CVS, folders do not have significant state so you can do what you like when you like. In SVN you MUST commit folder state immediately after committing file state. For instance if you add something to svn:ignore, you must commit the folder, otherwise you risk having an outgoing conflict between svn:ignore outgoing and child folder state incoming. Do not attempt to fix such a conflict with either of Subclipse or Subversive; you just dig a deeper hole. Use TortoiseSVN to get back to all green arrows."

"Unfortunately neither subclipse nor subversion have GUI state for folder state comparison so you get no clue as to why folders are in a mess. The GUI is generally inadequate in distinguishing between folder as content and folder as container of children."

  • Another drawback with svn is that the modtimes of the files are not

preserved. When I do a cvs checkout, the modtimes of the files are preserved. This makes it much easier to see what has recently changed.

Here's the svn tree:

bash-3.00$ ls -ltr ~/ptII/ptolemy/kernel/util/*Exception.java
-rw-r--r--   1 cxh      labstaff   18596 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/KernelException.java
-rw-r--r--   1 cxh      labstaff    9137 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/KernelRuntimeException.java
-rw-r--r--   1 cxh      labstaff    5858 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/InvalidStateException.java
-rw-r--r--   1 cxh      labstaff    3646 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/InternalErrorException.java
-rw-r--r--   1 cxh      labstaff    5121 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/IllegalActionException.java
-rw-r--r--   1 cxh      labstaff    4937 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/NameDuplicationException.java
-rw-r--r--   1 cxh      labstaff    2660 Jun 12 13:10 /home/eecs/cxh/ptII/ptolemy/kernel/util/NoSuchItemException.java

Here's the cvs tree:

bash-3.00$ ls -ltr ~/src/ptII.cvs/ptolemy/kernel/util/*Exception.java
-rw-r--r--   1 cxh      labstaff    5125 Jul  8  2005 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/IllegalActionException.java
-rw-r--r--   1 cxh      labstaff    3650 Jul  8  2005 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/InternalErrorException.java
-rw-r--r--   1 cxh      labstaff    4941 Jul  8  2005 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/NameDuplicationException.java
-rw-r--r--   1 cxh      labstaff    2664 Jul  8  2005 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/NoSuchItemException.java
-rw-r--r--   1 cxh      labstaff   18600 Dec  6  2007 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/KernelException.java
-rw-r--r--   1 cxh      labstaff    9141 Dec  6  2007 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/KernelRuntimeException.java
-rw-r--r--   1 cxh      labstaff    5862 Jun  6 08:13 /home/eecs/cxh/src/ptII.cvs/ptolemy/kernel/util/InvalidStateException.java                  
bash-3.00$

Being able to quickly see which Java file was last updated in the cvs tree is a serious win.

Also, the ptII tree includes derived files that are generated by Autoconf, JavaCC and Antlr. If the commit times are not preserved, then the source file might have a mod time later than the derived file which means that either the tool must be run or else the build system must touch the derived file. This is gross. Why not just preserve the commit time?

Why doesn't svn preserve mod times?

  • Ian Brown mentioned that TortoiseSVN has an option that does this:
Set filedates to the "last commit time"
This option tells TortoiseSVN to set the filedates to the last commit time when doing a checkout or an update. Otherwise TortoiseSVN will use the current date. If you are developing software it is generally best to use the current date because build systems normally look at the datestamps to decide which files need compiling. If you use "last commit time" and revert to an older file revision, your project may not compile as you expect it to.
  • Re: [Issue 1445] checkout/export must set last-commit-timestamps - 2004 post that discusses the problem.
  • SVN Book, Chapter 7, Advanced Topics - covers setting use-commit-times
    • use-commit-times can be set in the Subversion config file. The tricky part is finding that file. For me, under Cygwin, the file was $HOME/.subversion/config. Other places to look are c:/Documents and Settings/cxh/Application Data/Subversion/config. Make this change:
      ### Set use-commit-times to make checkout/update/switch/revert
      ### put last-committed timestamps on every file touched.
      use-commit-times = yes
  • svn2cl is slow. This is not a svn problem, but a problem with xsltproc. Running svn2cl on a the ptII tree takes 43 minutes. Running gnuify-changelog.pl takes 84 seconds. However, gnuify-changelog.pl does not include the names of the files that were committed.

Still, I miss cvs2cl, which had nice features like clumping non-atomic cvs commits into one block of commits. It is a drawback in svn log that the output is not similar to cvs log.

The developer of svn2cl said:

I'm afraid that XSLT processing can be quite slow. Some improvements have been made in svn2cl in the past to improve performance but using XSLT there is not much more room for improvement that I can think off while keeping the thing maintainable (not that XSLT templates are maintainable to begin with ;-( ).
You could however try the --group-by-day option or generate the ChangeLog in batches and combine those as a post-processing step. Some projects have a yearly ChangeLog file. Another option is to keep the previous ChangeLog file around and only generate new entries (svn2cl can't do this automatically yet though).
  • Conversion from CVS lost a few files.
  • Conversion from CVS requires --default-eol=native. Instead, it should just convert files they are in the tree. Why do I have to specify a native default?

Conversion from CVS

CVS to SVN conversion procedure

  1. Download and install cvs2svn
  2. Make sure that all the binary files (.pdf, .eps, .gif, .png) etc. in the cvs repo have -kb set.
  3. Copy the ptII repository for testing. cvs2svn will not work remotely see the How can I convert a CVS repository to which I only have remote access?. On the cvs repository machine (source), I copied /user/cvs/CVSROOT and /users/cvs/ptII
     cd /home
     gtar -zcf ptII.tar.gz --exclude=commitlogs cvs/CVSROOT  cvs/ptII
  4. On a fast machine other than the CVS repository, run the command
    cd /tmp
    mkdir cxh
    cd cxh
    scp source:/home/ptII.tar.gz .
    tar -zxf ptII.tar.gz
    cd cvs2svn-2.4.0
    ./cvs2svn --default-eol=native --fallback-encoding=utf_8 -s /tmp/cxh/repos/ptII /tmp/cxh/cvs/ptII
  5. Then try out the repository
    cd /tmp/cxh
    mkdir tmp
    cd tmp
    svn checkout file:///tmp/cxh/repos/ptII/trunk

Script to fix up cvs

#!/bin/sh
# $Id$
# Fix up common problems in a CVS repository
# 1. Binary files are not properly -kb

# Generate a list of files
find . -type f | grep -v /CVS/ | sort > /tmp/fixcvs.files

# Generate a list of all the files that are listed in
# ~/.subversion/config that are not text files.
# That is, all the files that are not native line endings
egrep -v "(`egrep -v "^#" $HOME/.subversion/config | grep 'svn:eol-style=native' | sed 's@^\([^= ]*\).*$@\1@' | sed 's@\*.@\\\\.@' | awk '{if (NR\
> 1) { printf("|")};printf("%s", $1)}'`)$" /tmp/fixcvs.files | sort > /tmp/fixcvs.nonative

# Generate a list of all files that are mime-types
egrep "(`egrep -v "^#" $HOME/.subversion/config | grep 'svn:mime-type' | sed 's@^\([^= ]*\).*$@\1@' | sed 's@\*.@\\\\.@' | awk '{if (NR > 1) { pr\
intf("|")};printf("%s", $1)}'`)$" /tmp/fixcvs.files | sort > /tmp/fixcvs.mime

echo "The extensions of the files below are not listed in ~/.subversion/config:"
comm -23 /tmp/fixcvs.nonative /tmp/fixcvs.mime

echo "Checking for binary files that are not -kb"

finalResult=0

binaries=`cat /tmp/fixcvs.nonative`
for binary in $binaries
do
echo -n "."
#echo -n $binary
cvs status $binary | egrep -e "-kb$" > /dev/null
status=$?
if [ $status -ne 0 ]; then
echo "$binary is not -kb?"
finalResult=1
fi
done

exit $finalResult

Moving the ptpapers repo

I wanted to move the ptpapers repo from CVS to SVN. After doing so, I did a diff and realized that the pdf files in the cvs repo were not checked in with -kb.

So, I ran cvs admin -kb on the .pdf and .eps files

diff -I '\$Id:' -r ptpapers cvs/tmp/ptpapers/trunk/ptpapers >& /tmp/ptpapers.diff
egrep "diff -I" /tmp/ptpapers.diff | awk '{print $5}' | egrep -v '(Makefile|sty|xml|bst|tex|cls|bib|txt|figtex2eps|\.m)$' > /tmp/f1
cvs admin -kb `cat /tmp/f1`

mkdir /home/cxh/src/cvs2
cd /home/cxh/src/cvs2
ssh source "cd /home; gtar zcf - --exclude=commitlogs cvs_chess/CVSROOT cvs_chess/ptpapers" > ptpapers.cvs.tar.gz
tar -zxf ptpapers.cvs.tar.gz
mkdir repos
cd cvs2svn-2.4.0
./cvs2svn --default-eol=native --fallback-encoding=utf8 -s /home/cxh/src/cvs2/repos/ptpapers /home/cxh/src/cvs2/cvs_chess/

Problems

  • Why do is --default-eol=native not the default? My repository CVS repository is fine with regard to the binary mode, yet the tool ignores this? That is like having cp ignore null characters in files because they might cause a NullPointerException. Sheesh.
  • Problems with a file being in the Attic. This is a sign of corruption, see the cvs2svn faq. I just removed that attic file
  • Problems with encoding. Some of the files in ptII/ptolemy/domains/wireless/demo/CooperativeTerminals/Attic/ have the character \222. The fix is to use --fallback-encoding=utf_8. The command to restart is
 ./cvs2svn --pass CleanMetadataPass --dry-run -s /tmp/cxh/repos/ptII /tmp/cxh/cvs/ptI
  • vergil/graph/EditorDropTarget.java was still in svn, though it had been removed from cvs.
cd ptolemy/vergil
svn delete graph
svn commit -m "Removed vergil/graph from svn, cvs2svn failed" graph
  • ptII/vendors/jhdl/ptjhdllib/JHDL.jar did not get

added

Under Eclipse, a Cygwin ssh window flashes

With the default subeclipse installation I get a Cygwin ssh window flashing.

describes a fix which is to tell Subclipse to use SVNKit

C:\\Program Files\\TortoiseSVN\\bin\\TortoisePlink.exe

The backslashes are important However, with this method, I have to type in my passwd everytime or otherwise setup Tortoisesvn to do the right thing

Using Subversion with a different ssh user name

The svn --username command line option does not work with ssh. To use ssh with a different username, edit or create ~/.ssh/config with

Host source.eecs.berkeley.edu
User yourlog

where yourlog is the login on the remote server. (A tip of the hat to Chad Humphries, see Using a different username with SVN+SSH)

An alternative is to use your remote login in the URL, for example:

svn co svn+ssh://yourlog@source.eecs.berkeley.edu/chess/ptII/trunk ptII

To illustrate that --username does not work with, ssh I was able to set up svn to use a debugging ssh server as follows:

  • On the svn server, start up sshd in debug mode on a non-standard port: /usr/lib/ssh/sshd -d -p 29934
  • On the svn client, set SVN_SSH@ to use the same port: export SVN_SSH="ssh -p 29934"@@
  • On the svn client, try to connect as a user other than yourlog by su-ing or sudo-ing to another account (I chose apache) and running:

(:source) svn co --username yourlog svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk ptII (:sourceend:)

Sadly, the --username argument has no effect on ssh, the output on the svn server will say something like:

Keyboard-interactive (PAM) userauth failed[13] while authenticating: No account present for user
Failed keyboard-interactive for <invalid username> from 128.32.48.120 port 55524 ssh2

If I restart sshd on the server with /usr/lib/ssh/sshd -d -p 29934 and then run

ssh -p 29934 -l yourlog source svnserve -h

then the output on the svn server will say:

Accepted keyboard-interactive for yourlog from 128.32.48.120 port 55526 ssh2

Subclipse vs Subversion

Two Eclipse plugins: Subclipse and Subversion Subclipse

  • + Hosted on tigris.org, which is where Subversion is hosted.
  • - Team -> Share Project does not allow multiple projects to be added to svn

Subversive

  • + Hosted at the Eclipse site.
  • + Team -> Share Project does allow multiple projects to be added to svn

Chess svn setup design issues

  • Use svnserve instead of Apache. See http://svnbook.red-bean.com/en/1.4/svn.serverconfig.choosing.html#svn.serverconfig.choosing.svn-ssh
    • source.eecs does not currently run Apache for security reasons
    • We have ssh accounts already on source.eecs
  • Note that setting up anonymous svn access requires only one repository, per machine, unlike what we do with cvs. See One svnserve, multiple repositories
  • We use shortened urls like svn co svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk ptII. The way we do this is by creating links in / for chess and other directories. svnserve has a -r option, which sets the virtual root. We don't use that, because if we did, then paths in svn co svn+ssh://source.eecs.berkeley.edu/home/svn/chess/ptII/trunk would not work. Instead, we create links for in /.

Chess svn set up todo list

  • Document set up of user directories with svnserve
  • Nightly builds

Setting up anonymous access under Solaris

#!/sbin/sh
#
# SVNSERVE - manage the SVN server.
# From http://java.sun.com/developer/technicalArticles/javame/add_ons/
# See http://chess.eecs.berkeley.edu/ptolemy/wiki/Ptolemy/Subversion
#

case "$1" in
'start')
        if [ -x /usr/local/bin/svnserve ]; then
                /usr/local/bin/svnserve -d -r /export/home/svn
        fi
        ;;

'stop')
        if /usr/bin/pgrep svnserve >/dev/null 2>&1; then
                pkill svnserve
        fi
        ;;

'restart')
        /etc/init.d/svnserve stop
        /etc/init.d/svnserve start
        ;;

esac

exit 0
  • We then create a link
cd /etc/rc2.d
ln ../init.d/svnserve S98svnserve
  • Set up the firewall.
  1. Edit /etc/ipf/ipf.conf
# svn (subversion)
pass in quick on bge0 proto tcp from any to 128.32.48.234 port = 3690 flags S keep state group 100
pass in quick on bge0 proto udp from any to 128.32.48.234 port = 3690 flags S keep state group 100
  1. Update the firewall: ipf -Fa /etc/ipf/ipf.conf
  • Test the setup:
svn co svn://source.eecs.berkeley.edu/ptII/trunk

Setting up SVN Email

See How do I get email about each ptII SVN check in?

This is not well documented, so here goes.

The post-commit script is run after a commit. This script must be in the hooks/ directory of the repository on the server.

  1. Go to the hooks/ directory and copy post-commit.tmpl
    cd /home/svn/chess/ptIIprivi/hooks
    cp post-commit.tmpl post-commit
    chmod a+x post-commit
  2. Edit post-commit so that it looks like:
    #!/bin/sh                                                                      
    # See post-commit.tmpl                                                          

    REPOS="$1"
    REV="$2"

    /export/home/svn/chess/local/commit-email.pl -h source.eecs.berkeley.edu "$REPOS" "$REV" devels@yourdomain
    On the CHESS server, the address would be workgroup-cvs at chess.
  3. Copy commit-email.pl from the subversion sources:
    cp /export/home1/root/tools/subversion-1.4.5/tools/hook-scripts/commit-email.pl .

Setting up email for people with non-EECS addresses.

By default, source.eecs assumes everyone has an EECS.Berkeley.EDU address. If this is not the case, then as root on source:

  1. Edit /etc/mail/sendmail.cf and add the user account to the C{E}root line
  2. Edit /etc/mail/aliases and add an alias for the user account to the outside email address
  3. Run newaliases
  4. Restart sendmail with /usr/bin/pkill -HUP -x -u 0 sendmail
  5. Check the alias with
mconnect
expn username
quit

Email problem

Warning: 'post-commit' hook failed with error output:
/export/home/svn/chess/ptIIprivi/hooks/commit-email.pl: use of either `-h' or `--from' is mandatory when sending email using direct SMTP.

subversion: not sending emails any more suggests editing commit-email.pl and changing

#$sendmail = "/usr/sbin/sendmail";
$smtp_server = "127.0.0.1";          

to

$sendmail = "/usr/sbin/sendmail";
#$smtp_server = "127.0.0.1";

Setting up web access

Installing Apache

  1. Download Apache
  2. Configure Apache. The INSTALL doc from the subversion sources says:
    At the top of the httpd-2.0 tree:

    $ ./buildconf
    $ ./configure --enable-dav --enable-so --enable-maintainer-mode

    The first arg says to build mod_dav.

    The second arg says to enable shared module support which is needed
    for a typical compile of mod_dav_svn (see below).

    The third arg says to include debugging information. If you
    built Subversion with --enable-maintainer-mode, then you should
    do the same for Apache; there can be problems if one was
    compiled with debugging and the other without.

    Note: if you have multiple db versions installed on your system,
    Apache might link to a different one than Subversion, causing
    failures when accessing the repository through Apache. To prevent
    this from happening, you have to tell Apache which db version to
    use and where to find db. Add --with-dbm=db4 and
    --with-berkeley-db=/usr/local/BerkeleyDB.4.2 to the configure
    line. Make sure this is the same db as the one Subversion uses.
    This note assumes you have installed Berkeley DB 4.2.52
    at its default locations. For more info about the db requirement,
    see section I.5.

    You may also want to include other modules in your build. Add
    --enable-ssl to turn on SSL support, and --enable-deflate to turn on
    compression support, for example. Consult the Apache documentation
    for more details.
    The options I used were:
    ./configure --prefix=/usr/local/apache-2.2.16 --datadir=/home/www
    --enable-so --enable-ssl --enable-rewrite --with-ssl=/usr/local/ssl \
    --disable-ipv6 --with-included-apr --enable-dav
  3. Set the path so that /usr/bin/ln is in the path before /usr/ucb/ln, then make and install apache:
    export PATH=/usr/bin:${PATH}
    make >& make.out
    make install >& install.out
  4. Set up the startup scripts
    cd /etc/rc3.d
    ln -s ../init.d/apachectl S30apachectl
    ln -s ../init.d/apachectl K30apachectl
    cd ../rc0.d
    ln -s ../init.d/apachectl K30apachectl
    cd ../rc1.d
    ln -s ../init.d/apachectl K30apachectl
  5. Start up apache:
    /etc/init.d/apachectl start
  6. Verify that there is a web page at http://source.eecs.berkeley.edu
  7. Edit /usr/local/apache/conf/httpd.conf:
    # Subversion
    LoadModule dav_svn_module modules/mod_dav_svn.so

    <Location /svn/chess>
    #
    # Note: The following must must be present to support
    # starting without SSL on platforms with no /dev/random equivalent
    # but a statically compiled-in mod_ssl.
    #
    <IfModule ssl_module>
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin
    </IfModule>

    # Subversion
    LoadModule dav_svn_module modules/mod_dav_svn.so
    LoadModule authz_svn_module modules/mod_authz_svn.so

    <Location /svn/chess/ealpapers>
    DAV svn
    SVNPath /home/svn/chess/ealpapers
    #SVNParentPath /home/svn
    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /usr/local/apache/local/svn-chess-passwd
    Require valid-user
    # For any operations other than these, require an authenticated user.
    #<LimitExcept>
    # Require valid-user
    #</LimitExcept>
    # User svn
    # Group svn
    </Location>
    <Location /svn/chess/ptII>
    DAV svn
    SVNPath /home/svn/chess/ptII
    #SVNParentPath /home/svn
    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /usr/local/apache/local/svn-chess-passwd
    # For any operations other than these, require an authenticated user.
    <LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    </LimitExcept>
    # User svn
    # Group svn
    </Location>

    <Location /svn/chess/triq>
    DAV svn
    SVNPath /home/svn/chess/triq
    #SVNParentPath /home/svn
    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /usr/local/apache/local/svn-chess-passwd
    # For any operations other than these, require an authenticated user.
    <LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    </LimitExcept>
    # User svn
    # Group svn
    </Location>

    <Location /svn/kepler>
    DAV svn
    SVNPath /export/home2/svn/kepler
    #SVNParentPath /export/home2/svn
    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /usr/local/apache/local/svn-chess-passwd
    # For any operations other than these, require an authenticated user.
    <LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    </LimitExcept>
    # User svn
    # Group svn
    </Location>

    <Location /svn/chess/ojdcheck>
    DAV svn
    SVNPath /home/svn/chess/ojdcheck
    #SVNParentPath /export/home2/svn
    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /usr/local/apache/local/svn-chess-passwd
    # For any operations other than these, require an authenticated user.
    <LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    </LimitExcept>
    # User svn
    # Group svn
    </Location>


svn: Certificate is not issued by a trusted authority

The problem is that running svn sometimes not recognize authority.

TortoiseSVN does not have this problem. (Update: Windows 7 users have problems with: Tortoise SVN, try WinCVS instead.)

Versions of svn compiled using OpenSSH under Mac OS X and Solaris do have this problem.

The error is:

bash-3.2$ svn co https://source.eecs.berkeley.edu/svn/chess/ptII/trunk ptII
Error validating server certificate for 'https://source.eecs.berkeley.edu:443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: source.eecs.berkeley.edu
- Valid: from Tue, 15 Jul 2008 08:00:00 GMT until Fri, 16 Jul 2010 07:59:59 GMT
- Issuer: Terms of use at https://www.verisign.com/rpa (c)05, VeriSign Trust Network, VeriSign, Inc., US
- Fingerprint: da:ff:45:c9:df:6f:b5:73:ec:9c:f8:a7:cc:93:08:84:22:a8:78:3d
(R)eject, accept (t)emporarily or accept (p)ermanently?

We could just blindly permanently accept the certificate, but this is really wrong since it is bad security hygiene. Finding the cause of the problem and possible solutions is of interest. Below are two workarounds, see also Ptolemy Bugzilla:139 Convert the Ptolemy II repository from CVS to SVN

Going to https://source.eecs.berkeley.edu/svn/chess/ptII/trunk in a web browser such as Firefox and examining the certificate does not indicate an error.

Editing ~/.subversion/servers

Kevin Miller's Blog Subversion and Certificate Authorities discusses editing ~/.subversion/servers and also a couple of patches to svn-1.5.

See also Making SVN trust a new root CA certificate

To modify ~/.subversion/servers, download a copy of the missing Certificate Authority (CA) file as follows

  1. In Firefox, go to https://source.eecs.berkeley.edu
  2. Click on the lock in the bottom right
  3. In the Page Info window, select the Security Tab and then select View Certificate
  4. Select Details and then select the top level CA, which will be something like Builtin Object Token VeriSign Class 3 Public Primary CA
  5. Export the file as an X.509 PEM file
  6. Use that file in ~/.subversion/servers for the ssl-authority-files value, for example:
    ssl-authority-files=/Users/cxh/tmp/BuiltinObjectToken:VerisignClass3PublicPrimaryCertificationAuthority

However, the problem here is that we are downloading the CA file from the website we are trying to verify. Another workaround is to download the root CAs from Verisign. The file we are interested in is PCA3ss_v4.509.

openssl x509 -inform DER -in PCA3ss_v4.509 -out PCA3ss_v4.pem -outform PEM

To view the contents of the CA file:

openssl x509 -text -in PCA3ss_v4.pem

Then adjust ~/.subversion/servers:

ssl-authority-files=/Users/cxh/tmp/PCA3ss_v4.pem

and do the checkout. At least here, we have the authority file downloaded from Verisign, so we can be fairly certain that svn repository has a legitimate certificate.

Fixing OpenSSL

It turns out that the problem is that svn is using OpenSSL. The OpenSSL FAQ question How can I set up a bundle of commercial root CA certificates? says

The OpenSSL software is shipped without any root CA certificate as the OpenSSL project does not have any policy on including or excluding any specific CA and does not intend to set up such a policy. Deciding about which CAs to support is up to application developers or administrators.
Other projects do have other policies so you can for example extract the CA bundle used by Mozilla and/or modssl as described in this article:

However, an updated version of that script can be found at http://www.opensource.apple.com/darwinsource/Current/apache_mod_ssl-690/mod_ssl/pkg.sslcfg/ca-bundle.pl To run this, download ca-bundle.pl and do

perl ca-bundle.pl > ca-bundle.crt

Running:

openssl s_client -connect source.eecs.berkeley.edu:443 -CAfile ca-bundle.crt

will return 0, which is correct.

See also http://svn.haxx.se/dev/archive-2006-06/0547.shtml - a note about this problem in the svn devel log, which points out http://curl.haxx.se/docs/caextract.html which includes cacert.pem.

Here's how to set up openssl on the Mac so that it does not prompt for accepting the certificate:

  1. Download the root CAs from Verisign. The file we are interested in is PCA3ss_v4.509.
    openssl x509 -inform DER -in PCA3ss_v4.509 -out PCA3ss_v4.pem -outform PEM
  2. To view the contents of the CA
  3. Become root with:
    sudo -
    and copy PCA3ss_v4.pem:
    cp PCA3ss_v4.pem /System/Library/OpenSSL/certs
    For details, see Chris Adams' Installing Custom Certificate Authorities for OpenSSL
  4. Run the c_rehash command from OpenSSL:
    c_rehash /System/Library/OpenSSL/certs/

The certificate authority will be present and svn users should not be prompted to accept the certificate from source.eecs. Also, the

openssl s_client -connect source.eecs.berkeley.edu:443 -CApath /System/Library/OpenSSL/certs/

should return

Verify return code: 0 (ok)

It would be nice if we could import all the Commercial CAs in one fell swoop.

Firewall and Proxy Problems

If you get:

RA layer request failed svn: OPTIONS of 'https://source.eecs.berkeley.edu/svn/chess': Could not resolve hostname `source.eecs.berkeley.edu': The requested name is valid and was found in the database, but it does not have the correct associated data being resolved for.

Or if the Apache error_log says Error writing base64 data: Connection reset by peer [500, #131] (the see http://svn.haxx.se/users/archive-2007-09/0388.shtml)

Then the problem is likely with a corporate firewall.

The first thing to try is to point your web browser at the SVN url. For example, try opening https://source.eecs.berkeley.edu/svn/chess/ptII/trunk, you should get a listing of the files at that location. If you do, then the problem is that the SVN client needs to have its proxy set, see the Subversion FAQ for details.

Under TortoiseSVN, you could try setting the TortoiseSVN proxy to match the proxy of your web browser. If you have a proxy autoconfiguration URL, then try entering the URL into your browser and analyzing the code to determine the name of your proxy machine.

Another thing to try is to edit the \Documents and Settings\user\Application Data\Subversion\servers file and add

http-proxy-host = <proxyname>
http-proxy-port = <proxyport>

(Many Thanks to Thomas Mandl and Thomas Huining Feng for the above)

Another possibility is discussed at http://groups.google.com/group/pyjamas-dev/browse_thread/thread/682bc1991c2326b6

also - are you behind a firewall, in a corporate environment of some kind? it's very common for corporate firewalls to block HTTPS or to require NTLMSSP authentication on HTTPS.
if that's the case, you will need either:
or:
and then you must do export http_proxy=http://127.0.0.1:NNNN where NNNN is the port that the proxy is running on.

Keyword Substitution and end of line issues

To enable keyword substition, such as $Id$ being changed to $Id: bif4.txt 7 2009-03-05 01:13:48Z cxh $, you need to set up ~/.subversion/config so that each file extension has the appropriate settings. See http://chess.eecs.berkeley.edu/ptexternal/nightly/doc/coding/eclipse.htm#Subversive for details which involve adding ~/ptII/doc/coding/svn-config-auto-props.txt to ~/.subversion/config

Why is it necessary to add have a pattern for every file? The answer is that Subversion decides that everything is a binary file and that it is safer to check things in and not modify them. However, there should be a repository wide way to set up config instead of requiring each user to do so.

Another limitation is that the enable-auto-props=yes line must be before the [auto-props] line. This is really lame. It is a natural mistake to have enable-auto-props=yes after [auto-props]. Instead, auto-props should be enabled if [auto-props]] is uncommented.

To test this out, if you have read/write permission to the source.eecs.berkeley.edu SVN repositories, try:

bash-3.2$ svn co svn+ssh://source.eecs.berkeley.edu/chess/svntest
A    svntest/README.txt
Checked out revision 7.
bash-3.2$ cd svntest
bash-3.2$ echo '$Id$' > testfile.txt
bash-3.2$ svn add testfile.txt
A         testfile.txt
bash-3.2$ svn commit -m "A test for svn keywords: testfile.txt " testfile.txt
Adding         testfile.txt
Transmitting file data .
Committed revision 8.
bash-3.2$ cat testfile.txt
$Id: testfile.txt 8 2009-03-05 01:27:44Z cxh $
bash-3.2$ svn proplist testfile.txt
Properties on 'testfile.txt':
  svn:keywords
  svn:eol-style

Note that testfile.txt had $Id$ properly substituted. If it had only $Id$ and not $Id: testfile.txt 8 2009-03-05 01:27:44Z cxh $ then keywords were not being substituted and that ~/.subversion/config had a problem.

Checking Keyword Substitution

To check keyword substitution on a file:

bash-3.2$ svn proplist README.txt
Properties on 'README.txt':
  svn:keywords
  svn:eol-style
bash-3.2$ svn propget svn:keywords README.txt
Author Date Id Revision
bash-3.2$ svn propget svn:eol-style README.txt
native

See ptII/doc/coding/releasemgt.htm for information about how to use ptII/adm/bin/svnpropcheck.

Fixing Keyword Substitution

To set the keywords:

svn propset svn:keywords "Author Date Id Revision" filename
svn propset svn:eol-style native filename

Keyword problem: one line in auto-props masks another

http://subversion.tigris.org/issues/show_bug.cgi?id=2036 describes a four year old bug where the ~/.subversion/config file has problems with lines masking each other. For example, if ~/.subversion/config has:

makefile = svn:eol-style=native;svn:keywords=Author Date Id Revision
Makefile = svn:eol-style=native;svn:keywords=Author Date Id Revision

then if the user creates a makefile, it will have keywords set, but if the user creates a Makefile it will not have the keywords set. If course, this only matters on real file systems such as Mac OS X or Unix. Windows users have the oddly antiquated case insensitive, case preserving file system.

Problem creating two filenames that differ only in case

Under MacOS with svn-1.4.4:

bash-3.2$ echo '$Id$' > makefile
bash-3.2$ echo '$Id$' > Makefile
bash-3.2$ svn add makefile Makefile
A         makefile
A         Makefile
bash-3.2$ svn commit -m "config has makefile first" makefile Makefile
Adding         Makefile
Adding         makefile
svn: Commit failed (details follow):
svn: Can't copy '/Users/cxh/src/svntest/.svn/props/makefile.svn-work'
   to '
/Users/cxh/src/svntest/.svn/tmp/props/makefile.svn-work.tmp': No such file or directory

However, the same command works under Solaris with svn-1.4.5

Problems with Eclipse and ssh

The problem was that if a user had an incorrect password, then the ssh server would block ssh connections:

cxh@carson 37% ssh -v xxx.eecs.berkeley.edu
>>> Sun_SSH_1.1, SSH protocols 1.5/2.0, OpenSSL 0x0090704f
>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>> debug1: Rhosts Authentication disabled, originating port will not be
>>> trusted.
>>> debug1: ssh_connect: needpriv 0
>>> debug1: Connecting to source.eecs.berkeley.edu [128.32.48.XX] port 22.
>>> debug1: Connection established.
>>> debug1: identity file /home/eecs/cxh/.ssh/identity type -1
>>> debug1: identity file /home/eecs/cxh/.ssh/id_rsa type 1
>>> debug1: identity file /home/eecs/cxh/.ssh/id_dsa type -1
>>> ssh_exchange_identification: Connection closed by remote host
>>> debug1: Calling cleanup 0x341a0(0x0)
>>> cxh@carson 38%

netstat on source's console shows that soda1am.Equip has a bunch of TIME_WAIT connections on sshd. soda1am.Equip is the machine through which the connections from AirBears arrive.

Looking around on the net finds http://archive.netbsd.se/?ml=openssh-unix-dev&a=2007-10&t=5430083 which suggested editing /etc/ssh/sshd_config and changing MaxStartups.

However, the real question is: Why does this occur?

It turns out one of our Eclipse users was using either Subclipse or Subversive to connect via ssh. They had the wrong password, so there were many bogus connections.

The account that was causing the problem was using Eclipse on the Mac with these versions of Subversive: (:source) Subversive SVN Connectors 2.1.0 Subversive SVN Team Provider (Inclubation) 0.7.3 SVNKit 1.2.2 Implementation (Optional) 0.7.3 (:sourceend:)

I edited /etc/ssh/sshd_config:

# The maximum number of concurrent unauthenticated connections to sshd.
# start:rate:full see sshd(1) for more information.
# The default is 10 unauthenticated clients.
#MaxStartups 10:30:60
# cxh: Updated to 20 to avoid
# "ssh_exchange_identification: Connection closed by remote host" messages
MaxStartups 20:30:60

# Default is 600 seconds. 0 means no time limit.
#LoginGraceTime 600
# cxh: changed to 60 seconds to avoid lock out
LoginGraceTime 60

# Maximum number of retries for authentication
# Default is 6. Default (if unset) for MaxAuthTriesLog is MaxAuthTries / 2
#MaxAuthTries 5
# cxh: changed to three tries
MaxAuthTries 3
MaxAuthTriesLog 0

Svnsync

We use svnsync on source to get a copy of the Kepler tree. The svnsync Unix user has a script called /export/home2/svn/synckepler

#!/bin/sh

/usr/local/etc/rotate_log -L /export/home2/svn -n 20 svnsync.out
/usr/local/bin/svnsync sync file:///export/home2/svn/kepler 2>&1 > svnsync.out

The script is run from a cron job every 10 minutes.

svnsync: Couldn't get lock on destination repos after 10 attempts

If svnsync fails with svnsync: Couldn't get lock on destination repos after 10 attempts, then run

su - svnsync
/usr/local/bin/svn propdel svn:sync-lock --revprop -r 0 file:///export/home2/svn/kepler

See http://malsserver.blogspot.com/2008/12/failed-to-get-lock-on-destination-repos.html

Get repository folder children operation failed

After upgrading the svn server to 1.7.2 on source, Eclipse Indigo SR1 (M20110909-1335) with "SVNKit 1.3.5 Implementation (Optional) 2.2.2.I20110819-1700 org.polarion.eclipse.team.svn.connector.svnkit16.feature.group Polarion Software" was failing when the URL was svn+ssh://source.eecs.berkeley.edu/chess/ptII.

The problem is that clicking on trunk results in <An error occurred whil accessing the repository entry >. The stack trace is

org.eclipse.team.svn.core.connector.SVNConnectorException: svn: URL 'svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk' non-existent in that revision
        at org.polarion.team.svn.connector.svnkit.SVNKitConnector.handleClientException(SVNKitConnector.java:1415)
        at org.polarion.team.svn.connector.svnkit.SVNKitConnector.list(SVNKitConnector.java:1264)
        at org.eclipse.team.svn.core.extension.factory.ThreadNameModifier.list(ThreadNameModifier.java:278)
        at org.eclipse.team.svn.core.utility.SVNUtility.list(SVNUtility.java:335)
        at org.eclipse.team.svn.core.svnstorage.SVNRepositoryContainer.getChildren(SVNRepositoryContainer.java:79)
        at org.eclipse.team.svn.core.operation.remote.GetRemoteFolderChildrenOperation.runImpl(GetRemoteFolderChildrenOperation.java:75)
        at org.eclipse.team.svn.core.operation.AbstractActionOperation.run(AbstractActionOperation.java:81)
        at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTask(ProgressMonitorUtility.java:104)
        at org.eclipse.team.svn.core.operation.CompositeOperation.runImpl(CompositeOperation.java:95)
        at org.eclipse.team.svn.core.operation.AbstractActionOperation.run(AbstractActionOperation.java:81)
        at org.eclipse.team.svn.core.operation.LoggedOperation.run(LoggedOperation.java:39)
        at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTask(ProgressMonitorUtility.java:104)
        at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTaskExternal(ProgressMonitorUtility.java:90)
        at org.eclipse.team.svn.ui.utility.DefaultCancellableOperationWrapper.run(DefaultCancellableOperationWrapper.java:55)
        at org.eclipse.team.svn.ui.utility.ScheduledOperationWrapper.run(ScheduledOperationWrapper.java:37)
        at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Caused by: org.tigris.subversion.javahl.ClientException: svn: URL 'svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk' non-existent in that revision
        at org.tigris.subversion.javahl.JavaHLObjectFactory.throwException(JavaHLObjectFactory.java:779)
        at org.tmatesoft.svn.core.javahl.SVNClientImpl.throwException(SVNClientImpl.java:1924)
        at org.tmatesoft.svn.core.javahl.SVNClientImpl.list(SVNClientImpl.java:365)
        at org.tmatesoft.svn.core.javahl.SVNClientImpl.list(SVNClientImpl.java:337)
        at org.polarion.team.svn.connector.svnkit.SVNKitConnector.list(SVNKitConnector.java:1253)
        ... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: URL 'svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk' non-existent in that revision
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
        at org.tmatesoft.svn.core.wc.SVNLogClient.doList(SVNLogClient.java:1334)
        at org.tmatesoft.svn.core.wc.SVNLogClient.doList(SVNLogClient.java:1217)
        at org.tmatesoft.svn.core.javahl.SVNClientImpl.list(SVNClientImpl.java:350)
        ... 16 more

svnserve Processes

It turns out that there were lots of svnserve processes on source.eecs.berkeley.edu. Details may be found at http://svn.haxx.se/tsvnusers/archive-2007-09/0056.shtml and http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2407949

My solution was to write a script that killed those processes. However, this script was not super-reliable as it was still possible to run out of swap between times the script ran.

#!/bin/sh
# If there are more than 100 svnserve processes, then kill them.
#
process=svnserve
max=100
mailto=root

# For testing
#process=http
#max=2
#mailto=cxh

processes=`/usr/ucb/ps -auxgww | grep $process | wc -l`
if [ "$processes" -gt $max ]; then
   maillog=/tmp/killsvnserv.$$
   count=$processes
   echo "Starting with $count $process $processes" > $maillog
   netstat >> $maillog
   while [ "$count" -gt $max ];
   do
      pkill -o svnserve >> $maillog
      #echo $count >> $maillog
      count=`expr $count - 1`
   done
   /etc/rc2.d/S98svnserve start >> $maillog
   /usr/ucb/ps -auxgwww | grep $process >> $maillog
   /usr/ucb/Mail -s "Source.eecs had $processes $process processes" $mailto < $maillog
fi

Unfortunately, this script is not super-reliable in that snv: connections will sometimes start to fail because rerunning S98svnserve fails.

The cause was Kepler users with IntelliJ were setting up svn in such a way that IntelliJ made many connections to the server.

Sean Riddle writes:

"From my Kepler directory, I just run:
svn ps svn:ignore "src" ptolemy/
to except ptolemy from version control. "

Sean also pointed out that http://youtrack.jetbrains.net/issue/IDEA-56714 describes the issue.

svn: This client is too old to work with working copy '.'; please get a newer Subversion client

If you use Subversion with Eclipse and install a Subversion plugin that is newer than the command line svn command, then when you run svn, you will get svn: This client is too old to work with working copy '.'; please get a newer Subversion client. The solution is to update the version of svn on your machine.

Subversion downloads may be found at http://subversion.tigris.org/project_packages.html

Mac svn details

I have an updated version of svn installed at /opt/subversion/bin/svn.

My ~/.MacOSX/environment.plist file includes that version in the path with:

{
"DYLD_LIBRARY_PATH" = "/Users/cxh/ptII/vendors/misc/dbxml-2.5.16/install/lib";
"PATH" = "/Users/cxh/ptII/vendors/misc/dbxml-2.5.16/install/bin:/Users/cxh/bin:/Users/cxh/ptII/bin:/usr/local/apache-maven/bin:/opt/subversion/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/usr/X11/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin:/usr/local/texlive/2007/bin";
}

You can edit environment.plist with either a text editor or with /Developer/Application/Utilities/Property List Editor.app

Mac: svn killed by signal 15

If running svn co fails on the Mac with Killed by signal 15, then edit ~/.subversion/config and set ssh to $SVN_SSH ssh -q

See http://tech.barwick.de/development/svn-subversion-fixing-killed-by-signal-15-error.html and http://subversion.tigris.org/issues/show_bug.cgi?id=2580

(Thanks to Gilles Lasnier)

Eclipse: svn: Compression of svndiff data failed

Eclipse was failing with:

Checkout operation for
'svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk' failed.
svn: Compression of svndiff data failed

It could be that SVNkit needs to be updated.

  1. Go to the SVN Eclipse instructions at: http://www.eclipse.org/subversive/downloads.php
  2. Get the Polarion update URL from http://www.polarion.com/products/svn/subversive/download.php?utm_source=eclipse.org&utm_medium=link&utm_campaign=subversive As of 9/10, for Helios, it was: http://community.polarion.com/projects/subversive/download/eclipse/2.0/helios-site/ Other versions of Eclipse should use the appropriate update site
  3. In Eclipse, do Help | Update Software and then Add the http://community.polarion.com/projects/subversive/download/eclipse/2.0/helios-site/ URL
  4. Select the SVNkit appropriate for your version of Java. To check your version of Java, run svn --version. See http://www.polarion.com/products/svn/subversive/connector_discovery.php?utm_source=eclipse.org&utm_medium=link&utm_campaign=subversive for details.
  5. Complete the installation.

TortoiseSVN

Summary: Windows 7 users have problems with Tortoise SVN, try http://cvsgui.sourceforge.net/ WinCVS instead.

Details: In Windows 7, Microsoft changed how some registry entries are handled. TortoiseSVN was using these entries so that in the Windows Explorer file browser, users could right click on a file or directory and perform SVN operations. Apparently, there are a limited number of images that can be used. See http://stackoverflow.com/questions/1057734/tortoisesvn-icons-not-showing-up-under-windows-7

Merging Two SVN Repositories

Dump with:

svnadmin dump /export/home/svn/chess/ealpapers > ealpapers.dump

I had to create trunk/ptpapers/ealpapers and commit it.

Then do the merge:

svnadmin load --parent-dir trunk/ptpapers/ealpapers /home/cxh/src/cvs/ptpapers < ealpapers.dump

svn externals property: How to check out a repo within another repo

To get accessors to work, we need to do (:source:) cd $PTII/org/terraswarm/accessor svn co https://repo.eecs.berkeley.edu/svn/projects/terraswarm/accessors/trunk/accessors

Summary: I did not find a way to do this. The workaround is to have configure check out the repo, which I will do now.

Details:

See https://www.terraswarm.org/accessors/wiki/Version0/Implement as to why we need to do this.

SVN supports externals. http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html states:

"Fortunately, Subversion provides support for externals definitions. An externals definition is a mapping of a local directory to the URL—and ideally a particular revision—of a versioned directory. In Subversion, you declare externals definitions in groups using the svn:externals property. You can create or modify this property using svn propset or svn propedit (see the section called “Manipulating Properties”). It can be set on any versioned directory, and its value describes both the external repository location and the client-side directory to which that location should be checked out."

So, in theory, we could add a svn:externals property to the ptII svn repo.

However, there are at least two issues here:

  1. If we set up the ptII repo so that it has the svn:external property set to check out https://repo.eecs.berkeley.edu/svn/projects/terraswarm/accessors/trunk/accessors, then this will only work for someone who has access to that repo. It will not work for anonymous users who are checking out the ptII tree who do not have access to the accessor repo. If we tell anonymous users to use "svn co --ignore-externals https://repo.eecs.berkeley.edu/svn-anon/projects/eal/ptII/trunk ptII", then the svn command would work, though the accessors would not.
  2. If we had the accessors repo anonymously readable, then we will have problems because the urls for the read/write and read only versions are different.

The property can have only one value.

The above is partly an artifact of repo.eecs, but it is a general problem. I don't think repo.eecs can be modified to better handle this.

One idea was to do something complex with HTTPS and redirects, but that is not likely to work. I tried it before and reviewing the web shows that it would work with the browser, but svn will want the user to run svn relocate.

Thus, in the near term, we should have $PTII/configure check out https://repo.eecs.berkeley.edu/svn/projects/terraswarm/accessors/trunk/accessors.

In the longer term, we should make https://repo.eecs.berkeley.edu/svn/projects/terraswarm/accessors/trunk/accessors publicly readable. Perhaps $PTII/configure could check out the proper URL.

Ideally, the code that requires that https://repo.eecs.berkeley.edu/svn/projects/terraswarm/accessors/trunk/accessors be checked out would have a good error message that would tell the user what to do.

Edit - History - Print - Recent Changes - Search
Page last modified on June 29, 2015, at 10:38 pm