In November, 2012, the problem was that the old crufty version of mailman requires an old crufty version of Python.

Running
<pre>
[mailman@moog bak]$ which python
/usr/local/pythonmailman/bin/python
[mailman@moog bak]$ cd /export/home/mailman_4
[mailman@moog mailman_4]$ bin/list_lists
Traceback (most recent call last):
  File "bin/list_lists", line 108, in ?
    main()
  File "bin/list_lists", line 86, in main
    mlist = MailList.MailList(n, lock=0)
  File "/home/mailman_4/Mailman/MailList.py", line 79, in __init__
    self.Load()
  File "/home/mailman_4/Mailman/MailList.py", line 903, in Load
    raise Errors.MMCorruptListDatabaseError, e
Mailman.Errors.MMCorruptListDatabaseError: bad marshal data
[mailman@moog mailman_4]$ 
</pre>

Apparently, the .db files got updated with a newer version of Python?

One solution is to recreate remove the lists, recreate them and then go to the website and repopulate them.
<ol>
 <li> Remove the lists.  Become root and do
<pre>
bin/rmlist -a trust
bin/rmlist -a trust_archive
bin/rmlist -a trustlocal
bin/rmlist -a trustlocal_archive
bin/rmlist -a trustseminar
bin/rmlist -a trustseminar_archive
bin/rmlist -a execboard
bin/rmlist -a execboard_archive
bin/rmlist -a webmaster
bin/rmlist -a webmaster_archive
bin/rmlist -a reu
bin/rmlist -a reu_archive
bin/rmlist -a reu-devel
bin/rmlist -a reu-devel_archive
</pre>
</li>
<li> Make sure that there are no locks in  the <code>locks/</code> directory.

<li> Then recreate the lists. 
<pre>
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 trust trust cxh@eecs.berkeley.edu 771dab3f1d41d3af63e1c1366f0b20f4 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 trustlocal trustlocal cxh@eecs.berkeley.edu ab6830ad0d0d8d5b4af4197a28fabafa trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 trustseminar trustseminar cxh@eecs.berkeley.edu f8de2c6d3e1c4434ce332c74b41d1097 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 execboard execboard cxh@eecs.berkeley.edu 7216f298316b6052b9a8bac4256fa4a7 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 webmaster webmaster cxh@eecs.berkeley.edu 46755f8bfb93ce397d48e6b7bc0e6ece trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 trustadmin trustadmin cxh@eecs.berkeley.edu eaa6e3517969e57f43bac1cde74e367c trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 securit securit cxh@eecs.berkeley.edu d7e714acec2247443037429c0c54277b trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 securit securit-devel cxh@eecs.berkeley.edu 3b7cfb495b3122671f2f809eb8613d3b trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 star star-devel cxh@eecs.berkeley.edu 734c95d83d66db94779db7501ae6d057 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 wise wise cxh@eecs.berkeley.edu b4764364bf6ab530451fb7ec7ba71932 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 wise wise-devel cxh@eecs.berkeley.edu 77fdd95bfd2dff8988a898b583884425 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 reu reu cxh@eecs.berkeley.edu 4004a1cb51103463f41079d736ddadb9 trust "www.truststc.org www.truststc.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_4 reu reu-devel cxh@eecs.berkeley.edu af5f42b6f6d92f81a935e55a964a8e0e trust "www.truststc.org www.truststc.org"
</pre>
<li>Then go to <a href="https://www.truststc.org/options/power/list">https://www.truststc.org/options/power/list</a>
and populate each list</li>
</ol>

Mailing lists for icyphy:
<pre>
 /home/www/php/bin/newgrouplist  -t -h /home/mailman_6 webmaster webmaster cxh@eecs.berkeley.edu 46755f8bfb93ce397d48e6b7bc0e6ece icyphy "icyphy.org icyphy.org"
/home/www/php/bin/newgrouplist  -t -h /home/mailman_6 icyphy icyphy cxh@eecs.berkeley.edu 71397a467b0a3b076baa137f1e72f0fc icyphy "icyphy.org icyphy.org"
</pre>

<h2>Solution</h2>
It turns out that RHEL supports setting the PATH environment variable in the crontab
<pre>
PATH=/usr/local/pythonmailman/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/lib/mailman/bin
</pre>

<h2>Recovery</h2>
<p>We are now backing up the mailman config files.</p>

<p>Mailman runs a cron job <code>~www/php/mailman/mailmansaveconfigs</code> that saves the *.db files
to a tar file on an hourly basis.</p>

<p>If the bad marshal data error appears, use
<code>lastcomm</code> to track down which script is
running the wrong python, fix it and then restore 
the configuration files by first checking for when
the tar files changed:
<pre>
cd /home/mailman_4
ls -ltr bak
md5sum bak/*
</pre>
Then, as root, extract the tar file:
<pre>
tar -xf bak/configs.tar.1
</pre>

<h2>A fix?</h2>
In August, 2013, I noticed that when a user tried to log in and their account had expired, the mailing list database would get corrupted.  I could see that this code in user.inc.php was getting run:
<pre>
$cmd = "${mailmanHome}/bin/remove_members $listName $lcEmail";
$returnMessage .= "Unsubscribing $lcEmail from $listName...<br/>";
$returnMessage .= shell_exec($cmd);
</pre>
I checked the path with: <a href="https://embedded.eecs.berkeley.edu/options/power/phpinfo">https://embedded.eecs.berkeley.edu/options/power/phpinfo</a> and indeed <code>/usr/local/pythonmail/bin</code> was not in the path.

<p>The <code>/etc/init.d/gsrc</code> script runs <code>/etc/init.d/httpd</code>, which reads <code>/etc/sysconfig/httpd</code>.

<p>I updated <code>/etc/sysconfig/httpd</code> to include:
<pre>
export PATH=/usr/local/pythonmailman/bin:$PATH:$HOME/bin
</pre>

and then, as root, stopped and started with:
<pre>
/etc/init.d/gsrc stop
/etc/init.d/gsrc start
</pre>

Then, go to <a href="https://embedded.eecs.berkeley.edu/options/power/phpinfo">https://embedded.eecs.berkeley.edu/options/power/phpinfo</a> and search for path.

<pre>
 PATH 	/usr/local/pythonmailman/bin:/sbin:/usr/sbin:/bin:/usr/bin:/root/bin 
</pre>