Free Source Code: Paper Review/Grade System

By Michael McLennan 1998, De Clarke 1999


If you are running a conference or other refereed event, one of your challenges is to maintain the list of persons on the program committee, assign their list of papers to be reviewed, and then collect and summarize their reviews. When we faced this problem for the Tcl/Tk Usenix Conference '98, Michael McLennan (co-chair) slapped together a twisty little maze of tcl scripts, gdb files, and HTML that provided a web-based interface to all this housekeeping. In '99, as co-chair for the 2000 conference, I inherited the code and (of course, who can resist?) re-wrote a bunch of it for easier porting. I threw out Mike's dynamic HTML mechanism and used Don Libes' CGI-tcl because imho this improved legibility and maintainability.

This code is freely available with absolutely NO support or guarantees. It is not warranted for any e-commerce or other secure application. Its security mechanism is a big joke: apache httpd password-per-directory protection via apache access.conf. However, if you understand that you'll need to do some hacking to get started, and that you should definitely not be keeping any industrial or military secrets online by this mechanism, you may find it useful for, e.g., committee grading for papers or projects; refereed conferences or anthologies; or (who knows) movie reviews or a restaurant guide!


Overview

The fundamental idea is this: to maintain using gdb (I personally would have used Sybase or Postgres, but I have to admit gdb is Q&D to install) several little tables of information, such as There are two classes of "user" for this suite. One is the Admin, who can maintain the list of reviewers and assign papers to them (these reviewers are called the "Program Committee" in the source, because the code was written for a conference, but you can change all the labelling on the HTML forms as you please). The Admin of course is also responsible for placing the materials to be reviewed online (there's a form for doing this as well) and associating them correctly with their authors.

The other user class is the Reviewer.

The Reviewer can see but not touch information such as the list of other Reviewers, the list of papers, and who's assigned to read what. The Reviewer can read other people's reviews as well; there's no access control mechanism segregating reviewers from one another, so don't try to use this system to build your own ebay :-) A malicious user could easily mess up the works, so restrict use of this product to a group of people with strong mutual trust and maturity.

The Reviewer can submit a review in one of two ways: by filling out an online Review Form (HTML), or by downloading a plain-text Review Form, editing it, then mailing it back to a pseudo-recipient at the Admin's site. To make the mail-based reviewing work you must be able to get at your local mail hub's aliases, or mess about with a fake user account and .forward files. NOTE that the ingestion s'ware for the incoming email message is very, very picky about format. You have to dot your i's and mind your whitespace carefully. Most people have no trouble with this, but some reviewers persist in deleting the lines marked "don't delete this text."

The Web-based Review form deposits the submitted info directly into the gdb "database". The incoming mail file, however, is processed by a mail receiver robot which eventually does the same. BTW, the review text is stored as a text file, but the meta-data about the review go into the "database"; no large text blobs in the database.


Screen Shots (sorry, PostScript)

Note: the looknfeel you see here is my doing. Mike's version a couple of years ago looked quite different! You can customize the logos, etc however you like. I can't really show you much more, because there are real review data in this copy which are moderately confidential :-)

Requirements

To make this suite of software work you will need:
Tcl/Tk
go to Scriptics and tcltk.com to find out about Tcl/Tk and get the source. It's free.
TclX
Get this also from Scriptics or tcltk.com, see above.
CGItcl
This is a pureTcl extension for cgi scripting (and very nice too) by Don Libes. You should be able to find it via tcltk.com, or use your favourite search engine. It's probably been updated quite a bit since I last got v 0.6, but knowing Don's meticulous coding style I bet nothing has been broken.
libgdbm.a, libgdbm.so, etc.
If you are running Linux you already have these libs in /usr/lib. If you are running some other OS, you'll probably have to go out and find the source for libgdbm and build/install it yourself.
The Tclgdbm extension
I'm offering this here in case it is hard to find. I don't know whether any other version of Tclgdbm work with this software suite, and I am sure you would rather not find out the hard way; this version was known to work.
One more thing . . .
You will need enough privs on your http server to set up CGI scripts.
When you have all this stuff built and working, you should be ready to play. The next step is to get the actual code, offered here as 2 gzipped tarballs. Each will make its own subdir. Keep them separate.
The CGI Scripts and data
This is stuff that goes under the cgi-bin dir on your http server.
Working Web Directory, Scripts, Data
This is stuff that lives in some normal web subtree out in homepage space.
In each tarball I have provided a README which I hope will make your initial experience less bewildering than mine was :-) Feel free to send me questions, I'll try to help if I have a spare moment, but don't expect a very quick turnaround. Remember this is UNSUPPORTED. Having said all that . . . enjoy! it's actually pretty nifty stuff and can make the coordination of a refereed event easier for you.

NOTE

The UC Regents don't care if my code works for you or not, so there are no guarantees. Read the COPYRIGHT statement on anything you download from us. Everything you get from us is free. As software ought to be. :-)

de@ucolick.org
webmaster@ucolick.org
De Clarke
UCO/Lick Observatory
University of California
Santa Cruz, CA 95064
Tel: +1 408 459 2630
Fax: +1 408 454 9863