Free Source Code: Paper Review/Grade System
By Michael McLennan 1998, De Clarke 1999
If you are running a conference or other refereed event,
one of your challenges is to maintain the list of persons
on the program committee, assign their list of papers
to be reviewed, and then collect and summarize their
reviews. When we faced this problem for the Tcl/Tk
Usenix Conference '98, Michael McLennan (co-chair)
slapped together a twisty little maze of tcl scripts,
gdb files, and HTML that provided a web-based interface
to all this housekeeping. In '99, as co-chair for the
2000 conference, I inherited the code and (of course,
who can resist?) re-wrote a bunch of it for easier
porting. I threw out Mike's dynamic HTML mechanism
and used Don Libes' CGI-tcl because imho this improved
legibility and maintainability.
This code is freely available with absolutely
NO support or guarantees. It is not warranted for
any e-commerce or other secure application. Its security
mechanism is a big joke: apache httpd password-per-directory
protection via apache access.conf. However, if you understand
that you'll need to do some hacking to get started, and that
you should definitely not be keeping any industrial or
military secrets online by this mechanism, you may find
it useful for, e.g., committee grading for papers or
projects; refereed conferences or anthologies; or (who
knows) movie reviews or a restaurant guide!
Overview
The fundamental idea is this: to maintain using gdb
(I personally would have used Sybase or Postgres, but I
have to admit gdb is Q&D to install) several little
tables of information, such as
- Reviewers
- Authors
- Papers by authors
- Reviews by reviewers (of papers by authors)
- Grades of papers
There are two classes of "user" for this suite. One is
the Admin, who can maintain the list of reviewers and
assign papers to them (these reviewers are called the
"Program Committee" in the source, because the code was
written for a conference, but you can change all the
labelling on the HTML forms as you please). The Admin
of course is also responsible for placing the materials
to be reviewed online (there's a form for doing this
as well) and associating them correctly with their
authors.
The other user class is the Reviewer.
The Reviewer can see but not touch information such as
the list of other Reviewers, the list of papers, and
who's assigned to read what. The Reviewer can read
other people's reviews as well; there's no access control
mechanism segregating reviewers from one another, so
don't try to use this system to build your own ebay :-)
A malicious user could easily mess up the works, so
restrict use of this product to a group of people with
strong mutual trust and maturity.
The Reviewer can submit a review in one of two ways:
by filling out an online Review Form (HTML), or by
downloading a plain-text Review Form, editing it, then
mailing it back to a pseudo-recipient at the Admin's
site. To make the mail-based reviewing work you must
be able to get at your local mail hub's aliases, or
mess about with a fake user account and .forward files.
NOTE that the ingestion s'ware for the incoming
email message is very, very picky about format. You
have to dot your i's and mind your whitespace carefully.
Most people have no trouble with this, but some reviewers
persist in deleting the lines marked "don't delete this
text."
The Web-based Review form deposits the submitted info
directly into the gdb "database". The incoming mail
file, however, is processed by a mail receiver robot
which eventually does the same. BTW, the review text
is stored as a text file, but the meta-data about the
review go into the "database"; no large text blobs
in the database.
Screen Shots (sorry, PostScript)
Note: the looknfeel you see here is my doing. Mike's
version a couple of years ago looked quite different!
You can customize the logos, etc however you like.
I can't really show you much more, because there are
real review data in this copy which are moderately
confidential :-)
Requirements
To make this suite of software work you will need:
- Tcl/Tk
- go to Scriptics and
tcltk.com to find out about
Tcl/Tk and get the source. It's free.
- TclX
- Get this also from Scriptics or tcltk.com, see above.
- CGItcl
- This is a pureTcl extension for cgi scripting (and very nice
too) by Don Libes. You should be able to find it via tcltk.com,
or use your favourite search engine. It's probably been updated
quite a bit since I last got v 0.6, but knowing Don's meticulous
coding style I bet nothing has been broken.
- libgdbm.a, libgdbm.so, etc.
- If you are running Linux you already have these libs in /usr/lib.
If you are running some other OS, you'll probably have to go out
and find the source for libgdbm and build/install it yourself.
- The Tclgdbm extension
- I'm offering this here in case it is hard to find. I don't know
whether any other version of Tclgdbm work with this software suite,
and I am sure you would rather not find out the hard way; this version
was known to work.
- One more thing . . .
- You will need enough privs on your http server to set up
CGI scripts.
When you have all this stuff built and working, you should be ready
to play. The next step is to get the actual code, offered here
as 2 gzipped tarballs. Each will make its own subdir. Keep them
separate.
- The CGI Scripts and data
- This is stuff that goes under the cgi-bin dir on your http server.
- Working Web Directory, Scripts, Data
- This is stuff that lives in some normal web subtree out in homepage space.
In each tarball I have provided a README which I hope will make your initial
experience less bewildering than mine was :-) Feel free to send me questions,
I'll try to help if I have a spare moment, but don't expect a very quick
turnaround. Remember this is UNSUPPORTED. Having said
all that . . . enjoy! it's actually pretty nifty stuff and can make
the coordination of a refereed event easier for you.
NOTE
The UC Regents don't care if my code works for you or not, so there are no
guarantees. Read the COPYRIGHT statement on anything you download from us.
Everything you get from us is free. As software ought to be. :-)
de@ucolick.org
webmaster@ucolick.org
De Clarke
UCO/Lick Observatory
University of California
Santa Cruz, CA 95064
Tel: +1 408 459 2630
Fax: +1 408 454 9863