| Submission time |
Follow these steps to prepare to accept paper submissions.
Set up PC
member accounts and decide whether to collect authors'
snail-mail addresses and phone numbers.
Set submission
policies, including whether submission is blind, whether
authors check off conflicted PC members (“Collect authors' PC conflicts
with checkboxes”), and whether authors must enter additional non-PC collaborators,
which can help detect conflicts with external reviewers (“Collect authors'
other collaborators as text”).
Set submission
deadlines. Authors first register, then submit
their papers, possibly multiple times; they choose for each submitted
version whether that version is ready for review. Normally, HotCRP allows
authors to update their papers until the deadline, but you can also require
that authors “freeze” each submission explicitly; only
administrators can update frozen submissions.
The only deadline that really matters is the paper submission
deadline, but HotCRP also supports a separate paper registration deadline,
which will force authors to register a few days before they submit. An
optional grace period applies to both deadlines:
HotCRP reports the deadlines, but allows submissions and updates post-deadline
for the specified grace period. This provides some
protection against last-minute server overload and gives authors
some slack.
Define
submission options (optional). You can add
additional checkboxes to the submission form, such as "Consider this
paper for the Best Student Paper award" or "Provide this paper to the
European shadow PC." You can
search for papers with or without
each option.
Define paper
topics (optional). Authors can select topics, such as
"Applications" or "Network databases," that characterize their
paper's subject areas. PC members express topics for which they have high,
medium, and low interest, improving automatic paper assignment. Although
explicit preferences (see below) are better than topic-based assignments,
busy PC members might not specify their preferences; topic matching lets you
do a reasonable job at assigning papers anyway.
Set
up the automated format checker (optional). This adds a
“Check format” button to the Edit Paper screen.
Clicking the button checks the paper for formatting errors, such as going
over the page limit. Papers with formatting errors may still be submitted,
since the checker itself can make mistakes, but the automated checker leaves
cheating authors no excuse.
Take a look at a paper
submission page to make sure it looks right.
Open the site
for submissions. Submissions will be accepted only until the
listed deadline.
|
| Assignments |
After the submission deadline has passed:
Consider looking through all papers for
anomalies. Withdraw and/or delete duplicates or update details on the paper pages (via “Edit paper”).
Also consider contacting the authors of papers that
were never officially submitted, especially if a PDF document was
uploaded (you can tell from the icon in the search list). Sometimes a
user will uncheck “The paper is ready for review” by mistake.
Prepare the
review form. Take a look at the templates to get
ideas.
Set review
policies and deadlines, including reviewing deadlines, whether
review is blind, and whether PC members may review any paper
(usually “yes” is the right answer).
Collect review
preferences from the PC. PC members can rank-order papers they
want or don't want to review. They can either set their preferences all at once, or (often more
convenient) page through the list of submitted papers
setting their preferences on the paper pages.
If you'd like, you can collect review preferences before the submission
deadline. Select “PC can
see all registered papers until submission deadline”, which
allows PC members to see abstracts for registered papers that haven't yet
been submitted.
Assign
conflicts. You can assign conflicts by PC member or, if
PC members have entered preferences, automatically by searching for
preferences of −100 or less.
Assign
reviews. You can make assignments by paper, by PC member, by uploading an assignments
file, or, even easier, automatically. PC
review assignments can be “primary” or “secondary”; the difference is
that primary reviewers are expected to complete their review, but a
secondary reviewer can choose to delegate their review to someone else.
The default assignments pages apply to all submitted papers. You can
also assign subsets of papers obtained through search, such as papers
with fewer than three completed reviews.
Open the site
for reviewing.
|
| Chair conflicts |
Chairs and system administrators can access any information stored in the
conference system, including reviewer identities for conflicted papers. For
this reason, some chairs prefer not to use the normal review assignment
process for their own submissions, and HotCRP supports an alternate review
mechanism. For each chair conflict:
- A chair or system administrator goes to the paper's assignment page and clicks
on “Request review” without entering a name or email address.
This creates a new, completely anonymous review slot and reports a
corresponding review token, a short string of letters and numbers
such as “9HDZYUB”. The chair creates as many slots as
desired.
- The chair sends the resulting review tokens to a PC member designated as
the paper's manager. This trusted party decides which users should
review the paper, and sends each reviewer one of the review tokens.
- When a reviewer signs in and enters their review token on the home page,
the system lets them view the paper and anonymously modify the corresponding
review.
Reviews entered using this procedure appear to be authored by “Jane
Q. Public.” Chairs can still see (and edit) the reviews if they
override their conflicts, but reviewer identities are not stored in the
database at all.
Alternately, the trusted manager can send the reviewers the paper and an
offline review form via email (not using HotCRP). The reviewers complete
the offline forms and send them to the manager, who uploads them into the
“Jane Q. Public” review slots using the review tokens. This
way, even web server access logs store only the manager's identity.
|
| Before the meeting |
Before the meeting, you will generally set “PC can see all
reviews”, allowing the program committee to view reviews and scores for
non-conflicted papers. (In many conferences, PC members are initially
prevented from seeing a paper's reviews until they have completed their own
review for that paper; this supposedly reduces bias.)
Collect
authors' responses to the reviews (optional). Some conferences
allow authors to respond to the reviews before decisions are made, giving
them a chance to correct misconceptions and such. Responses are entered
into the system as comments. On the decision settings page,
update “Can authors see reviews” and “Collect responses to the
reviews,” then send mail to
authors informing them of the response deadlines. PC members will still
be able to update their reviews, assuming it's before the review deadline; authors
are informed via email of any review changes. At the end of the response
period it's generally good to turn off “Authors can see
reviews” so PC members can update their reviews in peace.
Set PC can
see all reviews if you haven't already.
Examine
paper scores, either one at a time or en masse, and decide
which papers will be discussed. The tags system lets you prepare
discussion sets. Use search
keywords to, for example, find all papers with at least two overall
merit ratings of 2 or better.
Assign discussion order using tags (optional). Common
discussion orders include sorted by overall ranking (high-to-low,
low-to-high, or alternating) and sorted by topic. Explicit tag-based orders
make it easier for the PC to follow along.
Assign discussion leads
(optional). Discussion leads are expected to be able to
summarize the paper and the reviews. You can assign leads either paper by paper or automatically.
Define decision
types (optional). By default, HotCRP has two decision types,
“accept” and “reject,” but you can add other types of acceptance and
rejection, such as “accept as short paper.”
The night before the meeting, download all
reviews onto a laptop (Download > All reviews) in case the
Internet explodes and you can't reach HotCRP from the meeting
place.
|
| At the meeting |
It's often useful to have a PC member or scribe capture the discussion
about a paper and enter it as a comment for the authors'
reference.
Paper decisions can be recorded on the paper pages or en masse via search. Use decision settings to expose
decisions to PC members if desired.
Shepherding (optional). If your conference uses
shepherding for accepted papers, you can assign shepherds either paper by paper on the
assignments screen or automatically.
|
| After the meeting |
Enter
decisions and shepherds
if you didn't do this at the meeting.
Give reviewers some time to update their reviews in
response to PC discussion (optional).
Set “Who can
see decisions?” to “Authors, PC members,
and reviewers.”
Send mail to
authors informing them that reviews and decisions are
available. The mail can also contain the reviews and comments
themselves.
Collect final
papers (optional). If you're putting together the program
yourself, it can be convenient to collect final copies using HotCRP.
Authors upload final copies the same way they did the submission, although
the submitted version is archived for reference. You can then download
all final copies as a .zip archive.
|