| Ranking basics |
Paper ranking is an alternate method to extract the PC's preference order for
submitted papers. Each PC member ranks the submitted papers, and a voting
algorithm, the Schulze
method by default, combines these rankings into a global preference order.
HotCRP supports ranking through the tags system. The chair can define a tag to be used for ranking. PC members vote by assigning the corresponding twiddle tags. For instance, a paper tagged “~rank#1” is the user's first preference, a paper tagged “~rank#2” is the second preference, and so forth. To combine PC rankings into a global preference order, the PC chair goes to the search page, selects all papers, and chooses Tags > Calculate rank, entering “rank” for the tag. At that point, the global rank can be viewed by a search for “order:rank”. PC members may enter rankings by manipulating tags directly, but it will generally be easier to use the offline ranking form. Download a ranking file, rearrange the lines to create a rank, and upload the form again. For example, here is an initial ranking file: # Edit the rank order by rearranging this file's lines. # The first line has the highest rank. # Lines that start with "#" are ignored. Unranked papers appear at the end # in lines starting with "X", sorted by overall merit. Create a rank by # removing the "X"s and rearranging the lines. A line that starts with "=" # marks a paper with the same rank as the preceding paper. A line that starts # with ">>", ">>>", and so forth indicates a rank gap between the preceding # paper and the current paper. When you are done, upload the file at # http://your.site.here.com/offline Tag: ~rank X 1 Write-Back Caches Considered Harmful X 2 Deconstructing Suffix Trees X 4 Deploying Congestion Control Using Homogeneous Modalities X 5 The Effect of Collaborative Epistemologies on Theory X 6 The Influence of Probabilistic Methodologies on Networking X 8 Rooter: A Methodology for the Typical Unification of Access Points and Redundancy X 10 Decoupling Lambda Calculus from 802.11 Mesh Networks in Moore's Law X 11 Analyzing Scatter/Gather I/O Using Encrypted Epistemologies The user might edit the file as follows: 8 Rooter: A Methodology for the Typical Unification of Access Points and Redundancy 5 The Effect of Collaborative Epistemologies on Theory = 1 Write-Back Caches Considered Harmful 2 Deconstructing Suffix Trees >> 4 Deploying Congestion Control Using Homogeneous Modalities X 6 The Influence of Probabilistic Methodologies on Networking X 10 Decoupling Lambda Calculus from 802.11 Mesh Networks in Moore's Law X 11 Analyzing Scatter/Gather I/O Using Encrypted Epistemologies Uploading this file produces the following ranking:
Since #6, #10, and #11 still had X prefixes, they were not assigned a rank. The user can search for “order:~rank” to see their ranking, and inside the system; an administrator can search for “order:pcname~rank” to see a particular user's ranking. Once a global ranking is assigned, “order:rank” will show it. |