Review Guidlines

Your review must follow the following template:

Summary: Provide a short summary (~3 - 6 sentences) of the paper. This should be based only on what’s in the paper itself, without any opinion or analysis of your own.

Pros: Bullet points indicating good things about the paper (e.g., well-written, good results, thorough experimentation, clever idea, etc.)

Cons: Bullet points indicating bad things about the paper (e.g., difficult to understand, improperly evaluated, unoriginal, etc.)

Overall thoughts: Your overall thoughts on the paper. This is free form, and should justify your scores. This should be relatively detailed, and provide useful feedback to the authors.

Detailed comments: If there are any minor details (e.g., typos, issues which were too small to mention in overall thoughts), these should go here. Depending on how detailed your overall thoughts are, it may be appropriate to leave this section blank.

In addition, for each of the six paper types, there are certain specific criteria which should be kept in mind while writing your review. These criteria are listed below:

1. Traditional technical - is the solution novel? It is non-trivial? Is it useful? Is the solution clearly described? Is relevant existing work cited? Is the evaluation of the solution convincing?

2. Experimental - Is the algorithm or tool important? Is there something different about the reported experiment from earlier experiments on the same algorithms/tools? Are the evaluation metrics meaningful? Was care taken in the design of the benchmark data set? Do the experiments test for something of real interest, and are the results clearly reported through tables and plots?

3. Implementation - Is the algorithm important? Is there a significant issue with existing implementations that motivates another implementation? Is the idea behind the new implementation non-trivial? Are the experiments well-designed and are the experimental results reported clearly?

4. Literature surveys - is the survey comprehensive and up to date? Are sufficient details of cited papers provided? Is the survey easy to read and understand? Does the survey connect work in the field? There should be a sufficient number of references in the paper, which should be relevant to the paper, and the references should be of high quality.

5. Tutorials - Is it well-written? Is the pedagogical approach good? It is technically sound? Does it improve upon other tutorials on the same subject (if available)?

6. Instructional pearls - how useful is this technique/approach/design? How new or unfamiliar is it? Is it relatively clean and elegant? Is the paper easy to read and understand? Are examples informative and complete? Is the paper engaging?

Keep in mind that this will likely be the first experience the students have had with peer review. Whether the paper is accepted or rejected, students should know exactly why from the reviews. Students at the start of their research careers will need this information to improve their research and paper-writing skills.