•     Cool or significant ideas. What is new here? What are the main contributions of the paper? What did you find most interesting? Is this whole paper just a one-off clever trick or are there fundamental ideas here which could be reused in other contexts?
•     Fallacies and blind spots. Did the authors make any assumptions or disregard any issues that make their approach less appealing? Are there any theoretical problems, practical difficulties, implementation complexities, overlooked influences of evolving technology, and so on? Do you expect the technique to be more or less useful in the future? What kind of code or situation would defeat this approach, and are those programs or scenarios important in practice? Note: we are not interested in flaws in presentation, such as trivial examples, confusing notation, or spelling errors. However, if you have a great idea on how some concept could be presented or formalized better, mention it.
•     New ideas and connections to other work. How could the paper be extended? How could some of the flaws of the paper be corrected or avoided? Also, how does this paper relate to others we have read, or even any other research you are familiar with? Are there similarities between this approach and other work, or differences that highlight important facets of both?

Exam

There will be three exams. All exams are in-person, closed book. The instructor is unable to accommodate different dates for mini exams. No exceptions will be made. However, you may drop one exam out of three exams and your best two exams will be counted towards the final grade.

Mini Project

• proposal of a new research project
• develop and assessing new algorithms to replace currently-used ones
  
• translate a methodology to a new problem domain
  
• apply known techniques to new problem domains
• port an existing tool to a new domain (e.g., a new programming language)
  

Each team should submit a written report (max 10 pages) to the BruinLearn system. Each team should consist of 5 to 6 people. You may include an appendix beyond 10 pages, but your paper must be understandable without it. Submissions should be in the ACM format.  In Week 2, I will release a few sample project ideas to guide you with the process of choosing a project topic. A short proposal is due on Week 3 and a final project report is due on the final exam week. A part of instruction time will be reserved for project team meetings with the instructor and there will be a sign up via Piazza. While it is a team project, each team member is not automatically entitled to receive the same grade and their score will be adjusted based on each one's contribution, if necessary. Multiple sources of evidence including a peer evaluation will be collected.

Your report should be structured like a conference paper, meaning that your report should contain:

• Abstract
• A well-motivated introduction
• Related work with proper citations
• Description of your methodology
• Evaluation results 
• Discussion of your approach, threats to validity, and additional experiments
• Conclusions and future work 
• Appendix: Describe how to run and test your implementation.

Motivation & Problem Definition

• Does this report sufficiently describe motivation of this project?
• Does this report describe when and how this research can be used by whom in terms of examples and scenarios?
• Does the report clearly define a research problem?

Related Work

• Does the report adequately describe related work?
• Does the report cite and use appropriate references?

Approach

• Does the report clearly & adequately present your research approach (algorithm description, pseudo code, etc.)?
• Does the report include justifications for your approach?

Evaluation

• Does this report clarify your evaluation’s objectives (research questions raised by you)?
• Does this report justify why it is worthwhile to answer such research questions?
• Does this report concretely describe what can be measured and compared to existing approaches (if exist) to answer such research questions?
• Is the evaluation study design (experiments, case studies, and user studies) sound?

Results

• Does the report include empirical results that support the author’s claims/ research goals?
• Does the report provide any interpretation on results?
• Is the information in the report sound, factual, and accurate?

Discussions & Future Work

• Does the report suggest future research directions or make suggestions to improve or augment the current research?
• Did the report demonstrate consideration of alternative approaches? Does the report discuss threats to validity of this evaluation?

Clarity and Writing

• Is the treatment of the subject a reasonable scope as a class project?
• How well are the ideas presented? (very difficult to understand =1, very easy to understand =5)
• Overall quality of writing and readability (very poor =1, excellent =5)