Hackathons’s Docs ▸ Introduction to Checkers

Introduction to checkers

Checkers are bash scripts written to evaluate the correctness of the solution submitted to a problem. They are written to be more generic so that many problems can use the same checker. Before writing a new checker please check whether there is an existing checker which behaves in the way you need. For example following are few existing checkers and their behaviors.

Name

Description
sim_out Execute iverilog simulator for testbench.v TEST files. Compare the console output with SOLUTION file and report pass/fail status based on line match.

In order to use this checker, problem should include a SOLUTION file.

This checker doesn’t output the simulation output. Just the pass/fail status.
open_sim_out This checker is very much similar to sim_out. However it outputs the simulation output. Competition participants can use $display in there submission to display values.

IMPORTANT: Ability to use $display can help participants to monitor input values. This can be exploited to cheat in competitions.
flop_count Synthesize the design using yosys and compare the flop count with limit set in SOLUTION

 

How to write a new checker

Writing a new checker is straight forward. The automatic evaluator execute the checker script with two command line arguments:

  1. Absolute path to problem folder
  2. Absolute path to TEST file which contains user submitted answer

Any checker should use only these two inputs to evaluate the answer. checkers can create intermediate files. However it should clean all these intermediate files before exiting.

If the answer is correct checker should exit with 0 status. if not checker should exit with non zero value (i.e 1)

 

Example checker

Below is a very basic checker. Usually checker should include more safety checks to increase reuseablity.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#/usr/bin/bash
######### System Check ##############
if [ "`which iverilog`" == "" ]; then
echo "[ERROR ]: iverilog not found can't simulate"
exit 1;
fi
######### Input Check ###############
if [ -z $1 ]; then
echo "[ERROR ]: Problem path not found"
exit 1;
fi
if [ -z $2 ]; then
echo "[ERROR ]: Submission not found"
exit 1
fi

######### Run simulation #############
iverilog $2 $1/testbench.v
sim_out = "`vvp a.out`"

# a.out is generated by checker, hence it should be removed before exiting
rm a.out

if [ "$sim_out" == "`cat $1/SOLUTION`" ]; then
exit 0
else
exit 1
fi