Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to create an auto-grader for one of my beginning programming courses for python. From my online search, I've come to know that it is effectively a unit test framework that tests the student's code rather than production code but I'm not really sure how to structure the flow of the program. Can anyone please provide a strategy for submission of code by students and automating the whole process of marking?

For instance, how would the student code be submitted and then stored/structured on disk, how would the grades be stored/reported?

I'm only looking for a broad strategy and will try on my own to fill in the blanks.

(I asked this on initially but it's considered as off-topic and I was suggested to ask here.)

share|improve this question

closed as too broad by Ixrec, Snowman, jwenting, GlenH7, TZHX May 13 '15 at 22:00

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

Warning: Computers don't notice cheating unless you tell them to. The last time I graded in an automated fashion, I caught more than 10% of the class cheating with byte-for-byte identical answers. – Brian Jan 2 '13 at 14:25
Yes, that IS an issue but that will be worked on later :) Thanks for pointing it out though :) – recluze Jan 3 '13 at 5:10
As an aside, I've used doc-test + other unit tests before for this sort of thing, you give students the benefit seeing some sample inputs + outputs. Of course if you only do doctests cheating is trivial. – jozefg Jun 25 '13 at 16:29
up vote 3 down vote accepted

I'm great at broad strategies with no detail!

Run an Apache server. Put up a web-interface(1). User log-ins and all that jazz. The students navigate to the assignment they're working on, paste their code in a field(2), and submit it. Everything(3) gets stored in a SQL database, and kicks off the associated unit-testing script(4). Good unit-tests have descriptive reports about why units fail and test every unit independently, ie what you tell the student about what didn't work. These results are likewise stored in the database and reported to the student.

But maybe a little detail:

(1) You could have them directly ftp their data to a permissions restricted folder, or let them access the database directly, but you want to give them feedback about how they did. Plus kids these days are used to web-interfaces. And it's better to secure a website and validate input fields rather than exposing a database. Take that as a big hint that you will have students that will try to compromise your system.

(2) or select a file or whatever. Work with me here. You could also give them a space to ask questions, give excuses, rant, and/or beg forgiveness.

(3) Everything everything. What was submitted, by who, for what, from where, with a time-stamp. The lesson itself too: the problem, the example code, the unit testing, the scripts it uses, everything. You want to be able to add lessons to this and you want to be able to know what the heck happened when a student breaks it.

(4) The code the students submit, either a block of code, a function, or a whole program is loaded into the unit-testing framework which takes the code and runs a set of unit-tests against it. These are tests that you write. Each test sees if the code breaks in a specific way.

Other things to consider when making an automated system for programming homework:

  • Brand new programmers need hand-holding. They won't know they need to #include to use printf(). Give them the lines above and below the lesson section. Don't even let them edit those lines. Give them one blank box to fill in.
  • The lesson plan should include debugging other people's code. Including simple syntax errors and fundamental design issues.
  • Reading gcc error messages is a skill
  • Eventually students need the experience of being given a blank slate and making a whole program, with multiple files, that compiles on different architectures, in Malbolge.
  • Do not presume how the students will solve the problem. I've had classes where half the class initially failed an assignment because the automated test assumed "0" was the "front" of the array.
  • Ideally, problems should be adjusted on a student by student basis, to keep people from just swapping code. If it were a physics class, it'd be as simple as generating a set of slightly different variables for each student so everyone has to come up with their own solution. For a programming class, it will probably only be applicable for some lessons.
  • The linux guy will submit code that doesn't contain the standard windows end-line combo. You can't fail him because of that.
  • You're taking code written by others and executing it. SANDBOX IT. Likewise, sanitize the user input to the webserver. Always beware little Bobby Tables.
  • Writing the unit tests for the lesson plan which covers unit-testing is going to be very meta. Take some asprin. Good luck.
share|improve this answer
Very nice. That's the kind of stuff I'm looking for. The only issue I have is "kicks off the associated unit-testing script". How do I run tests on n number of scripts? I can't send the whole code to the test class, right? – recluze Jan 3 '13 at 5:09
@recluze Number of scripts? It depends on how you set up unit testing, but usually the framework allows a number of tests to be ran with the results being built up in a log file or something. Yeah, you send the whole code to the test class. Sometimes there's shenanigans with overwriting main(), but I'm a C guy. The script I was talking about was the one which takes the student's source code, saves it to a file, compiles it all with your unit test code, runs the damn thing, and then does something with the result. That you want to script. The unit-testing itself would be in python. – Philip Jan 3 '13 at 15:37
Yep, that makes a lot of sense. I think I can go ahead and start with this strategy. Thanks again. – recluze Jan 4 '13 at 13:30

Require the students to write functions or classes, then write code that includes their project and will test their classes and functions to see if they throw an exception for boundary conditions, and provide the correct result for a given input. A number would be provided for how many of each kind of tests passed or failed, and written to a file or database, prompting you to input the students name.

From there you can set up any way of interpreting that base data to determine a fair grade, also taking into account their syntax and style by reading the program.

share|improve this answer
If you take this approach, I recommend providing skeleton code so that no student has a right answer with a broken interface. Further, I recommend providing a sample testing program (hard-coded questions and answers). Don't use the same testing program yourself; students might hardcode their program. – Brian Dec 31 '12 at 14:17
Thanks but "prompting you to input the students name" .. that wouldn't scale well, no? That's part of the issue: how to differentiate files submitted by different students. – recluze Dec 31 '12 at 14:51
You can require them to put their name in the first part of the file as a comment, but if I know students that's going to go to hell fast. Maybe have it only prompt you if there's a missing name or the program can't detect a name. – BeardedO Dec 31 '12 at 15:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.