Rationale Behind our Automated Coding Challenge

You don't have time to interview every person that applies for a software developer position at your company. However, you want to give every candidate a chance to show you their skills.

There are two common approaches companies use as the first step in their hiring process: resume screening and homework assignments. Both of these approaches have some issues.

Resume screening allows you to look for relevant experience, but it doesn't give you any indication of the candidate's true skill level. Aline Lerner goes into depth on this topic in her article Resumes suck. Here’s the data.

Homework assignments do give you a measure of skill. However, it takes engineering time to create the test, figure out how to host or distribute it, and then grade it. The other problem is that many of these tests are tedious and take way too long to finish. This drives many good candidates away.

We wanted to build something that would do a good job of testing development skills, while still being quick and engaging.

Think about how developers spend their time in the real world. Most of it is spent reading and maintaining existing code. This seems like a good place to start when screening developers.

Rather than asking candidates to write a bunch of boilerplate code from a blank slate, we do the following:

  1. Give the candidate a piece of existing code and a description of what it's supposed to do
  2. Inform them that there is a bug in the code
  3. Ask them to find the bug

The ability to quickly read and understand existing code is something that every developer needs to excel at. You cannot add new features to an existing codebase, or fix bugs without this skill.

To prove that they've found the bug, we ask the candidate to provide a failing test case. Our challenge framework has unit testing built in. Testing is another skill that developers use in the real world.

Once they've found the bug and added a failing test case, we have them fix the code so that the tests passe. We also run a battery of hidden test cases to ensure the code fully meets the requirements.

Because the code was already provided, and our platform streamlines everything, most problems are solvable within a minute or two. When the candidate finishes solving their first question, we give them another. The candidate has 20 minutes to solve as many problems as possible. They are given a score based on how far they make it into the challenge.

Because this is designed to be a first level filter, all of the questions are relatively easy. A good developer will score fairly well on the challenge. Next, you can follow up on with a more traditional interview.

In order to make it as easy as possible to get started, we have created and configured all the questions for you. All you need to get started is the email address of the candidate you'de like to challenge.