How to Evaluate Technical Skills in Interviews

Designing a technical interview process isn’t just about asking hard questions. It’s about finding the right questions to ask for the role.

I’ve hired hundreds of engineers over the course of my career and interviewed thousands. I found that some of the methods used to evaluate engineers for technical skills were just worthless for what I needed, and others were absolutely spectacular. There’s no one method that works for every role or every company, though. Let’s look at the options and discuss what works in different scenarios and when you should and shouldn’t use these techniques.

Brain Teasers

First, let’s start with generally ineffective questions that you should avoid: brain teasers. Questions like “How many windows are in New York?” or “How many golf balls will fit in a Boeing 747?” don’t predict anything useful about a candidate. This isn’t just my opinion, either. There are multiple sources that all indicate that these won’t teach you anything about a candidate. Here are a couple of examples: The Validity and Utility of Selection Methods in Personnel Psychology and Work Rules!: Insights from Inside Google That Will Transform How You Live and Lead.

Use this technique if you want to impress people at cocktail parties.

Logic Puzzles

The same issues apply to logic puzzles—they’re just not useful in interviews. Unlike brain teasers, logic puzzles ostensibly have a correct answer or answers, but they’re still not predictive of success in a role. Read A review of structure in the selection interview for more information.

Use this if you need a fun activity during game night with friends.

Leetcode/DSA Questions

Next up are the Data Structures and Algorithms questions. These are of limited usefulness in evaluating candidates for roles. While you could learn a bit about whether the candidate has retained enough knowledge about these things to pass a test, it won’t tell you if they can do the job you’re hiring them to do. These tests also have high false positive rates, as you’re evaluating someone on one skill (the ability to take a DSA test) but asking them to do a different skill (connect an API to a front end, e.g.). Plus, some people get really mad about this.

Use this technique if you’re designing your own language or need to deeply optimize a system.

Whiteboard

Whiteboard interviews can be used effectively in some circumstances. If you’re using them to conduct system design interviews, they can help both the interviewer and the candidate visualize topology and communication paths.

However, many interviewers still pose a Leetcode or DSA-style problem to candidates and ask them to solve it using code or pseudocode. This suffers from the same issues as above but worsens it by adding stress and awkwardness to the process. This results in a very high false-negative rate and, again, poor predictions. Read “Does stress impact technical interview performance?” to get Microsoft’s take on this.

Use these for system design interviews, which are generally for more senior candidates. They’re a great option for that. Avoid them for DSA questions or junior candidates.

Code Reviews

Code review questions can be very effective in evaluating candidates. Because reviewing code is something that the candidate will do regularly, and it requires interacting with others while doing it, it very closely represents a real-world scenario, and that’s the gold standard of evaluation techniques. The downside to these tests is that they’re difficult to construct and require frequent updating to ensure that answers don’t “leak.” Outsourcing these tasks to a consultant can help mitigate that downside, though.

This is a technique that I highly recommend for most roles. It’s fairly simple, if time-consuming, to implement, has clear-ish answers, and gives you a clear signal about the candidate’s abilities. If you need to assess a candidate’s ability to code something start-to-finish, though, look at portfolio reviews or take homes for that.

Use this for most roles, and combine it with other tests if you need more confidence in your decision.

Portfolio Review

Having a large codebase to review can give you great insights into not only how well someone solves problems but also how they communicate that solution to others. This can be a great option for those candidates who have public portfolios to share. However, many candidates don’t have the free time to build their own projects because they have other responsibilities, which limits your applicant pool and unfairly screens out some candidates. This can bias your hiring toward certain demographics and potentially result in discrimination claims.

Use this with caution to fill high-impact roles that require highly independent coders with a proven track record of building projects solo.

Take Home Tests

Take-home tests can be quite useful in evaluating a candidate’s abilities. If the problems posed are representative of real-world work, they can provide a great signal that the candidate will be able to do the job they’re being hired for. There are even platforms that automate much of the process to help you out. These tests and platforms typically mimic the conditions that the candidate will be coding in, which helps to reduce stress and cognitive load. If you’re using one of these platforms, custom questions will give you the best signal, but they can be time-consuming to create. Using the off-the-shelf questions will still give you a decent signal, though.

Use this when you need to evaluate junior to mid-level engineers and focus on aspects of coding other than correctness and time and space efficiency, i.e., most of the time.

Take Home Projects

This is one of the more polarizing issues in technical testing. Some candidates will flat-out refuse to do a take-home project, while others are happy to accept the challenge. While this provides a great signal about how well candidates will perform, especially if they need to be independent and work on greenfield projects, there are definitely drawbacks. These tests are difficult to design and grade, requiring significant time and training for the reviewers.

Use this for high-impact engineering roles you don’t need to fill immediately. The additional time in the review cycle, combined with the number of candidates who will drop out, will result in a longer time to hire. Also, consider offering compensation to the candidates who complete the project. It improves your employer brand and the candidate experience and makes it more likely you’ll get people who’ll participate.

Behavioral Questions

Probably the simplest of all the methods is just asking candidates to tell you about a time they were able to solve a coding problem. This is a great option in many cases, especially with senior engineers, as they’ve likely had multiple opportunities to solve interesting coding problems. However, it can be difficult to truly gauge how someone actually codes by just asking them questions. For example, you might have someone who talks about maintainability but doesn’t practice it regularly.

Use this when you’re evaluating senior engineers who have long track records of success, and consider combining this with another method, like code reviews, to get a complete picture of their capabilities.

Summing Up

Tests are definitely not one-size-fits-all. There are pros and cons to each—too many to list in a short blog post. Pick your tests carefully and think about what you’re trying to accomplish and who you’re trying to attract to the role. And if you’re having trouble parsing all of this and just want to outsource it to someone else, get in touch. I’m always happy to chat.