There are plenty of guides you can find online, written by well–respected investors, about how to do a reference check. They all boil down to some version of:
- Find 3+ people who know the candidate
- Ask them a series of questions for which the “right” answer is obvious, e.g. “How would you rank this person amongst all the people you’ve worked with? Top 10%? Top 5%?”
You can see the problem. (There are actually several problems, but we’ll take the most glaring one first.)
Most reference checks are useless — or worse
First: If you ask a question where the “right” answer is clear, you make it too easy to game the reference process. You get responses that, at best, don’t add any resolution to your hiring process, and at worst actively mislead you about what a candidate is really like.
The most common version of this question is: “Is this one of the best people you’ve ever worked with?” Variants include:
- How would you rank [candidate] relative to other people you have worked with in terms of raw productivity?
- Does [candidate] work well with others?
- Knowing what you know now, would you hire [candidate] again?
And so on.
The common theme is anyone acting as a reference knows anything less than saying “oh yeah this person is certainly top quartile, if not top decile” will result in tanking the hiring process. This is especially true for warm references — ones provided by the candidate. (Backchannel references are much less likely to suffer this issue, which is why they’re better.)
I don’t mean to suggest people will lie outright. However, people are naturally agreeable, so on aggregate they will shade their answers more positively.
If you’ve done references for a while you’ll get good at answering these questions in ways that won’t set off the interviewer’s flags, even if you don’t think a candidate is great. This results in an arms race of “reading between the lines” by the interviewer vs. “giving PR-quality answers” by the reference. It’s certainly a way to conduct reference checks, but you’re stuck reading tea leaves (are you interpreting it correctly?) and you’re still susceptible to the second issue with this line of questioning
The second problem is: you don’t know if the person providing the reference is any good. A reference is only as reliable as the person providing it. One of the most famous Steve Jobs sayings is “A players hire A players. B players hire C players.” Unless you know whether or not the person providing the reference is good, it could just be one bozo recommending another bozo. And if one thing is universally true, it’s that bozos love other bozos.
The third problem is: someone being amazing in one context (or company) doesn’t mean they’ll be good in another. This is the most overlooked piece of Keith Rabois’s brilliant metaphor of employees at a company acting as “barrels vs. ammunition”:
Most people — most great people even — are ammunition. But what you need in your company are barrels. You can only shoot through the number of unique barrels you have. So that’s how the velocity of your company improves is by adding barrels, and then you stock them with ammunition, and then you can do a lot. So if you go from a one-barrel company, which is mostly how you start, to a two-barrel company and suddenly you get twice as many things done per day, per week, per quarter. If you go to three barrels, great, if you go to four barrels — awesome.
Barrels are incredibly difficult to find. But when you have them — give them lots of equity, promote them, take them to dinner every week, because they’re virtually irreplaceable. Because they’re also very culturally-specific. So a barrel at one company may not be a barrel at another company. Because one of the ways — and the definition of a barrel — is they can take an idea from conception all the way through shipping and bring people with them. And that’s a very culturally-specific skillset.
A barrel at one company might be ammunition at another, and vice versa.
So: what would a better, more helpful reference check look like?
How to do reference checks that deliver actual insights
Ask questions where the right answer isn’t clear. Only then will you get honest answers and reference checks that truly add value.
Here are some examples:
- If you were coaching this person on how to reach the next step in their career, what would you say they need to work on?
Everyone has things they need to work on. You, as the person hiring, should know what areas of development are acceptable vs. unacceptable. These will vary by company, even for the same role, depending on the competencies of team doing the hiring.
- Is this person a consensus-builder or a maverick? Give examples. What are the pros and cons of their approach?
Some companies value leaders who bring others along. Others value iconoclasts who ignore processes and frustrate colleagues. Because the person giving the reference doesn’t know what you value, you’re more likely to get an accurate assessment.
- How have you seen this person take initiative in areas outside of their direct responsibility? What was the impact on the company?
Any worthwhile candidate will push companies forward in areas outside of their job scope. This question tells you where their spikes are—their idiosyncratic superpowers or areas of interest — and shows you what kinds of problems they’ll be drawn to solving at your company. You then know whether they’re likely to create distractions or add far more value than their role description would indicate.
These sample questions are just a start. Find the 3–5 most important areas where you’d like additional data and formulate your own questions that don’t have a clear right or wrong answer.
The other upshot to questions that adhere to this rule is it doesn’t matter whether you’re talking to a warm reference or a backchannel one. Since there’s no right answer, there’s no way for the person providing the reference to shade their response in a way that’s predictably more favorable to the candidate. And, as a bonus, by comparing multiple references’ answers you can create a composite picture and sniff out exaggerations — or outright fabrications.
Reference checks are only a waste of time because the conventional wisdom on how to do them is so egregiously wrong. Done right, they can add tremendous predictive power to your hiring process. Just remember to ask questions where there’s no clear right answer and your references will transform from frustrating to insightful overnight.