As someone who has been an interviewer panelist for over 100 completed hires, with over 1,000 people interviewed, I have discovered something strange: very few people do reference calls with the intent of actually determining if someone is a good fit.
How do I know this? Because for the majority of reference calls that I see conducted across companies, the person passes. Not just some reasonable threshold, like 70-80%. More like 98%+.
One recruiter at a major tech company told me she has rejected “literally zero” people from a reference call over her career. What a colossal waste of time.
Here is the typical recruiting funnel that I see in tech, showing the percentage of candidates who move on to the next stage:
Step 1 – Resume review: 15%
Step 2 – Screening call: 50%
Step 3 - Hiring manager call: 50%
Step 4 - Bar Raiser call: 50%
Step 5 - Case study: 50%
Step 6 - Reference call: 98%+
So why is this bad? Isn’t it a wonderful thing that folks who get past the case study are such great, polished candidates?
No. For two reasons:
There are candidates left to screen – At least 25% of these hires are not going to out work out (some hiring managers estimate it closer to 50%). Your job is not done at the reference call stage – you just need the will and right knowledge to do it
Waste of time – It is not worth slowing a process down by 2-7 days for 2-3 reference calls with managers and peers, and potentially losing some of your best candidates due to competing offers, to have a performative final step. If you are going to do it, make it matter
Why do hiring managers and recruiters rubber stamp candidates at the reference check stage?
Happy eyes – The hiring manager has already passed the person forward from their interview. It can feel like at this point, you are rooting for them to get hired. You’ve accidentally switched your frame of mind from vetting to cheering. Once you do this, you’ll rationalize all sorts of negative signals from references, and interpret them in a rosy light
Sunk cost fallacy – Lots of time has been spent on this role once you are at the reference call stage. You’d hate to feel like you are throwing it all away and starting from square one if you say no. That said, the historical time spent is no longer relevant to your future decisions (it is a “sunk cost”, as economists say), and therefore you should not fixate on it.
You make what you measure – Charlie Munger famously said “show me the incentives and I will show you the outcome”. Most recruiters at least, and sometimes for hiring managers, have a hiring quantity goal that they want to hit each month and quarter. Adding one more rejection does not get them closer to achieving their hiring goals.
How can we combat a team’s habit of conducting ceremonial reference calls?
In sum, create a culture that celebrates keeping standards high, rewarding and recognizing folks when they protect the culture by saying “no”. You should not just positively recognize team members when they say “yes”, unless you want a culture that does not know how to say “no”. Ways to tactically do this:
Tone at the top – Your company’s execs, recruiters, and hiring managers need to positively recognize those who say “no” to candidates at all parts of the stage, not just those who say “yes”.
Metrics on quality of hires, not just quantity – Your recruiting team should have metrics not just on number of hires, but also on percentage that work out.
Hiring managers engage on reference calls – Ultimately, hiring managers will live with the results of this process. The reference call stage is the only one where you can peek under the curtain of a candidacy from a third party. For at least high stakes roles, you should have a hiring manager engage in this step.
Now, let’s get to how to do a proper reference call. First, it is worth noting that there are 4 main archetypes of people you are talking to.
They are as follows:
The Early Career Peer – Typically unvarnished in commentary, especially around social dynamics, but not in a great position to assess the work quality
The Experienced Peer – Often more nuanced in communication – including less direct about shortcomings – but can be better positioned to assess work quality if they open up
The Candid Manager – Someone who will tell you, without you needing to be clever or reassuring, how the person performs relative to others. This person is the hero of a hiring process
The Smokescreen Manager – Someone who will do backflips in an effort to aid the candidate in getting hired, including a sudden loss of memory around the candidate’s areas for growth. The sad irony about this person’s likely good intentions is that both the candidate and the hiring company would be better off if this reference was forthright, as it is much worse for the candidate to have a short stint and a black eye on their resume than get a simple pass in an interview process. Additionally, if the candidate is hired, accurate data from the prior manager’s reference call can help the hiring manager make their onboarding smoother. You’ll need to be thoughtful with this archetype to avoid a meaningless interview
Before I share the question list, here are some more thematic tips on conducting a successful reference call:
Know which archetype you are talking to and adjust your questions. I find it is still worth exploring someone’s performance when talking to a peer, but questions move from “someone you managed” to “other peers”, etc.
Listen 85%+ of the time. You should not be stating facts and asking them to confirm them. You should ask questions and give them space to editorialize. You will not learn much from hearing yourself talk
Have a healthy back-and-forth. Do not just have them answer a script of questions and let their insights come and go without a counter question. The most useful nuggets will come from you asking follow up questions that are specific to the discussion
When you ask for something constructive, lead with something constructive first about your company or yourself. Make it normal to have areas for growth. Few folks will just give you the unvarnished truth without you making the environment safe for them first
To extent possible, build rapport. They will feel bad lying to you or obfuscating if they start to know and like you over the call. Also, who knows, life is long and you end up working together!
In general, don’t share about the role and what makes someone successful in it unless you feel you must (The Smokescreen Manager archetype may ask for it). If you end up sharing this, expect them to basically recite those responsibilities and traits back to you as a description of this person (see: waste of time).
OK, so without further ado, here is a typical question list for a reference call at Eden, in order that they are asked, as some build on the former ones. For illustrative purposes, we’ll call the candidate Tina:
In what context did you know Tina?
When did you work with Tina?
How would you describe the culture at your company? You can be honest, no culture is perfect. For us, for instance, we tend to move too fast. It can lead to some small victories but also creates dysfunction on bigger projects, which can have the unintended side effect of slowing us down.
How would your colleagues describe Tina?
How many people have you worked with over your career in a similar role to Tina? [let’s pretend person says 10]
How would Tina honestly rank across those 10 people in a similar role in terms of effectiveness?
What separates Tina from the person that you ranked 1st a moment ago?
Now, across those 10 people, where does Tina stand out? Where would she reasonably be in the top 3-5?
Now everyone has room for growth. For me, for instance, I am working on my punctuality. I can sometimes be late and it is really pretty disrespectful to my peers, so I am working on improving it. I also know that I can be a little short with people sometimes when situations get a little tense, and it is something that I am working on. Compared to those 10 people from before, on what attribute is Tina realistically in the bottom 2-5?
Now let’s talk about a scale. From 1, which is “I never want to be in the same room as Tina again, to 10, which is “I am actively trying to hire Tina right now”, where would she land?
[for the inevitable < 10 answer, which can take a round of clarification if someone says “10” and in a follow up question they admit they are not trying to hire the candidate right now]: Why did you choose that number? What would make it so you were trying to hire her right now?
I find with the above questions, and some casual follow ups, you can have a 20-30 minute call that makes the round go from 98%+ acceptance to closer to 70%, as you actually learn things that are useful to the interviewing process.
Beyond just better screening, I find the areas for growth raised in the reference calls almost always show up later with the candidates you choose to hire, so properly anticipating and managing for these less developed areas helps the hired candidates be more successful at your company.
As Robin Williams’ aptly notes in Goodwill Hunting “You’re not perfect, sport, and let me save you the suspense: this girl you’ve met, she’s not perfect either”. You should remember that it is OK (and good, even) for someone to have areas for growth. Instead of finding the perfect candidate, your company’s objective is finding candidates that mesh with the organization based on a set of prioritized strengths, abilities, and values. That is where a successful reference calls help – you can filter out those who don’t have the right overlap, and better manage those who do.
Hope the above tips make your team more effective with their reference calls, and in doing so, improves your team’s recruiting efforts, culture, productivity, and happiness!