# How to Consistently Hire Remarkable Data Scientists Jeremy Stanley, Chief Data Scientist at Sailthru, shares the proven method that helped him build an exceptional team. _This article is by_ **_[Jeremy Stanley](https://www.twitter.com/jeremystan?ref=review.firstround.com)_** _, Chief Data Scientist and EVP Engineering at_ **_[Sailthru](http://www.sailthru.com/?ref=review.firstround.com)_** _, where he’s responsible for building intelligence into the company’s marketing personalization platform. His data science team works on algorithms for prediction, recommendations and optimization._ **Data scientists are trained to handle uncertainty.** The data we work with, no matter how “big” it may be, remains a finite sample riddled with potential biases. Our models tread the fine line between being too simple to be meaningful and too complex to be trusted. Armed with methodologies to control for noise in our data, we simulate, test and validate everything we can. A great data scientist develops a healthy skepticism of their data, their methods and their conclusions. Then, one day, a data scientist is promoted and presented with an entirely new challenge: Evaluating a candidate to become a member of their team. The sample size drops fast, experimentation seems impractical, and the biases in interviewing are orders of magnitude more obvious than those we carefully control for in our work. > Many data science leaders resort to following traditional hiring practices — but they shouldn’t. In setting out to build my latest team, I spoke with many data science leaders to gather their ideas and best practices. I was especially influenced by the ideas of **[Riley Newman](https://www.linkedin.com/in/rileynewman?ref=review.firstround.com)**, Head of Data Science at [Airbnb](https://www.airbnb.com/?ref=review.firstround.com), who [designed and implemented](https://www.quora.com/How-does-Airbnb-hire-data-scientists?ref=review.firstround.com) a radically different way of recruiting data science talent, and who I spoke to several times while devising the system I'll share with you here. I also learned a great deal from **[Drew Conway](http://drewconway.com/?ref=review.firstround.com)** at [Project Florida](http://projectfla.com/?ref=review.firstround.com), who has continually evolved his hiring process to select for talent that could squarely land in the middle of his [famed data science venn diagram](http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram?ref=review.firstround.com): In this article, I will outline the goals of a new process that Riley developed and I adapted based on this research, describe its underlying principles, and walk through the implementation we have experimented with at Sailthru. And of course, this guide wouldn’t be complete without looking ahead at opportunities to adapt and improve the process even further. ## How to Start a Recruiting Revolution In developing our recruiting process, we set out to improve the following measurable objectives: - **Accuracy**: Maximize the chances that new hires will become exceptional employees. - **Loss**: Minimize the chances that great prospects leave the hiring funnel early. - **Success**: Maximize the chance that offers will be accepted. - **Effort**: Minimize the long-term distraction to the hiring team. At first glance, any experienced manager would think that it’s impossible to improve all four of the above goals simultaneously. The first three tend to work against each other in practice (e.g., the greater the candidate, the harder it is to get them to accept an offer). Beyond that, improving them all would seem to dictate greater ongoing effort by the team. In a traditional hiring process, most managers feel fortunate if their accuracy is as high as 50%. That is, no more than half of their hires turn out to be exceptional. Loss is hard to measure (after all, candidates who fall out of the process didn’t come to work for you), and most managers worry that they regularly lose amazing talent because their process is so long and cumbersome. > In a competitive field like data science, strong candidates often receive 3 or more offers, so success rates are commonly below 50%. And the ongoing effort that hiring requires can easily consume 20% or more of a data science team’s time. After validating this experience with other data science leaders, I sought to implement a process that could achieve the following: - **Accuracy**: 90% of hires should in fact be exceptional employees. - **Loss**: We should make offers to 80% of the great candidates who enter our funnel. - **Success**: 65% of offers extended should be accepted. - **Effort**: Hiring should consume less than 10% of the team's time. By designing a hiring process that is smarter — both in identifying great candidates and simultaneously reducing the risk of losing them — it’s possible to improve on the first three goals simultaneously. And, by investing heavily upfront (an investment that pays off handsomely over time), the ongoing effort and distraction to the team can be managed. **To ensure that we met our objectives, we developed a set of core principles that can be applied to hiring for any function**. Principles that keep everyone focused and aligned can significantly help any big process change. Here they are: **_Ensure your hiring process is always on and continually improving_**. It’s common to think about hiring as either a task that you occasionally participate in, or as a blitzkrieg campaign that is periodically all-consuming. Instead, architect your hiring process to be an engine that is always on, with a predictable funnel of talent moving through clear stages. This ensures that you’re always recruiting, and that whenever great talent comes to the market, you’ll have the opportunity to engage. Investing in an always-on process will force you to treat hiring as a discipline. This will drive consistency in protocol and results, enable you to collect data about your successes and failures, and force you to manage your talent pipeline with the same care you manage your data pipelines. **_Make your process mirror the reality of your hiring needs._** > The brutal truth: Standard interviewing questions are fatally flawed. Ask candidates about their prior experience, and you'll discover whether they can articulate what happened around them at other jobs. Ask them technical questions, and you’ll uncover their ability to regurgitate knowledge. Make them solve a 'toy' problem on a white board, and you’ll discover how quickly they solve toy problems. **A candidate that passes all of these hurdles with flying colors may be a completely ineffective data scientist in practice.** To address these flaws, you must first have a very clear understanding of how you want candidates to perform data science. At the highest level, you should be clear on the end product your team will produce. Will it be visualizations and analyses that inform decision makers? Designs and prototypes that are given to developers? Or applications that can be scaled and supported in production environments? Next, you should have a clear understanding of what you want successful candidates to do. Identify five opportunities you would love to see a data scientist tackle. For each, ensure that you have (or could reasonably collect) the data required, and can envision a solution that would be effective even if you couldn’t design it yourself. These opportunities lie at the intersection of the near-term strategy of your company, the feasibility of how your organization or product functions, and the constraints of the data that you currently have or can reasonably generate. Knowing answers to how your team performs data science and what challenges you most want candidates to be able to handle, you can design a hiring process that closely reflects your working conditions. This means you should put candidates into an environment that closely resembles what their ‘day-to-day’ would be. If they can succeed in that environment during the interview process, then their chances of succeeding long-term are much greater. **_Run objective evaluations first to minimize your biases._** > Candidates who would be top performers may fail a traditional interview process. The culprit is interviewer bias. As soon as you enter the room with a candidate, you begin forming opinions (mostly unconscious) about their abilities. There are a wide array of such biases ( [check out the list of 100+ cognitive biases here](https://en.wikipedia.org/wiki/List_of_cognitive_biases?ref=review.firstround.com)), but the most common bias in interviewing is to prefer people who are similar to ourselves. Great data scientists must have very strong quantitative and programming skills. That’s non-negotiable. So we designed our process to test these skills first, then move on to more subjective (yet still measurable) skills like problem solving and communication. Only at the end do we get to the most subjective of all — how the candidate works on a team and fits into the culture. These later stage, more subjective criteria are the most time-consuming to evaluate and are where biases are most likely to creep in. Moving them late in the funnel has the combined benefit of reducing the load on the team (we don’t evaluate culture fit until we’re confident they have the skills we need) and minimizing the risk of losing a great candidate prematurely. **_Design your process to sell the candidate._** Most interviewing processes also fail to sell the highest-quality candidates on the role. Interviews are stressful at best and mundane and tedious at worst. Candidates are often forced to repeat their story to 4 or more interviewers and answer questions for hours on end. Afterward, while they may have been able to ask a few questions of their own, they often struggle to imagine what it would be like to work at the company. They then wait for days to receive feedback that is rarely honest or prompt. So how do you fix something so broken? Create a process where you give candidates the data and problems that reflect the real challenges they’ll face at your organization. On top of that, ensure that your hiring process engages the candidate with your team’s dynamic and culture so that they get a real taste of what it would actually be like to work with you. Each of these candidates should complete the interview process with a trusting sense of what it would be like to join your team. **_Make smart decisions with your team, not in your tower._** No matter how you hire, every manager has to make difficult decisions. To decide with confidence, establish clear frameworks for evaluating candidates at every stage of your funnel. This includes defining objectives and metrics that everyone on your team understands. Also, make decisions openly as a team. This ensures the hiring manager hears direct feedback about candidates from everyone involved in the process. Even more importantly, it makes sure that you’re all looking for the same qualities. An open forum helps to change your recruiting needs and strategy over time. Finally, engage your cross-functional partners in evaluating your candidates. Data science is never truly done in a vacuum. You will collaborate with decision-makers, engineers, and product managers. Involve key partners in those areas so you select talent that can be successful across departments and divides. **_Move faster than the market._** The market for great data science talent is incredibly competitive, so your process should ensure that you move candidates through your funnel as quickly as possible, keeping momentum high and minimizing the chance that they accept a competing offer. Moving fast requires a streamlined process that allows you to build confidence as well as speed. Invest in tools and logistics to track how long candidates stay in each stage of your funnel and aggressively change your system to gain keep your edge. ## The Implementation Game In the movie _[The Imitation Game](https://www.imdb.com/title/tt2084970/?ref=review.firstround.com)_, Alan Turing’s management skills nearly derail the British counter-intelligence effort to crack the German Enigma encryption machine. By the time he realized he needed help, he’d already alienated the team at Bletchley Park. However, in a moment of brilliance characteristic of the famed computer scientist, **Turing developed a radically different way to recruit new team members.** To build out his team, Turing begins his search for new talent by publishing a crossword puzzle in The London Daily Telegraph inviting anyone who could complete the puzzle in less than 12 minutes to apply for a mystery position. Successful candidates were assembled in a room and given a timed test that challenged their mathematical and problem solving skills in a controlled environment. At the end of this test, Turing made offers to two out of around 30 candidates who performed best. There’s a lot to learn from this anecdote. The process ensured that Turing had cast the widest possible net for available talent, attracted them with a challenging problem and intriguing offer for employment, and then validated their skill in a controlled environment. In an apocryphal turn of events in the movie, one of the candidates Turing recruited was a woman named Joan Clarke, who became a very close collaborator. Joan was incredibly talented, but given the biases of the time, would almost certainly have been overlooked for a role on a code-breaking team had it not been for Turing’s scientific approach to hiring. Just like in _The Imitation Game_, we put candidates through a sequence of experiences that approximate their potential working environment and evaluate their skills on problems that are highly predictive of their success once we hire them. Surprisingly, with the right planning and investment upfront, this can be done even more efficiently than in traditional interviews, and spare your team’s time. At the highest level, this interviewing process has two key components: - **Take-home test:** A short exercise that tests a candidate’s ability to solve a series of increasingly difficult challenges. - **Data Day:** A full day spent working beside the team on a more open-ended challenge, concluding with a presentation of their work to a group. We manage this process as a funnel. Of 500 inbound applicants, 250 (50%) will submit a take-home test, 25 (10%) will pass, 20 (80%) will come to the data day, 4 (20%) will pass the Data Day, and then 3 (75%) will accept the offer. That means in order to find a single great hire, we need over 150 applicants. The key levers to pull here are **(A)** the quality of the applicants in the funnel, **(B)** the success rates in submission of take-home tests and attending a Data Day, and **(C)** the accuracy of the take-home test and Data Day filters. By tracking your candidates through this funnel and examining the loss at each stage by channel (e.g. where they came from), you can begin to identify higher performing channels, and the stages in your funnel that are filtering too aggressively. Given our four distinct goals — to maximize _accuracy_ (hire exceptional employees) and _success_ (ensure they accept offers), while minimizing _loss_ (candidates abandoning early) and _effort_ (team distraction) — we invested a significant amount of time in designing a clear and efficient process that is data driven and appealing to candidates. **This process has the following six stages, which move from easiest and most objective to most difficult and subjective:** - **Pre-screen**: Check for a pulse - **Take-home test**: Test for sufficient skill - **Sales pitch**: Convince them to come to the ‘data day’ - **Data day**: Test competence in a realistic, controlled environment and evaluate culture - **Decision**: Make a quick and definitive decision - **Communicate**: Follow up with every data day candidate Let’s take a more in-depth, tactical look at each phase. **1\. Pre-screen** Note, at Sailthru we don’t pre-screen data science candidates at all. We don’t have to review their resume or debate their experience or qualifications. > If they have a pulse (and an e-mail address) we give them the take-home test. It's our version of _The Imitation Game_ crossword. This saves significant time and energy and allows you to engage with promising candidates faster. But the most important reason not to pre-screen is that it removes a huge source of initial bias. Many incredibly talented candidates won’t have the education or experience recruiters are trained to look for. This not only means you lose out on great candidates, but you’re also going to be competing furiously for those few candidates that look good on paper — everyone else wants them too. **2\. Take-home test** The take-home test is incredibly important. It’s the first line of defense in filtering out candidates and requires the most work from your team given the volume of potential submissions. It’s also the first time candidates will get a sense of what your team does. This stage is not only a critical hurdle to prevent you from wasting time on unqualified candidates, but also an extremely important part of selling candidates on the role. For all these reasons, you should continuously evolve this stage of your process as you gather data on candidate performance and interest through your funnel. **A sound take-home test should have the following attributes:** - **Self-explanatory** - **Bounded** - **Desensitized** - **Relevant** - **Direct** - **Gradated** To design your take-home test, first start with the most pressing problems you want your existing data science team to address. Of those problems, pick one or two that you **(A)** have or can fabricate compelling data for, **(B)** will be fun for candidates to solve, and **(C)** can be simplified (perhaps grossly) to something that could be addressed by a very strong candidate in under 2 hours. After you've narrowed the problem space, compile the data needed to answer your take-home questions. Ideally, this is data that comes from your production environment and is sufficiently cleaned, permuted or aggregated so as to be harmless were it to fall into anyone’s hands (assume it eventually will). Once you have assembled the data, craft 2-3 very clear questions that escalate in difficulty and have definitive, measurable answers. Ensure that your questions will test not only the candidate’s ability to manipulate the data, but also their ability to think logically about the analysis and interpret results from any models built. **3\. Sales Pitch** Once a candidate has passed your take-home test, your next challenge is to convince them to come to your “Data Day” interview. Most will be expecting a traditional interview where they spend no more than 4 hours at your office — certainly not the whole day. It’s imperative that you convince them that it’s worth their time. The critical parts of your sales pitch are how you connect with the candidate, how you articulate the exciting opportunity you are presenting, and how you describe and prepare them for the Data Day. It should all be geared toward building their interest and enthusiasm — **this is not the time for you to evaluate them**. Every candidate is motivated by different factors, so it’s crucial for you to listen carefully and direct the conversation to the topics they care about most. In my experience, I have found the following to be key motivators: - The overall potential for the product and company. - How data science is organized, where it reports, and what impact it has had to date. - The key challenges or opportunities data science will be working on in the near future. - How data science works cross-functionally with other teams. - The scope, size and quality of data available, and opportunities for future collection. - How the team manages their work and collaborates in priorities and decisions. - The specific tools and technologies that the team uses. Ultimately, you will find candidates who are unable or unwilling to schedule a data day. In the end, while that may mean that you miss out, you have to be willing to take that risk. > The 'Data Day' becomes a gold standard by which you will evaluate all candidates. **4\. Data Day** The Data Day itself is in many ways the heart of this recruiting process. Done well, it encapsulates the final technical, strategic and skill evaluation of a candidate with an analysis of his/her cultural fit in an experience that really “sells” them on your team and company. With enough preparation, this can be accomplished with no more time commitment from you or your team than would be required by a traditional interview. The prep checklist includes: - **Instructions**: - **Data**: - **Laptop**: Preparation is critical to a successful Data Day. By ensuring the candidates have everything they need to be productive, you can maximize the time they have to accomplish meaningful work. _Instructions_ When a candidate arrives for their data day, the first thing you should provide is a printed set of instructions. The sections to consider including (as concisely as possible) are: **Introduction** - **Disclaimer (perhaps an NDA)** - **Goals** - **Suggested Timeline** - **Data** - **Topics** - **Evaluation** - **Technical Setup** - **Data Details** - > In the end, it will ideally surprise you how the strongest candidates use what you provide. An important consideration is how much you should preprocess the data in advance. In general, unless you specifically want to test for their ability to cleanse very messy data, I would suggest keeping this sample reasonably clean to ensure that they don’t waste valuable hours on munging that would otherwise have gone into analysis or modeling. **Laptop** Provide the candidate with a laptop that has the instructions, data and software all in one accessible place. At Sailthru, we use a MacBook Pro (all data scientists and engineers use Mac or Linux machines), and we install the following software: - [HomeBrew](http://brew.sh/?ref=review.firstround.com) - [Anaconda](https://store.continuum.io/cshop/anaconda/?ref=review.firstround.com) - [R](http://www.r-project.org/?ref=review.firstround.com) - [RStudio](http://www.rstudio.com/?ref=review.firstround.com) - [Emacs](https://www.gnu.org/software/emacs/?ref=review.firstround.com) and [Vim](http://www.vim.org/?ref=review.firstround.com) - [Java 7](https://www.java.com/en/download/faq/java_7.xml?ref=review.firstround.com) - [Eclipse](https://eclipse.org/?ref=review.firstround.com) > With HomeBrew, a data scientist can quickly install other software as needed. Further, we place the data in CSV files in their home directory. We suggest to candidates that they submit their take-home test using an open source scripting language (like Python, [R](http://www.r-project.org/?ref=review.firstround.com) or [Julia](http://julialang.org/?ref=review.firstround.com)) so that everyone is comfortable with it. **Schedule** The schedule for a typical Data Day at Sailthru looks like this: **10 a.m. - Welcome** **10:05 a.m. - Buddy** **10:15 a.m. - Orientation** **10:20 a.m. - Direction** **11:30 a.m. - Stand up** **12:30 p.m. - Lunch** **As needed - Questions** **5:30 p.m. - Presentation** **6 p.m. - Feedback** **6:15 p.m. - Decision** Overall, the time commitment from the team is quite reasonable. The buddy spends 15 minutes in the morning and maybe another 15 minutes in the afternoon answering questions. The stand up and lunch were going to happen regardless. The presentation and Q&A take 30 minutes for a group of five participants, and then the decision typically takes another 15 minutes afterward. In total, the team spends just over 4 person-hours with the candidate, no more than would be spent in a minimalist traditional interview. > The most insightful part of the day from a culture fit perspective is the lunch, where you get to see how the candidates act in an informal, social setting. ### Lessons Learned The Data Day should be a reflection of your team and your company, so you should adapt the process to suit your specific needs. We make a point to ask candidates for feedback at the end of the day, and have already made numerous changes based on their input. Here are some of the best lessons we learned: - **Candidates almost always run out of time**. - **Don’t let lunch run too long**. - **Invite a diverse group to the presentation**. - **Be transparent with candidates about what Data Day entails before they arrive**. **5\. Decision** We evaluate candidates based on the following dimensions at Sailthru: **1\. Problem structuring** **2\. Technical rigor** **3\. Analytical rigor** **4\. Communication** **5\. Usefulness** We include these criteria in the Data Day instructions so that candidates know what success looks like. After candidates complete their presentation and Q&A, their buddy shows them out of the office. Then we immediately begin discussing the candidate while everyone’s impressions are fresh. We give each participant a chance to share feedback on the above criteria (if they have an opinion), starting with people outside our team first. Then the least experienced members of our team speak, followed by the most experienced. This helps prevent the team or the leader from biasing the opinions of others in the room in advance. > In general, if any participant is a strong 'no' on the candidate, that’s reason enough to reject them. **6\. Communication** The final stage of the process is to communicate the outcome to the candidate. Those who fail the take-home test hear back from our recruiter. We would love to give every submission direct feedback, but it’s impractical given the scale of the number of submissions we review. That said, a member of the data science team follows up with every candidate that does not receive an offer after coming in for a data day. This ensures that these candidates receive constructive feedback and can learn more from the experience. Ultimately, we’re excited about the potential of every Data Day candidate and want to both be respectful of their time and stay connected in case our paths cross again. ## Challenges and Future Opportunities Hiring great data scientists is difficult, and while I’m convinced that this process has had a tremendously positive impact on our hiring at Sailthru, I also believe we have much more to learn. Here’s a taste of what we continue to wrestle with. - **Minimizing False Negatives** - **Getting Started** - **Tailoring This Process for Other Functions** ## The Takeaways This hiring process has been truly revolutionary for the data science team I lead. We’ve passed on candidates who seemed perfect on paper and in conversation, but were unable to structure open-ended data problems or defend their analytical choices. > We’ve hired candidates we might never have before.