Building and scaling your engineering organization

At least once a week, I meet a manager or founder who wants to scale their engineering organization, which is a natural step in the growth of any organization. Most people think of hiring as an art, luck, or output of brand. I’d argue it is none of these. Like all things related to company building, it is the result of good metrics, good people, good research, constant process iterations, and lots of hard work. 

I can share what I learned building an amazing engineering organization at Demandbase, where I was CTO and SVP of Engineering for almost four years. I spent a third of my time focusing on creating the engineering organization, mostly from scratch. I will break this topic in a series of posts starting with the two most important aspects of engineering recruitment: qualification criteria and the role of interviews. I will follow up with subsequent posts on how to source candidates, pitch your company to engineers, screen them over the phone, check references, and close the deal.

Seeding the organization

The first step is to “seed” the organization because talent begets talent. You cannot build and grow anything without great initial hires. If you have B- engineers as your seed, you will hire only C-level people down the line. 

But all is not lost, and you can break through this bootstrapping problem. Even if you have an amazing technical founding team, you still need to solve it for design or growth. For example, we hired two great security CISO-level advisors and that was good enough to attract brilliant security people. 

Sometimes people ask me, “How did you convince these renowned experts to join your advisory board?” It was easy, we paid them a lot of money per hour. But if you only need them for ten hours a year, it’s not a large investment. You can even start with a single consultant, a technically astute board member, or an amazing head of engineering or director. 

Writing down the qualification criteria

You now have the seed, but the job is far from done. With a good seed team and interviews, you can still get inconsistent talent. Then, once enough below-average people get into an organization, the superstars leave. 

Like any scaled product sales process, the most important part of an engineering recruiting process is a black-and-white set of qualification criteria. Bad qualification criteria are a recipe for disaster because they result in churn, unproductive interviews, and mis-hiring due to fatigue. 

Good qualification criteria consist of an internal checklist or scorecard with items that can be measured by numbers, hard skills, or years of experience. For example, “good communication skills” isn’t a good qualification, but “over three publications in peer-reviewed journals,” “contribution to one or more popular open source projects,” and “mentions the number of customers they worked with in a resume” can be measured and checked.

Managers and recruiters (especially external ones) often dislike qualification criteria because they feel they may miss out on some superstar candidates. This may be correct, but it’s also not relevant. Qualification criteria limit the risk of bad hires, and it’s always better to have an empty seat than the wrong person.

So what’s in the checklist? I think every engineering organization or founder has to build their own. Below you can find one that I have used consistently. And please note, I consider this far more important than interview performance. We have extensive research now to prove that the most important criteria to identifying talent is not the impression they make in an interview but what they have achieved in their past. And so, a successful engineer’s resume should look this:

You want to make sure that someone at a single company gave them increasingly more ownership, and they didn’t have to switch jobs to get promoted. Thus, you want to avoid resumes that look like either of these:

In example three, while the person has been promoted, the resume doesn’t show a good rate of progress, especially as companies often promote people because they have outlasted everyone else in that position. In example four, the person has switched jobs to get promoted and never been upleveled in the same company.

So, what’s the interview for?

The interview is the most error-prone and dangerous part of building an amazing engineering organization. Companies like Google have proven that a candidate’s interview performance is not correlated with success in the organization years later. I’ve personally experienced the same and would advise the interview process to be as cookie cutter and as standardized as possible.  This has the added advantage of being the most fair way to do interviews because personal bias doesn’t unduly sway the process. You are not trying to figure out if a person is a rockstar. You already have a hypothesis on his or her capabilities, and the interview is a validation of that hypothesis. Here are some areas where interviews are useful:

1. Did they actually build what they said they did in their resumes and what role did they play?

2. Do they collaborate well in a team environment? This is best uncovered through behavioral questions. You can break through the subjective nature of soft skill assessment by having multiple people touch on this topic.

3. How do they think about a problem and can you problem solve with them? How well do they communicate?

4. Can they code and solve problems as close to the actual problems on the job as possible? You can often find this out with a take home online exercise using a platform like CoderPad.

The idea of the interview is to verify that the candidate is a fit based on the qualification criteria that you have previously laid out and hasn’t been deceptive in their resume. Lastly, it is best to remove hiring managers from the interview process beyond the initial screening. That’s because a) their opinions can often bias the entire process towards what they think and b) they are susceptible to hiring the wrong person under pressure to get butts in seats. 

As I said before, I consider qualification criteria the most important consideration when making a hire and interviews the best way to validate the experience of a candidate. In my next posts on building and scaling an engineering team, I’ll look at everything from how to find candidates and pitch your company to prospective employees to screening them over the phone and checking references. If I’m leaving anything out, please let me know.