Improving Diversity Does Not Mean Lowering the Bar

In the last few months, I've had this question come up multiple times during conversations. After reading @shaft's article about diversity at Twitter, I realized I wasn't alone [2]. This question is problematic because it assumes the average performance of minority groups across gender and race are somehow below the overall mean. When people utter the phrase, "we want to hire more diversity, but we don't want to lower the bar," I assume they're picturing that the world looks like this:

But interviewing in software engineering is a highly subjective practice as it currently stands. A recent blog post called The Hiring Post notes that engineering hiring practices "would be made only marginally less rigorous if it also involved a goat sacrifice" [1].

The purpose of this post is to debunk the idea that diverse groups in tech are underperformers and that our interview processes are objectively weeding out certain groups based on merit. I'll use data to refute the idea that minority groups are somehow less capable than their majority peers, and suggest ways that companies can track data to recognize when they have created a biased hiring pipeline.

In order to see if there was a significant performance difference across gender and race, I took a look at data from a company called CodePath. CodePath is an organization that offers Android and iOS courses for engineers and designers, and "accelerates the time it takes senior engineers and designers to learn new technologies through in person, project-based, programming classes." The program has a rigorous application process and focuses on making sure that they have a diverse representation of students in their courses.

As a result, 24% of students in CodePath's technical courses are women. CodePath also has an honor roll, which is the top 18% of students in any class. If women perform differently from men in any way, we would find that the percentage of women in the honor roll is significantly different from the percentage of women overall. However, that's not the case. CodePath's honor roll is 29% women. A percentage that, if you run some basic stats, is not significantly different from the percentage of women overall.

So what does this mean, exactly?

It means that you can assume that women will be in the honor roll at roughly the same rate that they exist in the overall population. Until proven otherwise, we can assume that women will follow the same performance distribution as men. So regardless of where a company sets its hiring bar, the proportions of men and women above and below the bar should be roughly the same.

I also took a look at the data by ethnicity to see if there were any significant differences in performance across race. The breakdown by ethnicity of students at CodePath is 31% Asian, 30% Caucasian, 29% South Asian*, 5% Hispanic, 3% Middle Eastern, and 2% Black. The honor roll, or the top 18% of the class, had the following breakdown: 33% South Asian, 29% Asian, 24% Caucasian, 6% Hispanic, 4% Middle Eastern, and 3% Black. These numbers, when run through formulas to calculate statistical significance, are NOT different from the percentages in the overall population. What this means is that, as with gender, there is no evidence that ethnicity impacts performance.

*South Asian describes ethnicities from the Indian Subcontinent.

The question, "we want to hire more diversity, but we don't want to lower the bar," places the blame for performance on the individual without taking into account the idea that the "bar" candidates have to pass is likely biased. The CodePath data shows that race and gender don't have any correlation with performance, so the logical conclusion is that companies should be hiring candidates at the rates they are entering the industry.

What we're finding is that companies are not hiring candidates at the rates they are entering the industry. A USA Today report found that while companies are reporting numbers of black employees at 2-3%, graduation numbers are almost twice that [3]. In other words, "top universities turn out black and Hispanic computer science and computer engineering graduates at twice the rate that leading technology companies hire them" [4].

Furthermore, we also know from a multitude of research studies that the interview process is often significantly biased. One study found that more white-sounding names on resumes resulted in 50% more call-backs [5], and another found that changing the gender of the name favored male applicants [6].

When it comes to hiring more diversity, the problem isn't that tech companies have hiring bars that are too stringent. The problem is that tech companies have biased hiring practices. There is a clear and immediate answer to the question, "How do we get more diversity without lowering the bar?" The answer is to create hiring practices that fairly assess diverse candidates who are already in the industry.

The big question is how do we fix biased pipelines? If subconscious biases are what's keeping diverse candidates out of tech companies, how do we tackle the problem? After all, the reason that subconscious biases are difficult to combat is because...well, they're subconscious. When it comes to issues that involve subconscious bias, the solution is this—don't trust yourself. And the same thing that gives us visibility into complex computing systems can help illuminate our problems with diversity: data.

Data-driven diversity is about using data to track areas of potential bias. These areas are things like the hiring pipeline, promotions. and attrition. It's not about reducing people to numbers, or relying on over-simplified stereotypes to categorize people. Data-driven diversity is about using data to be truly introspective and self-aware as a company so you can identify and fix internal biases.

Knowing the ratios of gender/race among current employees is a good start, but by no means a sufficient amount of data to truly improve diversity at a company. The data needs to be more robust, and when it comes to hiring, companies should be tracking the candidates across gender and race at every point in the interview process. For companies that hire more than 100-300 people a year, those numbers should show that the percentage into the pipeline for a given group is roughly the same as the percentage hired. For companies who hire fewer people than that per year, tracking data will be an important way to prevent bias from seeping into the interview process as the company scales, even if the dataset​ is initially statistically insignificant.

For example, if women exist in software engineering at ~15%, and people who are Hispanic and/or Black exist at ~2-5%, then at least 15% of the people hired should be women and 2-5% of the people hired should be Hispanic or Black. Furthermore, if companies notice that the ratio of candidates entering the pipeline is much higher than the ratio of candidates they are hiring, that indicates a bias with the interview process. In other words, if 5% of candidates coming in are Hispanic and only 2% of candidates that are hired are Hispanic, you have a problem.

Data-driven diversity is not just about increasing the input of diverse candidates, it's about increasing the throughput of diverse candidates. Because why would diverse candidates stick around in an industry that says they want them, but hires them at lower rates than peers from other ethnicities and genders?

Demographic breakdowns of the hiring funnel are just the beginning of the data needed to isolate problems with diversity, and the pipeline extends far beyond the recruiting and hiring process. Knowing the gender and race profile of your company is not enough to fix diversity issues. The same executives who have said, "We want more diversity, but we won't lower the bar" have been unable to state the statistical breakdown of candidates entering their interviewing pipeline versus the actual demographics of those they hire. It's impossible to make an informed decision about how to improve diversity at a company without these basic numbers. And hiring is just the beginning of improving things for diverse groups in tech.

Gathering data about people's gender and ethnicity can be a sensitive practice. There are also a lot of limitations on the personal questions that you can legally ask candidates during interviews. The best way to think about tracking this data is to have a program that allows candidates to optionally participate after the interview process has concluded and a hiring decision has been made. Beyond potentially being illegal, asking a candidate about gender and race before interviews could increase the effect of stereotype threat.

Data should be tracked anonymously and reported in aggregate, and if a company has too little data to do that safely then data should only be used for internal purposes. People should self-report both their gender and ethnicity, since guessing at these characteristics is both impractical and potentially insulting (e.g. categorizing people by ethnicity based on their name). Here is some information on best practices for gathering data on ethnicity as set forth by the FDA.

Whatever you do, make sure to check with a lawyer to ensure your data collection practice is legally adhering​ to state and federal anti-discrimination laws.

• Thomas and Erin. The Hiring Post. Sockpuppet.org, 2015.
• @Shaft. Thoughts on Diversity Part 2. Why Diversity is Difficult. Medium.com, 2015.
• Guynn, Jessican and Elizabeth Weise. Tech jobs: Minorities have degrees, but don't get hired. UsaToday.com, 2015.
• Swartz, Jon. Raise the bar on how we talk about diversity. UsaToday.com, 2015.
• Bertrand, Marianne and Sendhil Mullainathan. Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. 2003.
• Moss-Racusin, Corinne A., John F. Dovidio, Victoria L. Brescoll, Mark J. Graham, Jo Handelsman. Science faculty’s subtle gender biases favor male students. 2012.