“Another batch of candidates with almost no one over 30?”
Stacy, George, and their team are a part of the hiring team in a tech company. They were not immediately phased the first time the top candidates recommended were predominantly younger professionals, with almost all men. However, as this pattern continued—they took a pause.
The new Artificial Intelligence (AI) system implemented by the company has been saving them countless hours in filtering out applicants and screening them for final interviews.
When they approached the developers—a team that also consisted of almost all younger professionals—about this pattern, they were brushed aside. The AI system was trained to analyze the past and present employees in the company to identify the types of businesses from which the employers typically hired. It had been designed to automate the hiring process and manage mass hirings, along with being unbiased about selected candidates.
Unfortunately, the AI system exhibited a clear bias, rejecting applications from older candidates, particularly those over 30. This bias seemed to have been unconsciously ingrained into AI during its development.
The AI had learned from data sets heavily skewed towards younger, male applicants and, as a result, developed a bias that caused it to favor certain age and gender.
In the end, the developers could not learn how to de-bias the system, leading to the dissolving of the team.
Unfortunately, this story is not unique.
The use of AI in recruitment and hiring represents 28% of the global generative AI market share, and its use is rapidly increasing.
This shows how important it is to understand and address the root causes of bias to curb it, or it may lead to discrimination, legal risks, and even some serious reputational damage.
First, let’s understand where AI bias stems from.
How AI mirrors human biases in decision-making
Think about it—if an AI recruitment system is trained on historical hiring data from a company where many hires were male, the system may learn to favor resumes that reflect characteristics similar to those of previous male employees.
This can lead to AI systems imitating the biases of their creators, thus mirroring human bias in decision-making and even amplifying existing biases.
For instance, imagine the same tech company where most of its past hires were male developers. This data reflects a preference for certain traits and roles that were predominantly filled by men. When AI is trained on this specific data, it learns from the patterns in these resumes and job descriptions. This means the AI will likely favor candidates similar to those of past successful male candidates, even if it encounters resumes from female candidates who have held similar roles.
In practice, if a female candidate’s experience doesn’t perfectly match the biased patterns—say she has worked in roles that weren’t traditionally filled by women—the AI might unfairly reject her application, rather than assessing her qualifications and skills for the role. The AI could also prioritize male candidates simply because the data shows a preference for them, mirroring the same biases the AI was supposed to address.
AI doesn’t just filter out certain applicants, but the ‘best’ ones
When we say AI is biased and rejects certain applications, chances are—those might be the very best talent you had set out to hire.
Consider a candidate with same skills and background as your top performer applies but gets rejected by AI as their age is older This change could lead to their application being rejected by the AI system even if the organization is not looking for younger candidates, simply because the algorithm favors younger candidates or certain backgrounds, hobbies, or experiences from others.
This isn’t hypothetical, it’s a real concern.
In one documented case, a candidate submitted the same application with only the birthdate changed. The result? They were selected for an interview only because they appeared younger.
Moreover, AI systems can also display biases based on seemingly trivial details. At another company, an AI resume screener was found to give extra points to candidates who listed sports like baseball or basketball—activities traditionally associated with men. On the other hand, those who mentioned playing softball, a sport more associated with women, were less likely to be considered.
This kind of bias doesn’t just intensify existing stereotypes but also leads to the unjust rejection of highly qualified candidates based on irrelevant factors.
It’s risky to rely on AI systems without properly addressing biases.
Now, let’s dive deeper and understand the source of bias to properly mitigate it.
Understanding the depths of AI bias
It’s important to do more than just identify the source of AI bias in recruitment. Understanding them is key. This means delving deep into how and why bias creeped in at various stages during its development. When you grasp the mechanisms behind these biases, you can better implement targeted interventions to mitigate them.
1. Data bias
AI bias in recruitment mostly stems from the data that feeds the algorithms. When the data itself is flawed, it leads to biased outcomes in AI-driven decisions. It occurs in various stages, like:
Data collection: If the data is not diverse or representative, the AI will likely produce biased outcomes.
Data labeling: If the people who label the data have their own biases, these could be passed on to the AI systems.
Model training: If the model architecture is not designed to handle diverse inputs, the AI system may again produce biased outputs.
Selection bias: When the data used to train an AI system doesn’t accurately represent the population it’s supposed to model, selection bias occurs. This happens due to incomplete data, biased sampling, or other factors.
Measurement bias: For example, an AI could be trained to predict job performance based on factors that don’t reflect a candidate’s true potential. It will unfairly overemphasize these factors, even if the candidate is extremely capable.
Confirmation bias: If the data relies heavily on existing trends and pre-existing beliefs, it prevents the AI from recognizing new, fairer patterns.
2. Algorithmic bias
Algorithmic bias arises when machine learning models produce unfair outcomes due to flawed training data biased programming.
Design flaws: Even if the data is unbiased, algorithms can introduce bias through design flaws. Certain features or inputs may be weighed more heavily, leading to biased outcomes.
Unintentional discrimination through algorithm weights: Algorithms used in recruitment may prioritize certain terms of experiences that are more common among one demographic or group.
Feedback loops: The way feedback is incorporated into the algorithm may introduce bias. If a user’s feedback on a recruitment tool favors certain characteristics, the algorithm might become even more biased towards those traits.
Bias reinforcement: This happens when algorithms trained sometimes reinforce societal biases by replicating patterns observed in historical data.
Societal bias: Algorithms that are designed to interact with societal systems can mirror the biases present in those systems.
3. User bias
User bias refers to biases introduced by the people who interact with or use AI systems. This significantly affects how the system is trained, deployed, and used—leading to biased outcomes.
Cultural or contextual bias: Users from diverse cultural or societal backgrounds may have varying biases that influence how they interact with AI systems. This may lead to reinforcement or exclusion.
Design and configuration choices: Decisions about feature selection, model parameters, and evaluation criteria affect AI decision-making.
Training input: Users may introduce bias into AI systems by providing biased or unrepresentative data for training.
4. Generative AI bias
These biases emerge from generative models, which are AI systems designed to create new data based on patterns learned from existing data.
Training data composition: If a generative model is trained on text data where certain groups are underrepresented or stereotyped, the model will generate biased or skewed outputs.
Inadequate data filtering: If the training data includes harmful and biased content, the generative model may learn and reproduce these biases. Example, a generative model trained on online forum data that contains racial or gender-based hate speech might generate content that reflects these.
Strategic actions for business leaders to mitigate AI bias during recruitment
When tackling bias, you must first understand its origins. Only by identifying where the bias stems from can we make meaningful progress. Once you’ve understood the root causes, you can take the strategic steps outlined below to address the issue—whether or not your AI system currently exhibits bias.
1. Team-based actions
To address AI bias, the composition and culture of the team responsible for developing and training AI models play a key role here. Here’s how focusing on team-based actions can mitigate AI bias in recruitment.
- Diversify the team behind AI models
Ensure that the team creating and training these models is diverse. A range of perspectives—across gender, ethnicity, and different backgrounds—enhance the decision-making process and introduce multiple perspectives to the design and evaluation of AI systems.
A recruitment AI built by a team that lacks diversity might favor candidates who fit a narrow profile.
- Promote a culture of ethics and responsibility
Create an environment where team members are encouraged to question assumptions, challenge biased practices, and prioritize fairness in every stage of AI development. Consider establishing clear guidelines and standards for ethical AI development to guide teams in creating systems that do not mirror nor amplify existing biases. Training sessions, workshops, and ongoing education about ethical AI practices must be an ongoing process of your team’s workflow.
This helps prevent biases related and create a more mindful and responsible approach to AI development.
2. AI model-based actions
AI model-based actions focus on ensuring diverse and high-quality training data and implementing bias detection techniques to promote transparency in algorithms. Continuous monitoring of hiring outcomes and incorporating human oversight are vital to validate decisions and catch biases.
At the model level, these are the targeted actions you can implement:
- Pre-process data to improve representativeness
Use techniques like oversampling, undersampling, and synthetic data generation to create a more representative dataset. For instance, a study by Buolamwini and Gebru (2018) showed that oversampling darker-skinned individuals significantly improved the accuracy of facial recognition algorithms for this group. This may have been a result of undersampling the same during the model’s development. Utilize data augmentation or adversarial debiasing to ensure that the AI models learn from a more balanced dataset.
Lastly, don’t forget to document these pre-processing steps.
- Select fairness-focused models
Researchers suggest using model selection methods that prioritize fairness. Techniques like group fairness and individual fairness ensure equitable outcomes across different groups. Additionally, regularization techniques can penalize models for discriminatory predictions, and ensemble methods, which combine multiple models, can help reduce overall bias.
In a practical example, you can apply penalties to models that show bias and use a combination of models to improve fairness.
- Implement post-processing techniques
After an AI model has been trained, post-processing is used to further address any bias. This may involve adjusting the model’s outputs to ensure fairness. Techniques where you balance errors across groups make sure that false positives and false negatives are distributed fairly across different groups. You can correct any residual biases that might affect decision-making.
For example, post-processing methods could adjust hiring recommendations to make sure that candidates from diverse backgrounds have equal chances of being considered, even if the initial model output was biased.
- Adopt comprehensive techniques for generative AI
Start with inclusive data collection to prevent overrepresentation of any single demographic. Use transparency-focused algorithms capable of detecting bias and applying techniques that correct biases. Techniques such as differential testing (comparing outputs for consistencies), exemplified by Columbia University’s Deep Xplore, highlight vulnerabilities in models by comparing outputs and finding inconsistencies.
3. Governance actions
Governance actions oversee and guide the ethical use of AI in recruitment—thereby ensuring that these tools are fair, compliant with regulations, and transparent in their operations. With this, companies can build trust with candidates as well.
- Ethics at the core
It’s important to note that unconscious bias can still find its way into AI models through those who create and train them. You must stay alert to the ethical and compliance issues that come with using AI in HR. For example, ethical oversight should make sure that AI tools are used responsibly and that their impact is regularly monitored to prevent unintended biases in hiring decisions.
- Combining small and big data
Accuracy in AI models improves when user-specific and detailed data along with large, diverse datasets are integrated. This helps identify trends as well as make correct assumptions. In recruitment, combining data on a candidate’s specific skills with broader industry hiring trends leads to more precise and fair decisions.
- Internal ethics governance
Implement regular ethics checks within the organization. For instance, Microsoft has an AI and ethics committee that reviews all AI products before they are deployed. Companies should have internal audits that regularly review AI tools used in recruitment.
- Addressing the “Black Box” problem
AI algorithms can sometimes operate like a ‘black box,’ where the decision-making process is not clear. To counter this, maintain transparency. Google’s Model Card function acts like a manual for algorithms, detailing how they work, their strengths, and weaknesses. This way, you can also demystify AI processes in recruitment, allowing for greater accountability and easier troubleshooting.
- Transparency and communication
Be transparent about how AI is used in the hiring process and ensure that AI tools are sourced from vendors who disclose the methods behind their technology. You can explain how the tool works, where the data comes from, and how it protects candidate privacy.
The future of AI in recruitment
AI technologies will continue to evolve, and their role in hiring will only deepen. This will also offer new ways to improve efficiency, accuracy, and fairness in the recruitment process. AI will increasingly become a partner to HR professionals, handling repetitive tasks and freeing up time for human recruiters to focus on more strategic and interpersonal aspects of hiring.
One of the key trends we will observe is the rise of personalized candidate experiences powered by AI. In fact, future AI systems will be able to tailor communication and job recommendations to individual candidates. This personalization will not only improve candidate satisfaction but also help companies attract and retain top talent.
However, the focus shouldn’t be just on attracting top talent—but finding the ‘right’ talent that truly aligns with your organization’s goals and culture.
