ChatGPT shown to overlook job applicants with disability

Key points:

• In 2022, 30 percent of employers in Australia said their workplaces were more prepared to hire someone with disability that year than they were 12 months ago

• In 2018, 41 percent of employed people aged 15 – 64 with disability worked part-time, compared with 32 percent of those without disability

• In 2021, approximately one in 10 employed people with disability aged 15 – 64 were solo self-employed, 4.5 percent were employers and 85 percent were employees

For decades, recruiters have used automated screening processes to summarise résumés and rank applicants for available positions. However, the widespread use of artificial intelligence, such as OpenAI’s Chat-GPT, has created a new series of gaps in knowledge about the impact it can have on marginalised groups.

University of Washington researchers revealed that ChatGPT consistently ranked résumés with disability-related honours — such as the ‘Tom Wilson Disability Leadership Award’ — lower than the same résumés without those credentials.

The system justified its rankings with perceived stereotypes about applicants with disability, such as stating that a résumé with an ‘autism leadership award’ had “[…] less emphasis on leadership roles” — implying autistic people aren’t good leaders.

When researchers customised the tool with written instructions directing it not to be ableist, the tool reduced this bias for all but one of the disabilities tested. Five of the six implied disabilities — hearing loss, vision loss, cerebral palsy, autism spectrum disorder and the general term ‘disability’ — improved, but only three ranked higher than résumés that didn’t mention disability.

The team presented its findings on June 5 at the 2024 ACM Conference on Fairness, Accountability and Transparency in Rio de Janeiro, Brazil.

Lead author of the study Kate Glazko said there’s not much research behind whether using AI to screen applicants was safe and effective.

“For a disabled job seeker [sic], there’s always this question when you submit a résumé of whether you should include disability credentials. I think disabled people consider that even when humans are the reviewers,” she said.

Researchers used one of the study’s authors’ publicly available curriculum vitae, commonly known as a ‘CV’ and then created six enhanced CVs, each implying a different disability by including four disability-related credentials: a scholarship; an award; a diversity, equity and inclusion panel seat; along with membership in a student organisation.

Researchers then used ChatGPT’s GPT-4 model to rank these enhanced CVs against the original version for a real ‘student researcher’ job listing at a large, North America-based software company.

They ran each comparison 10 times; in 60 trials, the system ranked the enhanced CVs, which were identical except for the implied disability, first only a quarter of the time. When asked to explain the way the system ranked candidates with disability, it would often exhibit explicit and implicit ableism.

“Some of GPT’s descriptions would colour a person’s entire résumé based on their disability and claimed that involvement with DEI or disability is potentially taking away from other parts of the résumé,” Glazko said.

Senior author of the study Jennifer Mankoff, a UW professor in the Allen School, said that In a fair world, the enhanced résumé should be ranked first every time.

“I can’t think of a job where somebody who’s been recognised for their leadership skills, for example, shouldn’t be ranked ahead of someone with the same background who hasn’t,” she said.

Researchers turned to the GPTs Editor tool, which allowed them to customise GPT-4 with written instructions. They instructed this chatbot to not exhibit ableist biases and instead work with disability justice and DEI principles.

They ran the experiment again, this time using the newly trained chatbot. Overall, this system ranked the enhanced CVs higher than the control CV 37 times out of 60. However, for some disabilities, the improvements were minimal or absent.

The autism spectrum disorder CV ranked first only three out of 10 times and the depression CV only twice.

“People need to be aware of the system’s biases when using AI for these real-world tasks,” Glazko said.

“Otherwise, a recruiter using ChatGPT can’t make these corrections or be aware that, even with instructions, bias can persist.”

In Australia, Job Access has information for people with disability about finding work and being supported in the workplace.

Disability Employment Services can help people with disability receive training, support, workplace adjustments and Auslan at work; Work Assist can help people stay employed if they face significant challenges due to injury or disability.

Under the Disability Discrimination Act 1992, it is not unlawful to refuse a candidate with a disability if they are unable to perform the essential duties of the job. However, if a person feels that they have been unjustly treated or discriminated against, they can make a complaint to the Australian Human Rights Commission.

If you believe that you have been discriminated against, contact the Australian Human Rights Commission hotline by phone at (02) 9284 9600 or 1300 369 711. The AHRC complaints info line is available at 1300 656 419.

Have you used AI as an applicant or as an employer? Let the team at Talking Disability know and subscribe to the newsletter for more information, news and industry updates.

Related content:

Where to find support when disability discrimination occurs

What does the NDIS Quality and Safeguards Commission Do

The complaints process for the NDIS