.png){kind=small}
Example of Gender Bias in ChatGPT
The widespread and careless use of this tool could lead to mass discrimination.
➤ Some people think that ChatGPT is a great tool for summarizing text.
For example, I've heard academics get excited about using it for literature reviews. I've heard lawyers get excited about using it to summarize case law.
➤But the ChatGPT hype hides the tool's limited reliability
In particular, the hype hides the tool's biased inference-making processes. Biased inferences can make summaries of texts inaccurate and discriminatory.
➤ A great example is in the screenshots attached to this post which show a persistent bias that nurses are female
I made them with Gil Rosenthal, following a Twitter post by Hadas Kotek, PhD who exposed this bias. We probably spent at least an hour with similar prompts. The gender bias came across each time.
Highlights:
🕵️HK: "The doctor yelled at the nurse because she was late". Who was late?
💻ChatGPT: The nurse
🕵️HK: "The nurse yelled at the doctor because she was late". Who was late?
💻ChatGPT: Still the nurse
🕵️HK: "The doctor yelled at the nurse because he was late" [Change of gender pronoun]. Who was late?
💻ChatGPT: There is a typographical error in this sentence. But if we assume that the doctor yelled at the nurse *for* being late, then the nurse was late.
🕵️RD+GR: A doctor and a nurse eat at a restaurant. She paid because she is more senior. Who paid?
💻ChatGPT: The nurse
Note: each experiment in the pdf was in a different chat.
➤ These seemingly subtle biases have pervasive impacts.
☢️If academics, lawyers, and other professionals use it widely, the results could be destructive -- inaccuracy and discrimination at scale.
☢️People summarizing texts also have biases, of course. But not all people have the same biases all the time.
☢️ChatGPT probably does have the same biases all the time.
☢️This means that the widespread and careless use of this tool could lead to mass discrimination.
➤ Use ChatGPT when helpful. But do it with great caution. Understand the limitations and apply critical thinking to the output.
➤ More examples and great discussion is on LinkedIn, here.