AI wanted to be neutral... but then it discovered our biases.

by Rosie Audino, expert in philosophy and communication of science and health

‍

Tall, blonde, and with machine guns for breasts: that's the Fembots, the female-looking artificial intelligences in Austin Powers, a parody of 1960s spy films. It's all in good fun, yet some gender stereotypes and biases can truly exist in AI. 

When we talk about FemTech, however, we're not talking about hypersexualized robots, but a rapidly growing technology sector. FemTech is technology designed around people's characteristics, recognizing gender differences and promoting equity and inclusivity in the world of health and well-being. FemTech arises from a very concrete need: the lack of data on the reproductive and sexual health of women and non-binary individuals creates a shortage of adequate products, treatments, services, and technologies for women and underrepresented people. The goal is to bridge this gap and leverage technology to make healthcare more accessible and representative for everyone.

‍

And this is where the crucial point comes in.

‍

Artificial intelligences, both generic ones and those beginning to be used in healthcare, risk reproducing the same prejudices that already exist in traditional medicine (Cirillo, D., Catuara-Solarz, S., Morey, C. et al. Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. npj Digit. Med. 3, 81 (2020). https://doi.org/10.1038/s41746-020-0288-5)

‍

Why?

‍

Because they are trained on data that already contains those biases. In short: just as Austin Powers' Fembots reproduce gender stereotypes, many healthcare AIs also end up, unintentionally, doing the same thing.

‍

But before addressing the issue, let's take a step back. Let's first try to understand how it's used, what its potential is in healthcare, and what it means to train an AI.

‍

Artificial intelligence is revolutionizing medicine

‍

Imagine arriving at the emergency room on a Saturday night, with the waiting room so full you hope you don't run into your aunt there too. You sit down, clutch your bag, wait for triage. "Tell me..." "I have a pain here, in my jaw... maybe I slept badly." Around you, chaos: someone calls a doctor, a computer freezes, a patient asks for the fourth time where the bathroom is. Now imagine that next to the nurse there's an artificial intelligence system that, while you're still trying to figure out exactly where your jaw pain is, analyzes all your symptoms in two seconds and flags: "Warning: this set of symptoms could indicate a greater risk than it seems. Code red." So the person who triaged you looks at the screen, looks at you, and in an instant switches from "I have 300 things to do" mode to "Okay, let's get them in right away." Is it science fiction? A virtual triage assistant doesn't exist? Not exactly; systems capable of supporting staff in risk assessment already exist, and are currently being piloted in some Italian regions.

‍

Speed makes the difference. AI analyzes in a few moments information that would take a human much longer, cross-referencing symptoms, vital signs, blood pressure, oxygen saturation, heart rate, and comparing them with millions of similar cases. Where healthcare staff rely on experience, AI draws from a potentially infinite reservoir and processes this information very quickly, faster than Road Runner with Wile E. Coyote on his heels.

But there's a catch...

‍

AI depends on the datasets it's trained on, and the less representative they are, the more they can reproduce biases and gender stereotypes (even in medicine)

‍

When it comes to health, data on women and trans or non-binary individuals is often insufficient or completely absent, a phenomenon known as the Gender Data Gap. This is why many medical AI systems do not consider sex and gender as fundamental variables at all, despite their influence on health differences between individuals (Cirillo, D., Catuara-Solarz, S., Morey, C. et al. Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. npj Digit. Med. 3, 81 (2020). https://doi.org/10.1038/s41746-020-0288-5). A very clear example: if, in this data reservoir from which the algorithm draws, a heart attack almost always appears associated with "chest pain," (on this topic, I refer you to the blog article "So Sweetly Complicated... or Rather, Ignored!") AI will learn to recognize it as the main symptom. But if symptoms more common in women (jaw pain, nausea, headache) rarely appear alongside the word "heart attack," the algorithm will not consider them relevant, even though they absolutely are. 

‍

It has already happened. In the United States, Optum's healthcare algorithm, used to identify patients who would develop serious conditions in the future and thus require more expensive interventions, proved discriminatory against Black individuals. To estimate the need for care, the algorithm used past healthcare spending. Those who had spent more were considered more "at risk." But because Black individuals, for economic and social reasons, spend less on average for their care, the system underestimated their condition. This fact does not concern gender stereotypes, but it shows how biases in data generate discrimination in results (Obermeyer, Ziad, et al. “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.”).

‍

Chatbots and Health: AI in Our Daily Lives

‍

Chatbots (or rather: conversational assistants based on a language model, like Chat GPT and Gemini) are probably the most recognizable AI because we actually talk to them and actively use them in our daily lives. Who hasn't asked ChatGPT for health advice? Okay, as long as you then consult a doctor. A case published in Annals of Internal Medicine reminds us of this: a 60-year-old man developed bromine poisoning after asking a chatbot what he could use to replace salt. After taking sodium bromide for three months, he ended up in the hospital with nausea, acne, gastrointestinal issues, depression, and hallucinations (Annals of Internal Medicine: Clinical Cases (2025). “A case of bromism influenced by use of artificial intelligence.”).

‍

Generic chatbots you find online are convenient, but they can't replace a doctor: they don't interpret clinical complexity, read test results, or know the individual, their body, or their history. The answers they provide are the result of statistical calculations based on an infinite reservoir of data they can process at lightning speed. 

‍

More importantly, like all AIs, they also risk reproducing gender biases already present in the data they were trained on, or those stemming from the representational gaps in the data used.  

‍

So, Geen? 

‍

This is the scenario in which Geen was created, with a clear objective: to use AI to facilitate the connection between people and health and wellness services. It does this by building and training its models on the most representative datasets possible, truly accounting for the differences and diversity among people. Geen collects and analyzes disaggregated data by sex, gender, age, ethnicity, and life and work context, avoiding the reproduction of that "neutral" model which, in practice, coincidentally aligns with the body of a middle-aged Caucasian male. 

Geen's AI thus learns to recognize how health experiences manifest in different bodies. Furthermore, it analyzes data from healthcare and wellness providers and services, cross-referencing it with user data to guide them towards more appropriate diagnostic and treatment pathways. Geen can be integrated into other systems, such as platforms for Local Health Authorities, Hospitals, Clinics, and welfare services, helping people find the most suitable service within the facility, thereby reducing healthcare times and costs. Imagine how much easier it could be to search among the services available in your area with Geen's help. 

‍