AI chest X-ray tools show wrong disease spots

Researchers at IIIT-H found that areas highlighted by AI models do not always match regions identified by radiologists when analysing chest X-rays.

Four AI models tested against thousands of scans

The Language Technologies Research Centre at IIIT-H evaluated MAIRA-2, MedGemma-4B, LLaVA-Med-1.5, and LLaVA-1.5 against thousands of publicly available chest X-rays.

Study led by Parameswari Krishnamurthy

The research team, led by Parameswari Krishnamurthy, sought to determine whether AI-generated heatmaps accurately represented regions that radiologists would identify as disease-related.

Syed Faizan explains the research question

Principal investigator Syed Faizan said the team wanted to examine whether heatmaps created by vision-language models correspond to where radiologists would say disease lies.

“We essentially wanted to examine whether the heatmaps created by vision-language models actually correspond to where radiologists, who look at the image, would say the disease lies”

AI may diagnose first, then place heatmap

A model could arrive at a diagnosis first and then use that conclusion to determine where to place its heatmap, rather than independently identifying the affected region.

Removing diagnosis info reduced AI performance

When the team removed diagnostic information and examined how models localised abnormalities, their performance declined, suggesting anatomical expectations influence highlighted regions.

AI models miss clinical context radiologists need

Dr. Faizan said a model focusing closely on a suspected abnormality may not provide all information radiologists need, including how far disease has spread to surrounding areas.

Study addresses gap in AI evaluation methods

Earlier studies focused on prediction and diagnostic accuracy rather than whether AI-highlighted regions matched those identified by radiologists, a gap this research addressed.

AI struggles with different question wording

The LTRC is investigating whether AI models respond consistently to medical questions worded differently, such as asking if pneumonia is present or can be ruled out.

AI should support doctors, not replace them

Parameswari said the laboratory's broader objective is to use AI to assist clinicians with documentation and report writing, reducing administrative work so doctors focus on clinical decisions.