Google AI can now interpret medical images (like skin rashes)

It's called Amie ( Articulated Medical Intelligence Explorer ) and it's the artificial intelligence system designed by Google to answer patients' questions. The first version was presented in January 2024 but an updated model is already being studied, based on Gemini 2.0 Flash and capable of interpreting - and requesting - not only textual information but also images and other clinical documents. In a study not yet subjected to the peer review process, published on the arXiv platform, this latest Amie model is said to have obtained higher scores than general practitioners in terms of diagnostic accuracy and the ability to interpret certain data or images. However, the system is still being studied and further research will be needed before it can be potentially used in real contexts.

The Anamnesis of “Dr. Arnie”

Before making a diagnosis, doctors need to gather the necessary information about the patient in front of them, asking specific questions about their medical history or, when necessary, requesting documentation relating to tests conducted in the past. With the new version of Amie, Google intends to simulate precisely this type of interaction. Obviously, we are in a completely different field from that of machine learning applied in the clinical field to assist doctors in the interpretation of imaging tests (for example mammograms), where numerous studies have already been published.

How the study was conducted

To analyze the new performances of the artificial intelligence platform, 20 professional actors were involved, who were asked to talk to Amie or a general practitioner (19 were recruited), simulating 105 clinical situations with related symptoms. The exchanges took place through a chat in which documents of various types could be uploaded, such as images relating to skin rashes or other dermatological problems, or files with the results of an electrocardiogram or blood tests.

Each exchange was then evaluated, through specific questionnaires, by three doctors specialized in cardiology, dermatology and internal medicine (the specialists involved in the evaluation were 18 in total).

AI and Clinical Images

Well, Amie would have obtained higher scores than general practitioners in terms of, for example, the ability to interpret images and the completeness of the differential diagnosis, which allows to distinguish different medical conditions associated with similar symptoms. Furthermore, its diagnostic accuracy would have been more robust in the case of poor quality images. From the point of view of the “patient actors”, the artificial intelligence platform would have often been perceived as more empathetic and trustworthy.

The limits of the "artificial doctor"

Of course, even according to the authors of the study, this type of approach has limitations and risks. First of all, the fact that the interaction via chat, used during the research with the aim of comparing the performance of general practitioners with those of the artificial intelligence platform, precludes the use of important communication channels, such as non-verbal language, which in reality contributes for example to building a relationship of trust with one's doctor. Furthermore, during a visit carried out in person, the doctor has the opportunity to perform more dynamic visual assessments than those made possible by static images.

The next step

Another fact to keep in mind, Google explains , is that the research involved “fictitious” patients, i.e. actors, and that real cases can have a significantly higher degree of complexity. Before being able to potentially exploit this system, they emphasize, further research will be necessary. In this direction, Google is reportedly starting a study in collaboration with the Beth Israel Deaconess Medical Center to evaluate Amie in a real clinical context.