A microscope enhanced with augmented reality is helping pathologists recognize cancerous tissue.
What’s new: The United States Department of Defense is using microscopes that use machine learning models based on research from Google to detect cancers.
How it works: The microscope, which costs $90,000 to $100,000, looks like a typical lab instrument, but it connects to a computer that superimposes the output of computer vision models over the view. Two controlled studies are underway at government hospitals, Defense Department research centers, and at the Mitre Corp., a nonprofit technology lab, where 13 units have been integrated into the regular pathology workflow.
- The Defense Innovation Unit (DIU) partnered with Google to develop the microscope’s software and German optics manufacturer Jenoptik to produce the hardware.
- The DIU and Google developed four machine learning algorithms to detect cancers of the breast, cervix, prostate and, as well as rapid mitosis, the uncontrolled cell division that occurs in cancer. The algorithms were trained on anonymized data from Defense Department and Veterans Affairs hospitals.
- If one of the algorithms detects a tumor, the models outline it, grade its severity, and produce a heatmap that displays its boundaries.
Behind the news: Google researchers proposed an AI-powered augmented reality microscope in 2018, and published its research in Nature in 2019. The U.S. government joined the project in 2020. A 2022 paper demonstrated the breast-cancer algorithm’s success at detecting tumors in lymph nodes.
Why it matters: Cancer can be deadly, and early identification of a cancer’s type — and thus how aggressive it is — is a key to effective treatment. Microscopes equipped with computer vision can help pathologists diagnose tumors faster and more accurately. They also may be useful for training new pathologists to identify cancers visually.
We’re thinking: Some previous medical AI projects, after initial excitement, turned out to be hard to operationalize due to variations in the surrounding environment and other factors. The relatively controlled nature of pathology samples seems like a good bet for deployment of augmented-reality microscopes. We look forward to the conclusions of the currently ongoing studies.