Computer vision systems are scanning satellite photos to track construction on the Earth’s surface — an exercise in behavior recognition on a global scale.
What’s new: Space-based Machine Automated Recognition Technique (Smart) is a multi-phase competition organized by the United States government. So far, it has spurred teams to develop systems that track large-scale construction in sequential satellite images, Wired reported.
The challenge: Barren earth, dump trucks, and large cranes are common markers of construction sites. But they aren’t always present at the same time, and they may be found in other contexts — for instance, dump trucks travel on highways and large cranes sit idle between jobs. Moreover, different satellites have different imaging systems, orbits, schedules, and so on — a stumbling block for automated classification. In the first phase of the contest, from January 2021 through April 2022, competitors built models that correlate features that were present in the same location but not at the same time, regardless of the image source.
How it works: The Intelligence Advanced Research Projects Activity (IARPA), a U.S. intelligence agency, organized the challenge.
- The agency provided 100,000 satellite images of 27 regions that range from fast-growing Dubai, where the population increased by nearly one million during that time period, to untouched parts of the Amazon rainforest. Roughly 13,000 images were labeled to indicate over 1,000 construction sites shot by multiple satellites at multiple points in time, as well as 500 non-construction activities that are similar to construction. Rather than dividing the dataset, which was made up of publicly available archives, into training and test sets, the agency split the annotations, withholding roughly labeled 300 construction sites for testing.
- The models were required to find areas of heavy construction, classify the current stage of construction, and alert analysts to specific changes. They were also required to identify features in areas of interest including thermal anomalies, soil permeability, and types of equipment present.
- The team at Kitware approached the problem by segmenting pixels according to the materials they depicted, then using a transformer model to track changes from one image to the next. In contrast, Accenture Federal Services trained its model on unlabeled data to recognize similar clusters of pixels.
Results: Judges evaluated contestants based on how they approached the problem and how well their models performed. The jury came from institutions including NASA’s Goddard Space Flight Center, U.S. Geological Survey, and academic labs.
- The judges advanced six teams to the second phase: Accenture Federal Services, Applied Research Associates, BlackSky, Intelligent Automation (now part of Blue Halo), Kitware, and Systems & Technology Research.
- In the second phase, teams will adapt their construction-recognition models to different change-over-time tasks such as detecting crop growth. It will continue through 2023
- The third phase, beginning in 2024, will challenge participants to build systems that generalize to different types of land use.
- Teams are allowed to use the systems they develop for commercial purposes, and all datasets are publicly available.
Behind the news: Satellite imagery is a major target of development in computer vision. Various teams are tracking the impact of climate change, predicting volcanic eruptions, and watching China’s post-Covid economy rebound.
Why it matters: Photos taken from orbit are a key resource for intelligence agencies. Yet the ability to see changes on Earth’s surface is a potential game changer in fields as diverse as agriculture, logistics, and disaster relief. It’s impractical for human analysts to comb the flood of images from more than 150 satellites that observe Earth from orbit. By automating the process, machine learning opens huge opportunities beyond Smart’s focus on national security.
We’re thinking: Large-scale events on Earth are of interest to all of the planet’s inhabitants. We’re glad to see that the contestants will be able to use the models they build, and we call on them to use their work to help people worldwide.