The real-estate website Zillow bought and sold homes based on prices estimated by an algorithm — until Covid-19 confounded the model’s predictive power.
What’s new: Zillow, whose core business is providing real-estate information for prospective buyers, shut down its house-flipping division after the algorithm proved unable to forecast housing prices with sufficient accuracy, Zillow CEO Rich Barton told investors on a quarterly conference call. Facing losses of over $600 million, the company will lay off around 25 percent of its workforce. (A related algorithm called Zestimate continues to supply price estimates on the website.)
What went wrong: The business hinged on purchasing, renovating, and reselling a large number of properties. To turn a profit, it needed to estimate market value after renovation to within a few thousand dollars. Since renovation and re-listing take time, the algorithm had to forecast prices three to six months into the future — a task that has become far more difficult over the past 18 months.
- The pandemic triggered a real-estate spree, driving price fluctuations that Zillow’s algorithm, which was trained on historical data, has been unable to foresee. It also disrupted the supply chain for products needed to renovate homes, extending turnaround time.
- The company bought 9,680 houses in the third quarter of 2021, but it sold only 3,032 at an average loss of $80,000 per property.
- Zillow has listed the majority of its remaining inventory in four major markets at prices lower than it paid, according to an analysis by Business Insider.
What the CEO said: “Fundamentally, we have been unable to predict future pricing of homes to a level of accuracy that makes this a safe business to be in,” Barton explained on the conference call. “We’ve got these new assumptions [based on experience buying and selling houses] that we’d be naïve not to assume will happen again in the future we pump them into the model, and the model cranks out a business that has a high likelihood, at some point, of putting the whole company at risk.”
Behind the News: Zestimate began as an ensemble of roughly 1,000 non-machine-learning models tailored to local markets. Last summer, the company revamped it as a neural network incorporating convolutional and fully connected layers that enable it to learn local patterns while scaling to a national level. The company is exploring uses of AI in natural language search, 3D tours, chatbots, and document understanding, as senior vice president of AI Jasjeet Thind explained in DeepLearning.AI’s exclusive Working AI interview.
Why it matters: Zillow’s decision to shut down a promising line of business is a stark reminder of the challenge of building robust models. Learning algorithms that perform well on test data often don’t work well in production because the distribution of input from the real world departs from that of the training set (data drift) or because the function that maps input x to prediction y changes, so a given input demands a different prediction (concept drift).
We’re thinking: Covid-19 has wreaked havoc on a wide variety of models that make predictions based on historical data. In a world that can change quickly, teams can mitigate risks by brainstorming potential problems and contingencies in advance, building an alert system to flag data drift and concept drift, using a human-in-the-loop deployment or other way to acquire new labels, and assembling a strong MLOps team.