Dear friends,

Welcome to our special Halloween issue of The Batch, in which we probe fears, anomalies, and shadows of AI.

In this letter, I’d like to explore why some people who are knowledgeable in AI take extreme positions on AI “safety” that warn of human extinction and describe scenarios, such as AI deciding to “take over,” based less on science than science fiction. As I wrote in last year’s Halloween edition, exaggerated fears of AI cause real harm. I’d like to share my observations on the psychology behind some of the fear mongering.

First, there are direct incentives for some AI scientists and developers to create fear of AI: 

• Companies that are training large models have pushed governments to place large regulatory burdens on competitors, including open source/open weights models.
• A few enterprising entrepreneurs have used the supposed dangers of their technology to gin up investor interest. After all, if your technology is so powerful that it can destroy the world, it has to be worth a lot!
• Fear mongering attracts a lot of attention and is an inexpensive way to get people talking about you or your company. This makes individuals and companies more visible and apparently more relevant to conversations around AI.
• It also allows one to play savior: “Unlike the dangerous AI products of my competitors, mine will be safe!” Or “unlike all other legislators who callously ignore the risk that AI could cause human extinction, I will pass laws to protect you!”
• Persuading lawmakers to place compliance burdens on AI developers could boost one's efforts to build a business that helps AI companies comply with new regulations! See, for example, this concerning conflict of interest from a prominent backer of California’s proposed AI safety law, SB-1047. 

I’ve seen people start off making mild statements about dangers of AI and get a little positive feedback in the form of attention, praise or other rewards, which encouraged them to double down and become more alarmist over time. Further, once someone has taken a few steps in this direction, the psychological effect known as commitment and consistency bias, where one feels obliged to stay consistent with one’s earlier statements, will lead some people to keep going in this direction.

To be clear, AI has problems and potentially harmful applications that we should address. But excessive hype about science-fiction dangers is also harmful.

Although I’m highlighting various motivations for AI fear mongering, ultimately the motivations that underlie any specific person’s actions are hard to guess. This is why, when I argue for or against particular government policies, I typically stick to the issues at hand and make points regarding the impact of particular decisions (such as whether it will stifle open source) instead of speculating about the motivations of specific people who take particular sides. This, too, is why I rarely make issues personal. I would rather stick to the issues than to the personalities.

When I understand someone’s motivations, I find that I can better empathize with them (and better predict what they’ll do), even if I don’t agree with their views. I also encourage expressing one’s own motives transparently. For example, I’m strongly pro the AI community, and strongly pro open source! Still, arguments based on substantive issues ultimately carry the most weight. By arguing for or against specific policies, investments, and other actions based on their merits rather than hypothetical motivations, I believe we can act more consistently in a rational way to serve the goals we believe in.

Happy Halloween!

Andrew

Disembodied Spirits Speak

Listen! Did you hear a rasping whisper say, “Beware”? Was it a rogue superintelligence? Or just a deepfake? We don’t know, but we heard it, too. It warns of machine learning algorithms that would devour electricity to leave us shivering in the cold night air, mislead us with increasingly inaccurate output, and take over the work that gives our lives meaning. In this special issue of The Batch, as in prior years at this season, we face our fears of AI. Stay close to your laptop’s screen. It may be the only light amid the growing darkness.

AI Burns All the Energy

The globe’s growing AI infrastructure requires huge amounts of electricity, possibly more than power providers can generate responsibly. Could AI models suck energy resources dry?

The fear: Demand for AI is skyrocketing, and with it the demand for energy to fuel training and inference. Power-hungry systems will overwhelm our current power sources. If unchecked, they could lead to energy shortages and runaway carbon emissions.

Horror stories: AI companies don’t disclose the percentage of their energy needs that AI consumes, but top companies, led by OpenAI, have pitched the U.S. government to build out new energy sources and infrastructure. The trend is clear: Escalating demand risks tapping out existing power plants, pushing carbon emissions higher, and delaying moves to more sustainable energy sources. 

• A Goldman Sachs analysis predicts that data centers’ electricity needs will increase by 160 percent from 2023 to 2030. AI represents about one-fifth of this growth, or roughly 200 terawatt-hours each year. Wells Fargo forecasts greater consumption, 300 terawatt-hours in the U.S. alone by 2030. This could help boost energy demand in the U.S. by as much as 20 percent, leading electricity providers to increase their reliance on natural gas and other fossil fuels.
• Demand for AI is reviving coal-fired plants that previously were laid to rest and reversing plans to decommission others. In Virginia and elsewhere, utility companies have delayed planned transitions to green energy to keep up with the AI boom. 
• Each Nvidia GPU that uses the next-generation Blackwell architecture consumes nearly twice as much energy as a current top-of-the-line Nvidia H200. Nvidia is on track to manufacture 1.5 million of these units by 2027. According to one estimate, Nvidia servers alone could consume 85 to 134 terawatt-hours of electricity by 2027.
• Tech giants that have pledged to reach zero net carbon emissions are falling behind their goals. Earlier this year, Google reported that its emissions of greenhouse gasses rose 48 percent in 2023 compared to 2019. Microsoft and Meta face similar challenges. All are using more low-carbon energy, but increases in overall energy consumption are pushing up their consumption of fossil fuels, too.
• Amazon, Google, and Microsoft are investing in nuclear energy alongside solar and wind. The new nuclear plants are not expected to begin generating power until the 2030s.

How scared should you be: The rapid growth of AI poses a sharp dilemma: How can we meet demand without releasing greater and greater amounts of heat-trapping greenhouse gasses into the atmosphere? AI companies’ two-pronged strategy of lobbying governments and investing in carbon-free energy resources suggests the problem requires both short- and long-term approaches.

Facing the fear: While AI poses a difficult problem for the world’s energy consumption, it’s also an important part of the solution. Learning algorithms are reducing energy consumption and managing distribution. They can help capture and store carbon dioxide from energy plants and manufacturers before it reaches the atmosphere. AI is also helping to monitor the atmosphere, oceans, and forests so we can understand the impacts of climate change and make policy accordingly. And processing in centralized data centers — as power-hungry as they are — is far more energy-efficient than using local servers or edge devices. Ongoing AI development will make such efforts more effective and help us build a more sustainable future.

Innovation Can’t Win

Politicians and pundits have conjured visions of doom to convince lawmakers to clamp down on AI. What if terrified legislators choke off innovation in AI?

The fear: Laws and treaties that purportedly were intended to prevent harms wrought by AI are making developing new models legally risky and prohibitively expensive. Without room to experiment, AI’s benefits will be strangled by red tape.

Horror stories: At least one law that would have damaged AI innovation and open source has been blocked, but another is already limiting access to technology and raising costs for companies, developers, and users worldwide. More such efforts likely are underway.

• California SB 1047 would have held developers of models above a certain size (requiring 10²⁶ floating-point operations or cost $100 million to train) liable for unintended harms caused by their models, such as helping to perpetrate thefts, cyberattacks, or design weapons of mass destruction. The bill required such systems to include a “kill switch” that would enable developers to disable them in an emergency – a problematic requirement for open-weights models that could be modified and deployed anywhere. Governor Gavin Newsom vetoed the bill in October, arguing that it didn’t target real risks and that it could have unintended consequences, but legislators may yet introduce (and the governor could sign) a modified bill.
• The European Union’s AI Act, implemented in August 2024, restricts applications deemed high-risk, such as face recognition and predictive policing. It subjects models to strict scrutiny in essential fields like education, employment, and law enforcement. It also requires developers to provide detailed information about their models’ algorithms and data sources. But critics argue that it could stifle European companies’ early-stage research. Meta restricted Llama 3’s vision capabilities in the EU, which may run afoul of the union’s privacy laws, and Apple delayed launching AI features in Europe due to regulatory uncertainties. Meta, Apple, Anthropic, TikTok, and other leading companies did not sign the EU’s Artificial Intelligence Pact, which would have committed them to comply with certain provisions of the AI Act before they take effect.
• In September, the U.S, UK, and many countries in Europe and elsewhere signed the Framework Convention on Artificial Intelligence and Human Rights, Democracy, and the Rule of Law. This treaty, which will take effect by the end of the year, requires that AI models respect democracy and human rights. It’s legally binding on signatories and may be enforceable by the council’s international Court of Human Rights. In practical terms, though, each member can impose its own definition of democracy and human rights, potentially creating a patchwork of legal uncertainties and burdens for AI companies worldwide.
• China has passed a number of laws that focus on reducing AI’s potential harms by exerting strong government control. Key laws require companies to label AI-generated output and disclose training sets and algorithms to the government, and mandate that AI-generated media align with government policies on inappropriate speech. Some companies, like OpenAI and Anthropic, have restricted their offerings in China.

How scared should you be: The veto of SB 1047 was a narrow escape for California and companies and labs that operate there. Yet regulations like the AI Act are poised to reshape how AI is trained and used worldwide. History suggests that restrictive laws often lead to more caution and less experimentation from technologists.

Facing the fear: AI needs thoughtful regulation to empower developers to help build a better world, avoid harms, and keep learning. But effective regulation of AI requires restricting applications, not the underlying technology that enables them. Policymakers should align with a wide range of developers – not just a few that have deep pockets – to address harmful applications without stifling broader progress.

No Work for Coders

AI coding assistants are brewing codebases that once were the sole province of human programmers. Will AI systems take over software development?

The fear: Programming jobs will vanish as tireless AI agents plan, write, debug, and document code as well as or better than humans. Software engineers will find themselves wandering the job market like restless spirits.

Horror stories: Since 2020, AI-powered coding tools have advanced from completing individual lines of code to generating complex programs. More and more coders work with an automated assistant. These tools are poised to take over more and more of the development cycle as they evolve.

• Microsoft’s GitHub Copilot took advantage of OpenAI’s large language models to become one of the first popular programming assistants, suggesting completed lines of code within popular development environments like Visual Studio. In a Github study of Accenture developers who used Copilot, 70 percent of respondents reported expending less mental effort while using the system. More than half rated it “extremely useful.” In an independent study, Copilot boosted developers’ productivity.
• Amazon CodeWhisperer and Cursor auto-complete code in languages like Python, Java, JavaScript, and C#. CodeWhisperer also flags lines that closely resemble open-source projects to facilitate proper licensing. Cursor allows developers to choose the underlying large language model, a capability that Copilot plans to add in coming weeks.
• OpenAI’s o1 promises reasoning in which the model breaks down complex problems into steps. Integrated into tools like Aider, o1 extends AI’s role to project planning, architecture design, and documentation.
• Replit Agent, Devin, and OpenHands bill themselves as full-fledged automated engineers. Replit Agent streamlines programming by generating code, fixing bugs, and managing project dependencies within Replit’s platform. Devin and OpenHands accept natural-language instructions to generate prototype programs. 
• Anthropic recently introduced an API that controls computer desktops just as humans would — a portent of future agentic programs that take over software engineers’ machines altogether. Future AI assistants could switch among desktop apps to write code, update tickets, message colleagues, and so on. What would be left for programmers to do?

How scared should you be: Nvidia CEO Jensen Huang predicted that AI would make “everybody in the world [a] computer programmer,” while observers fret that Copilot erodes problem-solving skills. But the reality is more nuanced. Research shows that automation is likely to perform certain coding tasks but not entire programming jobs. These tools excel at routine tasks and boilerplate code, but they amplify rather than automate the developer's core skills. Conceptual tasks like specifying what a program should do, collaborating with colleagues, and translating business needs into software design remain the domain of human coders — for now.

Facing the fear: Developers have more to gain by embracing AI assistants than fearing them. These tools don’t just automate tasks; they accelerate learning, refine problem-solving, and enhance programming skills. Developers who master both coding fundamentals and AI assistance won’t just survive — they’ll thrive!

Benchmark Tests Are Meaningless

Large language models are trained on datasets scraped from the web, which includes pages that contain answers to common questions that are used to test the models. How can we evaluate them if they’ve studied the answers before we give them the test?

The fear: Machine learning research marks progress based on trained models’ responses to benchmark problems they didn’t encounter during training. But the solutions to many problems used to evaluate large language models have made their way into popular training datasets, making it impossible to verify progress in precise ways. The state of the art is an illusion and researchers are shooting in the dark.

Horror stories: Researchers have found disturbing signs that the test sets of many widely used benchmarks have leaked into training sets. 

• Researchers tested popular models on both GSM8K, which tests grade-school math problems, and their own set of similar problems. Models including Mixtral 8x22B-Instruct, Microsoft Phi-3-Mini, Meta-Llama-3-8B-Instruct, and Google Gemma 7B achieved scores as much as 10 percent higher on GSM8K than the alternative set. Apparently the models had seen GSM8K’s test set — or similar problems — before.
• Researchers discovered that benchmarks had contaminated the dataset used to train GPT-4. They successfully prompted GPT-4 to reproduce material from AG News (which tests models’ ability to categorize news articles), WNLI (which challenges models to resolve ambiguous pronouns in complex sentences), and XSum (which tests a model’s ability to summarize BBC news articles).
• A 2023 study evaluated GPT-4’s ability to solve competition-level coding problems. The authors found that GPT-4 could easily solve problems in Codeforces contests held before September 2021, but it struggled to solve newer ones. The authors concluded that GPT-4 likely had trained on a 2021 snapshot of Codeforces problems. (Announcing its o1-preview model in 2024, OpenAI mentioned that o1 had scored in the 89th percentile in simulated Codeforces competitions.) 
• Even subjective evaluations like LMSys Chatbot Arena, which pits anonymous chatbots against each other and prompts users to judge which one generated a better answer, can be skewed if developers train their models on prompts that LMSys uses repeatedly. To address this issue, researchers built Arena-Hard and BenchBuilder, which remove the most common prompts.

How scared should you be: Leakage of benchmark test sets into training sets is a serious problem with far-reaching implications. One observer likened the current situation to an academic examination in which students gain access to questions and answers ahead of time — scores are rising, but not because the students have learned anything. If training datasets are contaminated with benchmark tests, it’s impossible to know whether apparent advances represent real progress.

Facing the fear: Contamination appears to be widespread but it can be addressed. One approach is to embed canary strings — unique markers within test datasets like BIG-bench — that enable researchers to detect contamination by checking whether a model can reproduce them. Another is to continually enhance benchmarks with new, tougher problems. Of course, researchers can devise new benchmarks, but eventually copies will appear on the web. Alternatively, they can keep new benchmarks under wraps and run them only on private servers.

Synthetic Data Distorts Models

Training successive neural networks on the outputs of previous networks gradually degrades performance. Will future models succumb to the curse of recursive training?

The fear: As synthetic text, images, videos, and music come to make up an ever larger portion of the web, more models will be trained on synthetic data, and then trained on the output of models that themselves were trained on synthetic data. Gradually, the distribution of the generated training data will deviate ever farther from that of real-world data, leading to less and less accurate models that eventually collapse.

Horror stories: Many state-of-the-art models are trained on data scraped from the web. The web is huge, but it’s not large or diverse enough to provide endless amounts of training data for every task. This tempts developers to train models on data generated by other models, even as the web itself becomes increasingly overrun by synthetic data.

• Last year, researchers from Oxford, Cambridge, and Imperial College London warned of model collapse in their paper, “The Curse of Recursion: Training on Generated Data Makes Models Forget.” At around the same time, a different study also found that models trained primarily on synthetic data suffered sharp declines in diversity and quality of output.
• In addition, builders of AI systems have incentives to train their models on synthetic data. It’s easier, faster, and cheaper to generate data than to hire humans to collect or annotate existing data.
• Generated media arguably is free of copyright, so training on it reduces the risk of lawsuits and the model regurgitating copyrighted materials in its training set. Similarly, generated data is less likely to include personally identifying information, such as medical images, that would pose a risk to privacy if a model that was trained on a dataset that included such information were to regurgitate it.

How scared should you be: Training on synthetic data is at the heart of some of today’s best-performing models, including the Llama 3.1, Phi 3, and Claude 3 model families. (Meta showed that using an agentic workflow with Llama 3.0 to generate data — rather than generating data directly — resulted in useful data to train Llama 3.1.) This approach is essential to the technique known as knowledge distillation, which makes smaller, more parameter-efficient models. Moreover, it’s valuable for building models that can perform tasks for which little real-world data is available, for instance machine translation models that can handle languages spoken by relatively small populations. Although the authors of “The Curse of Recursion” found that training a series of models, each exclusively on the output of the previous one, leads to rapid degradation in performance, introducing even 10 percent real-world data significantly curbed this decline.

Facing the fear: Model collapse is not a near-term risk, and perhaps not any risk at all, given research progress on generating synthetic data. Still, it makes sense to track the presence of generated data in training datasets and include it carefully. The large-scale web dataset Common Crawl captures regular snapshots of the web. If generated data were to inundate the online environment, using an earlier snapshot would eliminate a huge amount of it. More broadly, model builders increasingly curate high quality data, and whether a given example appears to have been generated will become a factor. Datasets can be filtered using algorithms designed to identify generated content. Increasing use of watermarking would make the job still easier. These measures will help developers ensure a healthy balance of real and generated data in training sets for a long time to come.