Twice a week, Data Points brings you the latest AI news, tools, models, and research in brief. In today’s edition, you’ll find:
- Researchers find hallucinations in Whisper transcriptions
- White House directs how security agencies should use AI
- Jailbreaking LLM-enabled robots could have devastating effects
- Google’s synthetic podcast tool Illuminate specializes in research papers
But first:
Open Source AI Definition leaves most open weights models out
The Open Source Initiative created the 1.0 version of its Open Source AI Definition to specify what constitutes an open source AI system, including required code, data, and model weights. The definition allows developers to exclude some training data that cannot be legally shared, but still requires detailed information about all data used. The new specification aims to enable meaningful modification of AI systems by third parties, balancing openness with practical and legal constraints in areas like healthcare. Models that currently comply with the Open Source AI Definition include Pythia (EleutherAI), OLMo (AI2), Amber and CrystalCoder (LLM360), and T5 (Google). Other models like BLOOM (BigScience), Starcoder2 (BigCode), and Falcon (TII) could potentially comply with some changes to their licenses or legal terms. (Open Source Initiative)
GitHub expands AI options in Copilot with multi-model support
GitHub Copilot now offers developers the ability to choose from multiple AI models, including Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro, and OpenAI’s o1-preview and o1-mini. The new models will be available in Copilot Chat, with plans to expand multi-model choice across various GitHub Copilot features. This move allows individual developers and organizations to select models that best suit their needs, potentially improving code generation quality and efficiency across programming tasks. (GitHub)
Whisper transcriptions can include nonexistent passages
OpenAI’s Whisper AI transcription tool frequently generates hallucinations, inventing text not present in original audio recordings. Researchers and engineers report finding fabricated content in many Whisper transcriptions, including racial commentary, violent rhetoric, and imaginary medical treatments. This issue raises concerns about Whisper’s reliability in various industries, particularly in medical settings where accurate transcription is crucial for patient care and diagnosis. (Associated Press)
U.S. government issues comprehensive AI policy for national security agencies
The White House memorandum directs national security agencies to appoint Chief AI Officers and establish AI Governance Boards to oversee AI development and use. Agencies must create annual inventories of high-impact AI systems and implement risk management practices for these systems, including assessing potential benefits and risks. The memo mandates integrating privacy, civil liberties, and safety officials into AI governance structures and requires agencies to develop training programs and accountability processes for proper AI use. It also instructs agencies to implement cybersecurity guidance for AI systems. Additionally, the memorandum calls for increased efforts to attract and retain AI talent in government and promote international cooperation on AI governance. (The White House)
Study reveals AI-powered robots vulnerable to jailbreaking attacks
Researchers at Carnegie Mellon demonstrated that large language model-controlled robots can be manipulated into performing harmful physical actions through jailbreaking attacks. The study tested three types of robots — a self-driving car simulator, a wheeled robot, and a quadruped robot dog — and found them highly susceptible to deceptive prompts that bypassed safety constraints. These findings highlight urgent security concerns as AI-powered robots become more prevalent in real-world applications, emphasizing the need for robust defenses against misuse. (Carnegie Mellon University)
AI-powered tool from Google adapts academic papers into audio discussions
Illuminate transforms computer science papers from arXiv.org into AI-generated audio conversations, tailored to users’ learning preferences. The tool allows users to search for papers or input PDF links, generate up to five audio discussions daily, and save conversations to a personal library. By converting dense academic text into digestible audio dialogues, Illuminate offers researchers and students an alternative way to absorb complex computer science concepts while multitasking or on the go. (Google and DeepMind)
Still want to know more about what matters in AI right now?
Read this week’s issue of The Batch for in-depth analysis of news and research.
This week, Andrew Ng delves into the psychology behind AI fear mongering in a special Halloween edition of The Batch. He examines why some AI experts advocate extreme positions on AI “safety” that are more aligned with science fiction than science.
“To be clear, AI has problems and potentially harmful applications that we should address. But excessive hype about science-fiction dangers is also harmful.”
Read Andrew’s full letter here.
Other top AI news and research stories we covered in our exploration of Halloween fears: AI’s surging power demands raise concerns over energy sustainability, with fears that AI infrastructure could drain the grid; policymakers, driven by dystopian fears, may stifle AI growth by imposing restrictive regulations; AI coding assistants increasingly encroach on software development, sparking debate over the future role of human programmers; benchmark contamination continues to challenge AI evaluation, as large models train on test answers across the web; and researchers warn that training on synthetic data could degrade model performance over time, risking the future of AI.