Google asserted its open source bona fides with new models.
What’s new: Google released weights for Gemma-7B, an 8.5 billion-parameter large language model intended to run GPUs, and Gemma-2B, a 2.5 billion-parameter version intended for deployment on CPUs and edge devices. Each size is available in two versions: pretrained base model and one fine-tuned to follow instructions.
How it works: Gemma models are based on the architecture used in Google’s larger Gemini. Unlike Gemini, they’re not multimodal.
- Gemma-2B and Gemma-7B were trained on 2 trillion and 6 trillion tokens, respectively, of English-language web documents, mathematics, and code snippets. They can process 8,192 tokens of context.
- The fine-tuned versions underwent further training: (i) They received supervised fine-tuning on human-written prompt-and-response pairs as well as synthetic responses that had been filtered for personal information, toxic responses, and other objectionable material. (ii) They were aligned using reinforcement learning with human feedback, in which their output was judged by a model trained on preferences expressed by users.
- Gemma’s license permits commercial use but prohibits a wide range of uses that Google deems harmful including copyright infringement, illegal activity, generating misinformation, or producing sexually explicit content.
- Gemma-7B ranks higher than comparably sized open models including Meta’s Llama 2 7B and Mistral-7B, according to HuggingFace’s Open LLM Leaderboard. By Google’s assessment, it outperforms the nearly double-sized Llama 2 13B in major question answering, reasoning, math, and coding benchmarks. Gemma-2B falls short of the most capable models of its size such as the 2.7-billion-parameter Phi-2.
Behind the news: Google has a rich history of open source AI projects including AlphaFold, TensorFlow, several versions of BERT and T5, and the massive Switch. Lately, though, its open source efforts have been overshadowed by open large language models (LLMs) from Meta, Microsoft, and Mistral.ai. LLMs small enough to run on a laptop have opened open source AI to an expanding audience of developers.
Why it matters: Gemma raises the bar for models of roughly 7 billion parameters. It delivers exceptional performance in a relatively small parameter counts, expanding the options for developers who are building on top of LLMs.
We’re thinking: Gemma confirms Google’s commitment to open source and outperforms top open models of equal size. It’s likely to spur further innovation, especially in AI for edge devices, and keep the Google name in front of enterprising open source developers.