The wait for Gemini felt a bit like waiting for Godot this year.
OpenAI, a relatively small, upstart company, seriously upstaged Google when it fatefully released ChatGPT in November 2022.
When would the tech giants, particularly Google, follow?
Now we know.
On Wednesday, Google CEO Sundar Pichai and Google DeepMind boss Demis Hassabis introduced the world to Gemini, Google’s new generative AI model, which it describes as its most capable and general model yet.
Gemini, which comes in three different flavors named Ultra, Pro, and Nano, is a multimodal AI system, which means it not only processes text, but code, audio, images, and video too to respond to user prompts. It also integrates directly into mobile devices, a first for an AI model, and a point of excitement for app makers.
It’s such an important moment for Google that cofounder Sergey Brin was hands on basically every day with its development, as one Gemini developer put it.
It’s early, but initial reactions suggest Gemini is putting in a good showing against GPT-4, the latest large language model powering ChatGPT.
First, some numbers that illustrate Gemini’s performance chops.
Google says its most powerful Ultra model, rolling out next year, exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks to assess large language models.
TL;DR: Gemini probably is better than GPT-4. Though a closer look at the details suggests the outperformance is only marginal.
On one benchmark, Gemini Ultra has a 74.4% success rate in Python code generation, for example, versus 67% for GPT-4. On another benchmark, Gemini Ultra has a reading comprehension score of 82.4 compared to GPT-4’s 80.9.
As marginal as this is, first impressions of Gemini still appear to be positive among users who are getting a taste of Gemini through Bard, Google’s version of ChatGPT.
Mihir Patel, a research engineer at MosaicML, posted screenshots to X that compared responses from Gemini and GPT-4 to the question what is mamba in deep learning?
Gemini’s response was, per his screenshots, more detailed and also linked out to external research papers. ChatGPT was closer to a smart Wikipedia entry.
Patel’s reaction: Gemini is so good. SO much better and SO much faster than GPT-4.
Another demo showed Gemini accurately describing a developing picture of a duck swimming in water. That elicited several semi-jokey Google is so back responses on social media.
Developers will likely welcome Gemini as an interesting alternative to OpenAI’s offering too. Google told the Financial Times that, because the Nano model is built to run natively on its Pixel phones, Android developers would have an easier time building AI apps.
The verdict is still out on just how successful Gemini will be, and whether or not Google can pry users away from ChatGPT with it. Lots of users wanting to test Gemini may have to wait, as the company continues to work on non-English language versions of the models.
Gemini also still appears to be vulnerable to the hallucination issues that have plagued ChatGPT.
Pichai, Hassabis, and other Google execs will be familiar with the innovator’s dilemma — the idea that big companies risk losing their market leadership if they don’t stay nimble on product development. Early reactions to Gemini suggest there’s life in the old search engine yet.
