To be clear, I love the Gemini models. I was one of the very few people who already used Gemini 1.5 Pro extensively. Due to certain internal tests I was running, I found a lot of "hidden gems" in this model, which gave me the impression that the model family would be in the top spot in the coming years.

Fast forward: after months of testing and building on 2.5 Pro, I was very hyped for 3.0 Pro.

Now, after one week of extensive testing, I must say the model is so bad that I need to switch to another model provider, especially since 2.5 Pro will go away by the middle of next year.

Keep in mind that these flaws are targeted at the Enterprise world, specifically when using the LLM via API in a RAG setup with a large context to handle.

What are the three major flaws?

1. It is utterly bad at following instructions

It simply won't follow instructions well, regardless of the context window size. Running the models (2.5 Pro vs. 3.0 Pro) side-by-side was truly baffling, as 3.0 Pro failed time and time again to even grasp what it should do, while 2.5 Pro understood it all the time. It gets even worse as the chat progresses, the model suddenly and completely forgets major system instructions and becomes a different "being."

2. It hallucinates a lot

The reason I loved 2.5 Pro was its low hallucination rate compared to GPT models. Now it is the opposite. This problem gets even worse as a chat progresses beyond three or four messages.

3. Its a one-shot monster and declines exponentially after that

The first answer is often good, but subsequent outputs quickly deteriorate into very stubborn and weird responses.

So, to be absolutely clear: I am still a huge fan of the Gemini model series. However, after testing and starting with a very positive mindset, I simply cannot understand what makes this model so good. It fails every single test I conduct time and time again, making it unsuitable for use in production.

To be clear, I love the Gemini models. I was one of the very few people who already used Gemini 1.5 Pro extensively. Due to certain internal tests I was running, I found a lot of "hidden gems" in this model, which gave me the impression that the model family would be in the top spot in the coming years.

Fast forward: after months of testing and building on 2.5 Pro, I was very hyped for 3.0 Pro.

Now, after one week of extensive testing, I must say the model is so bad that I need to switch to another model provider, especially since 2.5 Pro will go away by the middle of next year.

Keep in mind that these flaws are targeted at the Enterprise world, specifically when using the LLM via API in a RAG setup with a large context to handle.

What are the three major flaws?

1. It is utterly bad at following instructions

It simply won't follow instructions well, regardless of the context window size. Running the models (2.5 Pro vs. 3.0 Pro) side-by-side was truly baffling, as 3.0 Pro failed time and time again to even grasp what it should do, while 2.5 Pro understood it all the time. It gets even worse as the chat progresses, the model suddenly and completely forgets major system instructions and becomes a different "being."

2. It hallucinates a lot

The reason I loved 2.5 Pro was its low hallucination rate compared to GPT models. Now it is the opposite. This problem gets even worse as a chat progresses beyond three or four messages.

3. Its a one-shot monster and declines exponentially after that

The first answer is often good, but subsequent outputs quickly deteriorate into very stubborn and weird responses.

So, to be absolutely clear: I am still a huge fan of the Gemini model series. However, after testing and starting with a very positive mindset, I simply cannot understand what makes this model so good. It fails every single test I conduct time and time again, making it unsuitable for use in production.