Am I prompting wrong?

I want to create a storybook for my young children. I've written a story and want Gemini/Nano Banana Pro (I have Gemini Plus) to create the images. We've determined a visual style and look for the protaganist, and this is used as a reference image. However, it relies on it too much.

Image 1 is the reference image. I prompt something like:
This is what I want Hubert to look like. Don't add other colors to his tail and mane and make sure he stays the same age/height throughout the images. This picture will be used as scene 1.1, him just standing in a meadow with the junkfood pictured. Use this image as reference for the artstyle and Hubert's looks, not for the composition. Every scene should have a distinct cloud formation and foliage, no repeated patterns. Please create scene 1.2 in which we see Poepoe running down a hill, chased by a toilet roll. We see Hubert spring into action to save Poepoe.

Then it comes back with image 2. It added colors to his tail and mane and most of the grass and plants in the foreground are exactly the same even though I specifically told it not to.

I prompt it to change it, but it just keeps it mostly the same. Also, after a while of repainting it with prompts, it gets blocky and pixelated like in image 3.

How do I get it to use a reference image for artstyle and character consistency, without it just duplicating elements? And how do I get it to repaint certain elements without it losing quality?

Thanks in advance!