Advice on editting a picture, specifically face swapping.

So I’ve generated an image which i am happy with, except I’d like to face swap this ai person with myself. However Gemini is really stubborn and either does zero changes or completely changes everything, despite me giving clear instructions to not change anything other than the face, and giving it a few images of myself. Are there any prompts I need to include, or is it what it is?