the bottom curve of the e and teh ampersand should be equal or totally different, there is a good reason why every font has his own ampersand.
so in my opinion: keep the style of the font, or make a complete contrast with the ampersand (example: didot italc)