Expecting that the person you're with will never be attracted to another person in their life is unrealistic. It's how far they go with that attraction that matters. My wife and I were open about this early on and we agreed that we will both find other people attractive, but the "deal" is that such situations will hopefully remain just attractions. I've had a few temptations along the way and resisted them all. So far, so good.
I really try not to stare at women, because so many women I've known have told me that men staring at them makes them uncomfortable. Plus, it's not the most considerate thing one can do. I also don't always look at women because I find them attractive. Sometimes people are just interesting to look at. I could stare at a man for the same reason. But even if I do look/stare at a woman because I find her attractive, which still happens once in a while, I'm not fantasizing about leaving my wife for her. I don't even know the mystery woman, after all. Why would I throw away everything I have, a relationship that has built up trust over years, for what all I know is just another pretty face? After all, there are a lot of pretty faces in the world, they're not rare.
I'm not saying all guys react this way, but I know I do. I've known guys who fell for women they saw walking down the street only once. Some even tried to seek them out. I tried to tell them that, not only do you risk coming across as a creepy stalker, but you might not even like the woman once she opens her mouth. I even had to throw out the "don't judge a book by its cover" cliché and the "you're only suffering from lust at first sight" maxim. This is somewhat presumptuous, but based on conversations I've had with men, most men look/stare at a woman because they find her sexually attractive. Many men then confuse that feeling for "love." Many women in turn often confuse that staring for "love." To me, the "male gaze" represents lust and not much more. Sometimes men act on that lust. Sometimes they don't.