Dear Not ready to give up,
Personal Soapbox: Stop Having Sex
Why are you having sex with a man who won't commit to you?
I think that this is your fundamental problem. You and your boyfriend are not equally invested in this relationship. My advice, stop having sex until he is ready to marry you.
However, I will attempt to answer the question that you asked. I can not explain why this man is so infatuated with this other woman. However, speaking from the viewpoint of someone who almost married a man 11 years older than me, I can tell you that a relationship between the two of them is not out of the realm of possibility.
It's More Than an Infatuation
The fact that he has been interested in her for four years implies that it is more than an infatuation. I wonder how well they know each other. An infatuation characteristically is an all-consuming attraction to a person that you know little to nothing about. Infatuations tend to end when you get to know the person. If he has known her for 4 years it is probably not an infatuation but rather a mutual flirtation.
It is encouraging that you and he have been able to discuss his feelings toward her. That is important for a secure relationship. (I choose to believe that he has not acted on his feelings however, his behavior with you does not show him to be the most sexually discriminating man.)
If you really think that there is marriage potential in this relationship, let him go. If he comes back after dallying with her, he will have assured himself that you really are the best woman for him. IF he does not come back, of course you will be sad. But do you really want to marry a man who will be forever thinking of "what might have been" and pining after someone else? Until he is ready to give himself fully to you, do not marry him and don't have sex with him.