Friendship, Not "That Special Feeling"
There's much more to lasting romantic relationships than "that special feeling" or "magic" as you call it. That extra something is friendship, pure and simple. Not just tolerance or just putting up with someone, not "settling" for someone at the cost of some of your basic principles.
There is always compromise, yes, but it must be achieved in the context of mutual understanding, trust, and companionship that transcend the passion that often accompanies new relationships.
Growing Old Together
You didn't mention your age, but it doesn't sound like that's a major issue. So my bottom line is (and you mention this yourself) whether you can see yourself with this person at your side, intimately, a few or many years down the line. Such a partner has to be your friend, not just your lover.
If You Are Looking For A Life Partner...
There is, of course, nothing wrong with just dating, if you both understand that that's all it is, and neither of you is looking for anything more at the moment. But I fully understand the desire to find someone you feel you can spend the rest of your life with, and if that's your intent, you have to go with it.
If you're still just getting to know each other and you think there's a chance you'll discover much that is yet to be revealed, stick with it for a bit.
End It If It Ain't Happening
If you think you've got the hang of this guy, and it ain't happening, end it now, as amicably and respectfully as possible.
Good luck in love,