You're only 20, my God, what's the rush? I think that the conditions of your starting with this guy would push anyone to the brink you both seem to be on. You're flying into uncharacteristic rages, he's pulling back. Well, look at what's happened:
You both gave up real life choices for each other. That kind of weight will bring down any relationship. You two need to slow down and let the dust settle. Your passionate breakup with your ex indicates unresolved feelings, doesn't it? What was his breakup like? Unresolved feelings are totally natural in this circumstance.
No Relationship Is Worth That Kind of Sacrifice
Because you gave up a serious relationship and plans to go to grad school (and finish college?) this had better be worth it. But the truth is, no relationship is worth that kind of sacrifice. You have to take a step back and evaluate what you need for you for the long haul. Real love will endure the slower pace while you figure out what you're doing.
You Need To Figure Out Your Own Wants and Needs
You sound desperate ... but, it's only you, ultimately. What YOU want, what YOU need, what YOU hope for are the only things that really matter when making these monumental decisions. Your love for this man shouldn't be founded on some sense of obligation because of what you both sacrificed. Take it slower, think it through. Finish college. Don't move in with him, yet. Sit with the new status of you being on your own, for you, for a change.