I am very concerned for your health and well-being in this relationship. While he might not be currently physically abusing you, he sounds like he is being emotionally abusive. Please think twice before you move in with this man. You mention that you are seeing a counselor -- can you see the counselor for some one-on-one appointments?
I think it is admirable that you want to provide a stable home for your child, but I do not think your son would learn anything healthy from being around this man. I am worried about his past of physical violence. I also think he is projecting his untrustworthiness by questioning you about your sexual involvements.
You sound like an intelligent person in a bad situation. Please write and let us know what happens.