Hi Flyakite,
Glad you're here and talking about this before it escalates further. Good for you reaching out.
I wish I could jump in a time machine and rewind to when I was beginning my affair. Then I could slap my old self silly and tell her to get into counseling ASAP. That's what I'm going to recommend to you here. There seems to be a longing in you for something more, and your EA AP is an attempt to fill that gap. If you don't know what that gap is, where it comes from and how it affects your actions, you're not going to be able to really fix it long term.
It's like alcoholics trying to come clean. They often try the "cold turkey" method, often with limited results and relapses. Once they get into AA, they're forced to dig deep, keep accountability to their community (AA group and sponsor) and take responsibility for their choices.
I'd really recommend taking up IC and taking it seriously.
It seems there's a lack of emotional awareness/ intelligence in your BH. He doesn't even seem to care that your affections have wandered. I mean, you did the absolute right thing and told him straight out that you're attached to another man, and all he says is that he's not worried/doesn't care! 
No offense, but if my H told me that, and he has from time to time after my A, I would feel like I wasn't important to him.
Do you feel important to your BH?
If it's a combination of an emotional desert in your M and a lack of healthy coping mechanisms that allowed you to escalate a friendship into an EA, there's a few things that need addressing. You can only fix yourself and your coping mechanisms, and fix your communication with your BH. You can't fix your BH and make him into a person who actually takes his WW seriously when she comes to him confessing an EA.
Wishing you the best on this journey. If you don't address these things now, you're leaving yourself open to another A with another person down the line.
