I affirm your discomfort about drinking around people who are drug users. For one, they can just be finding a reason for them to indulge in their own vice when you are hanging out with them. They think you're both okay with it. You enjoy your own pastime, we leisurely get high on ours. No problem. Everybody happy! Secondly, in case you get drunk, and he's high as well, you will not be in total control of the situation. You may even have the drug for yourself! And the other way around! That can bring you to the next predicament of having both of you get hooked on each other's addiction.