Turns out that I can't have stereo mix unmuted if I just want to use my mic. More specifically, I can, but If any other sound comes through the computer (person's voice, music, video game sounds) while I'm trying to use the mic they get a raucously loud noise for however long my mic is on and sound is coming through.
So now that I know the cause, how do I fix it?
I want to be able to use vent and play sc2 and livestream it with procaster so that you can hear game sounds plus my mic and allies mic. Which is currently impossible for me.
It would also be a hell of a lot easier if I didn't have to toggle mute back and forth between the 2 settings. And since it was such a bitch to find the settings here is how I got there in Windows XP if anyone else wants to see.
PS- I can't move the sliders under recording controls, only the mute checkboxes.
Stereo mix seems to be really loud bass and you can't hear any highs. Check some of these short videos from today, to see what I mean: http://www.livestream.com/sporcharliemurphy
edit- Also make sure the stereo mix/Mic is checked under the properties (#5) so that the option for adjusting the volume and muting it is shown in the next window (#6).