Make sure the dwCallback parameter is the address of a function with the proper prototype. The callback function must also be declared with the CALLBACK calling convention.
Also try adding WAVE_ALLOWSYNC to the flags, just to see if this makes it work.
Ok, I tried that, but I'm getting the same return code. The thing is, I don't really think the flag is where the problem is. I can call waveOutOpen() with a format(3rd parameter) of PCM, and everything would be fine. But if I change the format to GSM610 and leave all the other parameters the same, I get an invalid flag return code. I don't think its the callback function either, since it works if I set the format to PCM.