well, the trick is.. you actually need an alpha value to determine transparency (and 1 or 0 == transparent is up to your set up of the blending equation..)
you just want one color, so what you need to do is to set there alpha to transparent..
for every pixel
if pixel.rgb == colorkey.rgb
pixel.alpha = transparent
pixel.alpha = opaque
you can do that eighter before uploading the texture (and upload as GL_RGBA!! )
or, depending on hw, you can do it in the pixelshaders..
distance = fragment.rgb - colorkey.rgb
out.alpha = dot_product(distance,distance)
and then using for example alpha test to check if alpha is greater than some threeshold, say 0.01.. that means, distance to colorkey is far enough => color != colorkey => passes the alpha test
those are the two ways possible..