I don't know if this will work, but it would be worth a try if you are up for it.
Treat each pixel as a 3d vector and calculate the angle between neighbours (dot product)
If this value is greater than some value, don't blend.
I think that should keep the detail you want and give some smoothing where you do.
Interesting! It seems to work quite well actually! Sometime, things are simpler than they seem. Have you tried it yourself?
It does not seem to work well on dead pixels. But that's probably because I'm not sampling properly. Should I use a Gaussian way to sample the pixels? How would you do it?