Exactly what are you doing?
A well-established library is LAPACK. Originally a Fortran library, there are umpteen language bindings for it. I'm sure you can find lots of instructions for using it under msvc on the net.
Check out http://www.alglib.net/ as another option.
Lastly, another algo to look at is RANSAC.