Interesting problem! Assuming you have pairs of corresponding points (e.g. you know that points x1, x2, x3, etc. before the rotation map to y1, y2, y3, etc. after the rotation), you can find the center of rotation using geometry. For each pair of corresponding points x and y, find their midpoint; then draw an infinite line perpendicular to the line between x and y, through the midpoint. The center of rotation must lie somewhere on this line. If you have multiple pairs of points, the lines constructed from all of them should converge (at least roughly) at a single point. Intuitively, the more pairs of points you use, the better an estimate you'd be able to get for the center.
Now you just need to figure out how to get the "average" convergence point of a set of lines. You could maybe look at all the intersection points of all pairs of lines and find the centroid of those, but that sounds expensive since it's O(n\\^2) in the number of lines. There's probably a better way, but I can't think of it at the moment.
Once you have the center, you can then find the angle of rotation by again considering each pair of points and calculating the angle between them relative to the center. Then averaging all the angles should give the best estimate for the true angle of rotation.