why is that
Because it's easier and you have time-to-market pressures
will new games be able to fix this?
Usually, only if they feel there's a reason to do this.
Can't you have an area pathing mob?
Yes. Pathing is well-understood and lots of canned algorithms exist out there. A*, D*, Floyd-Warshall and other graph algos, etc.
They still have to be incorporated into a game by the designers, though.