The main idea, is you have a separate neural net for each moving object the robot interacts with, each neural net is simply recording some relative vectors to its environment and other objects and the robot, and the derivatives of the objects activity.
This is the robot, trying to build a store that can plausibly play back the motions of the other moving objects, yet retaining plastic freedom to try anything, its a "plastic record".
Then, if everything plays back close to what actually happens (you only have vectors to playback anything successful, the rest of the information is unknown (he cant read minds - unless you give him brain wave receptors hehe)) then you should able to be able to sim mutations of his multidimensional calculus function, which is measurements to motor torque.
Its just a matter of how many video cards you want to devote to sim frames, more the better.
I think it has potential in the robopet industry, but unfortunately i think it may be a little too powerful to be safe to give out to people, because you could hack him into a death robot really easy. its not really safe.