To do 3d really well, you need a good way to get stuff out of the 3d application into the program quick easy and fast, including models textures and animation... this is what an engine essentially is, it talks to the 3d application and makes it easy to get stuff into the game.
Without an engine, youll have to code a basic "entry point" for all the game things, and you can actually work without an engine if you want to... its what I actually do myself.
For example, what im coding now is a thing that converts models from zbrush and 3dcoat to my game format, and I actually code the animator myself.
With that handy "asset entry point" which takes about a couple of weeks to write, I can then get on with the game without a bought engine and its just as easy.
You can actually write it separately from your main game program, make it form a "dat" file or something the game reads when it boots up.