I would like to create my own font rendering engine with DirectX: this shouldn't be a problem, however I want to be able to render popular .ttf / .otf fonts. This proposes a problem as those files are binary.
Is there a way to find out how I should read those file formats?
The easiest way would be to integrate FreeType. It has all the necessary logic in place to load font formats and rasterize the glyphs to a texture.
If you're feeling brave, TTF is a specification created by Apple. They have documentation on the TTF file format here. The page looks a bit different last time I used it, but the information still seems intact. Before you implement this format, you should implement bezier paths in your engine first. This is how all vector based fonts are stored in the file. The TTF format is a bit tricky to work with, especially with compound glyphs. I suggest you use a tool like FontForge to make a very simple font (with one or two glyphs) and try to start with that. Hinting, antialiasing, cleartype text, etc. These are all custom algorithms to make the fonts look better. You will have to devise your own algorithm if you want to support similar functionality. Most of these algorithms are patented too, so it's not like you can just copy them and be legally risk free.
OTF is a specification created by Microsoft and supported by Adobe. You can find it's specification here. I haven't touched this format, but it feels similar in structure to TTF. The largest difference is that TTF uses quadratic bezier paths whereas OTF uses cubic bezier paths. This is not an issue if you implement a general bezier path solution that can support any curve order.
Yes ofcourse, bezier text is the way I wanted to go from the very beginning.
Thanks for the helpful links!