Sometimes I see games using a custom window,
like this: http://img.informer.com/screenshots/3518/3518273\_1.jpg
I'm a total noob with windows programming, so could anyone explain how I can do this?
What you are seeing in that screenshot is a window without the title bar and border. Any Win32 resource editor can easily configure that out via the properties menu. The second part involves using a layered window style. The idea is you set the window to transparent and blend on top of that a PNG image (or some image with an alpha channel).
You can supply images to Win32 button controls and you can listen for mouse events to change the image accordingly. For labels and edit boxes, you can customize their font styles. To do more advanced editing, you will have to learn about the WM_PAINT message and override the standard paint routines with your own.
Once you become familiar with the Win32 message loop, have a look at this page. It describes all the Win32 controls that are available and what their control and notification messages are. You can intercept any of the notification message and perform some sort of logic. For example, you can listen for the BCN_HOTITEMCHANGE to determine if the mouse is hovering or leaving the button area and change the button's image accordingly.
Is it possible to just get the transparent window, and then write my own window logic + use DirectX to paint on that 'transparent' window?
Because I really dislike windows programming
I don't think you can use layers in that way. Not only do 3D APIs render to a rectangular region, they share control with the windows drawing context. If you attempt to mix both 3D rendering and Windows drawing, you're likely to run into frame buffer issues (flickering comes to mind). If you really want to work in that environment, you should just go fullscreen and display that window as a popup. You could alternatively look into WPF (C#) to render your menu screen. It's very easy to do UI stuff with it. When you're ready, you can just launch your unmanaged code via p/invoke.
I think you could try SetWindowRgn with GPU rendering.
I wrote an article about Window Skinning some years ago, you may read it if you're still interested (it's a C++ tutorial with source-code):
Long live to Flipcode!