Seriously though, if you want to do it just for fun/learning then go ahead, but i doubt very strongly that that the speed gain overweights anything else that higher-level languages offer.
Do you know assembly or are you just guessing? :mellow:
High-level languages use the lowest common denominator, which is just basic 32-bit operations. Modern processors offer a whole lot more, like complex operations and vector operations.
One example of a complex x86 instruction is 'bsr', which stands for 'bit scan reverse'. It returns the position of the first 1 bit, starting from the most significant bit. This is equivalent to an integer logarithm, and can be very useful in some projects. In a high-level language it takes many operations to compute this value. Vector operations use MMX and SSE. SSE can make many floating-point intensive calculations up to four times faster. Examples of real-life software that use (some) manually written assembly code are: audio and video codexes, physics engines, raytracers, drivers, low-level libraries, etc.
Obviously I agree that writing everything in assembly is useless, but using it in some hotspots can be very rewarding to take the application to a whole new performance level. And even if you have no direct use for it, it's still very valuable for debugging. Issues that can take days to debug at high level can become amost trivial when looking through the assembly code.