The operating system abstracts access to the underlying hardware, and makes it available to programmers via system calls. In Windows, these are done through the Windows API (which is usually further abstracted by libraries that make programming easier, like MFC, etc). In UNIX, this is often done through interrupts, which the system's C library makes a bit easier by following the POSIX api (often with a few system-dependent additions).
For example, on Linux, system calls are made through int 0x80, with several registers being loaded with the arguments to the function, and the C library makes this easier by allowing you to call, e.g. read, with the expected arguments ( int fd, void *buf, size_t count ). This gets translated into an interrupt call, which the kernel responds to.
These two manners of making requests against the operating system are incompatible, and thus you (generally) can't run a Windows executable on UNIX systems, and vice versa, without using some additional system that acts as a translation layer, like WINE, VMWare, etc. (though the way those two work is very different).
(Incidentally, a.out says nothing about the contents of the executable; it's the traditional filename given to executables compiled on UNIX systems, and is short for "assembler output". GCC allows cross-compiling, so you can even compile Win32-compatible .EXE files with it. You can use the -o flag to gcc to specify the output filename, which shows you that it has no bearing on the actual format of the outputted file.)