When you call a function f that has never been defined, an implicit declaration like this takes place:
int f();
Note that you can still pass arguments to f, since it was not declared int f(void); (this is for backwards compatibility with K&R C).
Thus, the compiler doesn't "know" that malloc receives a size argument, in fact you can pass it what you want, it doesn't care.
So, the mere fact that the code works is pure luck. In the case of malloc, if the code works, it just means that the size of an integer is the same size as a pointer - no more, no less - so, you can still call malloc and assign its result to a pointer, since no bits were trimmed / are missing.
The true function is found in the linking stage, after compilation takes place. By this time, your code was already compiled with a wrong function prototype. Of course, if the linker can't find the function anywhere in its path, an error is reported and everything is aborted.
For the case of malloc, and other standard library functions, there may be builtin functions to increase performance. There's even an option for gcc to disable builtin functions (-fno-builtin or -fno-builtin-function). From gcc manpage:
GCC normally generates special code to handle certain built-in
functions more efficiently; for instance, calls to "alloca" may become
single instructions that adjust the stack directly, and calls to
"memcpy" may become inline copy loops. The resulting code is often
both smaller and faster, but since the function calls no longer appear
as such, you cannot set a breakpoint on those calls, nor can you
change the behavior of the functions by linking with a different
library. In addition, when a function is recognized as a built-in
function, GCC may use information about that function to warn about
problems with calls to that function, or to generate more efficient
code, even if the resulting code still contains calls to that
function.
So, in the particular case of malloc, this is how the compiler "knows" its usual signature. Try to compile this code with gcc:
int main(void) {
char *a = malloc(12, 13, 14, 15);
return 0;
}
You will see it will abort with a compilation error:
test.c:3: error: too many arguments to function `malloc'
If you use the option -fnobuiltin, the error goes away and the warning is different:
test.c:3: warning: implicit declaration of function `malloc'
This is the same warning you get every time you use a regular function that was not previously defined, because now the compiler is ignoring what he "knows" about these functions. This example uses gcc, but other compilers will have similar behavior.