How can prot argument of mmap() syscall be 0x1007?

Question

While analyzing Metasploit's linux/x64/shell/reverse_tcp payload, I realized it was using the value 0x1007 as the prot argument of the mmap() syscall.

The mmap() man page says that the prot argument is either PROT_NONE or the bitwise OR of one or more of the following flags: PROT_EXEC, PROT_READ, PROT_WRITE PROT_NONE. According to the /usr/include/x86_64-linux-gnu/bits/mman.h file, the above mentioned flags have the values 0x4, 0x1, 0x2 and 0x0, respectively. How can this add up to 0x1007?

The syscall signature is:

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

The part of the code I was referring to is:

global _start
section .text
_start:

xor rdi,rdi                    
push byte +0x9                
pop rax                       
cdq                             
mov dh,0x10                    
mov rsi,rdx                     
xor r9,r9                       
push byte +0x22                
pop r10                        
mov dl,0x7                     
loadall286

The values you mention cover three bits: the kernel probably just ignores the values of any other bits. — gsg
The bit has no meaning for MMAP but if you review the code cdq sets RDX to zero. then mov dh,0x10 sets DH to 0x10 which makes RDX = 0x1000 (4096) which is used as the length parameter when RDX is then copied to RSI. The fact it is set in the flag argument is of no consequence. The value 7 (lowest 3 bits on) are the bits the kernel knows about, and the kernel just ignored bits it doesn't care about. It isn't the way I would have done it. — Michael Petch
When I compile and run this code it returns an address in RAX which is what I'd expect (this comment is in response to an edit that has been reverted) — Michael Petch
@MichaelPetch yes, I removed the edit after reading your comment, which clearly answered my question. I thought the edit was somewhat misleading. Once again, I cannot upvote your comment since I don't have enough rep. If you could provide an official answer, I would be happy to upvote. — adam

adam adam · Accepted Answer · 2017-05-09T05:00:19

As stated by @MichaelPetch, the most significant byte is ignored by the mmap() syscall. It only served to give a value to the length parameter by using the mov dh,0x10 and mov rsi,rdx instructions.

How can prot argument of mmap() syscall be 0x1007?

1 Answers