"shmget" is a Linux-Specific method of allocating shared memory implemented in the Linux kernel.
"shm_open" is a library function that emulates shared memory by mapping files using mmap. Because the files are mapped using the "shared" flag the memory is shared between the processes.
In Linux 1.x the "/dev/shm" directory (containing the files) was just a regular directory so the shared memory using "shm_open" really were disk files. In Linux 3.x "/dev/shm" is a special directory to avoid that shared memory really has to be written to disk.
I think both of the two methods can be replaced by the other one. It is only because of historical reasons why there are two different methods to create shared memory.