programming database in memory accessible to other programs

0
votes

I am currently in the process of writing a network data usage monitor in C (program A) on a 64 bit linux platform. As a packet sniffer there isn't much time for writing data to a file or db without risking the loss of packets. I could use another thread for such purposes, but I was thinking that a much cleaner solution (if it exists) would be to access this data stored in memory from another C program or ideally a python cgi script (program B). That way the data would be available on demand. The data in memory would be accessed read only from Program B. Is this possible? If so how? Thanks.

Updates:

I see that this is possible with mmap() to store the data in memory, and shmget() to retreive it. I've heard some say that shmget is old though. What other options are available?

The 2D arrays to be passed could be as big as 5000x4 int or 5000x15 char.

posix_ipc Sounds promising as a way of accessing shared memory from python (program B). Does anyone know if this will work with shared memory created in C (program A)?

I downloaded posix_ipc and it has some really cool demos. The first one demonstrates two processes talking to each other using shared memory. The two processes can be any combination of C and python and the source for each of the four are provided. It looks like the most efficient way to handle what I am trying to do, but I haven't had the time to play with it yet. I will report back when I do.

che's suggestion below sounds like it will also work, and I will keep that as my plan B.

Thanks to everyone for the help!

4
cshared-memorymmap
Is there some reason that “old” is a bad thing, here? … i.e. it's widely-available, well-documented, and well-tested … – BRPocock
@BRPocock, No old isn't bad. It was just describe as "crufty" and "some might say depricated", which didn't sound like a great place to start. It also sounds like it is C only. Ideally I'd like to be able to access the data read only from a python program B. – nomadicME
I don't know why it would be seen as “deprecated,” but mmap is perhaps a more general-purpose facility. But, yes, bit-twiddling from Python would be difficult. If both processes are in a language with binary-level access like C, though, it's pretty common to use shared memory in this way. Something like swig might help bind to Python. – BRPocock

4 Answers

1
votes

Shared memory sounds like the best way to do this between two C programs, especially if the second program will just read the stats written by the first one.

It might be a bit complicated to get to this memory from Python, as it's a bit low-level for this language. The easiest way might be to have a C program that would access the shared memory and print the info in plain text, and call that from your CGI script.

1
votes

Use SQLite, In-Memory mode. This will give you all the benefits that a database offers, without having a large footprint.

http://www.sqlite.org/inmemorydb.html

1
votes

In regards to posix_ipc: Yes, shared memory regions are “just memory,” they're completely agnostic to the language from which you access them.

However, the way in which you arrange the use of that memory might be limited to the “least common denominator.” An array of fixed-length structures containing fixed-length data fields will be simple and easy, as would certain arrangements of text content; something like “objects with variable fields and garbage-collection” would be difficult to bind between two languages.

For the record, mmap is one way to access shared/shareable memory; the shmget family of commands (shmget and shmat, shmctl and so forth) are another. Mixing the two would be difficult and not particularly advantageous. (Note that the name shmget refers to getting a shared memory region, and doesn't imply read-only access to that memory, necessarily.)

0
votes

Try memcached if you need simple key/value in-memory storage http://memcached.org/

With a plus bonus to learn a technology highly used in web app development as well