Avoid fatal error memory exhausted before it happens

0
votes

I'm trying to avoid a memory exhausted fatal error by figuring out if it will happen before it actually happens. Despite my PHP installation having a mem limit of 128M the allowed size is 134217728 bytes; I'm guessing PHP gives some headroom here which accounts for the extra bytes. In any case, my actual file size that I am trying to read is 134171737 which is less than 134217728. However PHP is saying it tried to allocate 134179929.

These numbers seem arbitrary. How can I check if reading a file to a variable will exhaust memory if (by comparing i.e. mem_size > file_size) these numbers isn't actually accurate?

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 134179929 bytes) in C:\xampp\htdocs\backup\vendor\cake\File.php on line 159

Note: I'm aware I can do a readfile. I simply am curious how I can avoid such an error in this case.

1
phperror-handlingout-of-memory
not sure what your asking - user6763587
If I read a file to a variable and echo that variable and if the file size is greater than the mem limit I can check for that before I echo the variable thus avoiding mem exhausted error. But since the file size doesn't seem to determine the memory allocated im stuck. - Alex
You already seem to know the answer - use filesize() to check the file size first, make sure you leave appropriate padding for memory the runtime/other parts of your code will need... - Dan Field
Exactly my point: how can I determine the appropriate padding? Hence why I stated these numbers seem arbitrary. Because I've read that file_get_contents uses some sort of output buffer which I think is accounting for the extra bytes, but I can't seem to find any documentation on how to determine how much these extra bytes will be for a file n size large. - Alex
do you have to read the whole file in to memory in the first place? - user6763587

1 Answers

0
votes

You should never try to allocate the total available memory to your runtime - this becomes more and more true the higher level your language is.

Even with very low level languages (which PHP is not, of course), you'll need not just the memory for the bytes that you're allocating (which will be going into heap space), but the memory for the instructions and call stack.

As you get into higher level and interpreted languages, that overhead only increases - the interpreter is going to need its own amount of memory, and there may not be any good or reliable way (whether across versions or even across configurations of the same version or executions within the same configuration) to determine exactly how much "padding" that environment/runtime will need.

In your case, you already suspect that you may come close - you should pick a "safe" amount of buffer space to leave for the runtime, for instance, if you know you're going to try to allocate more than 90% of the available memory to the runtime you shouldn't (and for a language like PHP, where you're very likely running in an server environment where other consumers are competing for the same resources, that threshold should likely be much lower).

In general, it's good practice to avoid consuming more memory than necessary, especially in interpreted and web-based environments. While memory trade-offs might be worth it to shorten execution time (particularly when loading more data in memory will save you time on lots of repeated calculations that would take longer to repeatedly allocate memory for and then deallocate memory for only to reallocate it), the case you're talking about here doesn't really make sense for that. This is the reason that you'd prefer a streaming model, where you only allocate as little memory as necessary to read the part of the file you can reasonably serve (as opposed to a situation where you have to repeatedly process different parts of that file, and then reprocess certain parts of them based on what you just processed - e.g. a routine that has to backtrack frequently in the file to know what to do next).