This feels like the dumbest problem, but hopefully someone can help. Sorry the post is so long, but I wanted to provide sufficient detail so that folks don't suggest things I've already tried.
I have isolated a problem with a C program that I wrote which is supposed to ship on Mac, Linux, and Windows. The program wasn't working on Windows but worked correctly on Mac and Linux and used to work on Windows before a recent change.
The proximate cause of the failure has to do with reading a file into a memory block - so I isolated just that code into a self contained program and tested it using some sample data which reliably fails on Windows and works correctly on Mac and Linux.
It should be noted that on Windows I am using Visual Studio 2019 (Version 16.5.5). I am testing it using Windows 10 Enterprise on a 64 bit Dell laptop. On Linux I am using gcc (I'm testing it with Ubuntu 20.04). On mac, I compiled it using clang. The program is intended to be portable (at least between these three platforms).
The basic strategy for loading the file is to open it using fopen(), then measure the file by using fseek() to move the file mark to the end of the file, use ftell() to get the location within the file, then fseek() back to the beginning, then use ftell() to get the location at the beginning of the file (which is usually zero in practice but that isn't guaranteed), and then I subtract the start location from the end location to determine the file size. This "measure the file" code in practice seems to work reliably to measure the file on the three platforms that I care about.
Then, I call malloc() to allocate a block of memory large enough to hold the file. This always works fine. The files I'm using are around 200K and they are binary files - but for isolation purposes I was able to get it to reliably fail with a 271 byte file. The original code just used a for loop from 0 to the size of the file and called getc(fileptr) repeatedly and then assigned each byte into the memory buffer. Then it closed the file. This code worked fine on Mac and Linux but didn't work on Windows. What I observed was that I would get the first part of the file - in some cases most of the file - and then I would start getting "ff" read back from the getc(fileptr) call which would fill up the rest of the memory - obviously wrong.
So I looked into the difference between getc() and fgetc() and apparently getc() can sometimes be a macro that evaluates things more than once. This didn't seem like an obvious culprit, but I changed to fgetc() anyway and it didn't change anything. I also changed the malloc() call to calloc() so that I could start off with all zeros and make it easier to see the file being read using the debugger (i.e. viewing the memory buffer and see it being written to).
I used a Hex editor to create a file that has the following data in it so that I could use it for more systematic testing. The file contains 271 bytes. The first 256 bytes are all of the possible byte values: 00 01 02 03 ... fc fd fe ff. The last sixteen bytes are 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f. That way I can see if the problem is caused by trying to read some specific byte value and that I can have it continue past all possible byte values and do another 16 bytes in the same pattern just for good measure and I can easily see if the last byte is 0f.
The next thing I did was I used a preprocessor #if 0/#if 1 toggle to switch between a version of the file read that uses fgetc() and a version that uses fread(). Here is where I got the first interesting clue about what might be happening.
On Mac/Linux both versions of this program print the values I expect properly. On Windows, though, the fread() version reads the first 26 bytes and after that all the bytes are 00 (because calloc set the values to 00 for the whole block and fread() is only setting the first 26 bytes). The getc() version of the file read reads the first 26 bytes correctly and then all the subsequent bytes are ff.
The first 26 bytes are: 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0a 0x0b 0x0c 0x0d 0x0e 0x0f 0x10 0x11 0x12 0x13 0x14 0x15 0x16 0x17 0x18 0x19.
The full (correct) output of the program on Mac is:
szFile: 271 Read 271 bytes load_ggx_file: 0 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0a 0x0b 0x0c 0x0d 0x0e 0x0f 0x10 0x11 0x12 0x13 0x14 0x15 0x16 0x17 0x18 0x19 0x1a 0x1b 0x1c 0x1d 0x1e 0x1f 0x20 0x21 0x22 0x23 0x24 0x25 0x26 0x27 0x28 0x29 0x2a 0x2b 0x2c 0x2d 0x2e 0x2f 0x30 0x31 0x32 0x33 0x34 0x35 0x36 0x37 0x38 0x39 0x3a 0x3b 0x3c 0x3d 0x3e 0x3f 0x40 0x41 0x42 0x43 0x44 0x45 0x46 0x47 0x48 0x49 0x4a 0x4b 0x4c 0x4d 0x4e 0x4f 0x50 0x51 0x52 0x53 0x54 0x55 0x56 0x57 0x58 0x59 0x5a 0x5b 0x5c 0x5d 0x5e 0x5f 0x60 0x61 0x62 0x63 0x64 0x65 0x66 0x67 0x68 0x69 0x6a 0x6b 0x6c 0x6d 0x6f 0x70 0x71 0x72 0x73 0x74 0x75 0x76 0x77 0x78 0x79 0x7a 0x7b 0x7c 0x7d 0x7e 0x7f 0x80 0x81 0x82 0x83 0x84 0x85 0x86 0x87 0x88 0x89 0x8a 0x8b 0x8c 0x8d 0x8e 0x8f 0x90 0x91 0x92 0x93 0x94 0x95 0x96 0x97 0x98 0x99 0x9a 0x9b 0x9c 0x9d 0x9e 0x9f 0xa0 0xa1 0xa2 0xa3 0xa4 0xa5 0xa6 0xa7 0xa8 0xa9 0xaa 0xab 0xac 0xad 0xae 0xaf 0xb0 0xb1 0xb2 0xb3 0xb4 0xb5 0xb6 0xb7 0xb8 0xb9 0xba 0xbb 0xbc 0xbd 0xbe 0xbf 0xc0 0xc1 0xc2 0xc3 0xc4 0xc5 0xc6 0xc7 0xc8 0xc9 0xca 0xcb 0xcc 0xcd 0xce 0xcf 0xd0 0xd1 0xd2 0xd3 0xd4 0xd5 0xd6 0xd7 0xd8 0xd9 0xda 0xdb 0xdc 0xdd 0xde 0xdf 0xe0 0xe1 0xe2 0xe3 0xe4 0xe5 0xe6 0xe7 0xe8 0xe9 0xea 0xeb 0xec 0xed 0xee 0xef 0xf0 0xf1 0xf2 0xf3 0xf4 0xf5 0xf6 0xf7 0xf8 0xf9 0xfa 0xfb 0xfc 0xfd 0xfe 0xff 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0a 0x0b 0x0c 0x0d 0x0e 0x0f
On Window using the fread() version it prints:
szFile: 271 ferror: 0 feof: 1 Read 26 bytes load_ggx_file: 0 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0a 0x0b 0x0c 0x0d 0x0e 0x0f 0x10 0x11 0x12 0x13 0x14 0x15 0x16 0x17 0x18 0x19 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
On Windows when fread returns a value below the value you are asking for (i.e. the third argument in my case), you are supposed to check ferror() and feof(). What I found was that ferror() returned 0 and feof() returned 1. So the problem seems to be that Windows thinks it has reached the end of the file. The question is why does it think that and what is a reasonable alternative to use given my constraints? (i.e. I want to write portable C code using just the standard library - not a bunch of platform specific code).
I did check to see if the problem was simply due to the 0x20 character. I tried inserting a 0x20 after the 0x01 in my test file using a HEX editor and what happened was that it read and printed a representation of that character just file and still stopped after the 0x19 character. There does not appear to be any specific character that always causes it to choke.
Here is the complete test program:
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
typedef struct {
long long szFile;
unsigned char* ggx_file;
} ggx_t;
int load_ggx_file(const char* ggx_file_path, ggx_t* outGGX)
{
int rc;
FILE* ggx_file;
unsigned char c;
long long szFile;
long fend_offset;
long fstart_offset;
ggx_file = fopen(ggx_file_path, "r");
if (!ggx_file || NULL == outGGX) {
return -1;
}
rc = fseek(ggx_file, 0, SEEK_END);
assert(0 == rc);
fend_offset = ftell(ggx_file);
rc = fseek(ggx_file, 0, SEEK_SET);
assert(0 == rc);
fstart_offset = ftell(ggx_file);
szFile = fend_offset - fstart_offset;
printf("szFile: %lld\r\n", szFile);
outGGX->szFile = szFile;
outGGX->ggx_file = (unsigned char*)calloc(szFile, 1);
int i = 0;
#if 0
for (; i < szFile; ++i) {
c = fgetc(ggx_file);
outGGX->ggx_file[i] = c;
}
#else
i = fread(outGGX->ggx_file, 1, szFile, ggx_file);
if (i < szFile) {
int rc2;
rc2 = ferror(ggx_file);
printf("ferror: %d\r\n", rc2);
rc2 = feof(ggx_file);
printf("feof: %d\r\n", rc2);
}
#endif
printf("Read %d bytes\r\n", i);
fclose(ggx_file);
return 0;
}
int main(int argc, const char* argv[]) {
const char * ggx_file_path = argv[argc - 1];
ggx_t ggx_file;
int rc = load_ggx_file(ggx_file_path, &ggx_file);
printf("load_ggx_file: %d\r\n", rc);
for (int i = 0; i < ggx_file.szFile; ++i) {
printf("0x%02x ", ggx_file.ggx_file[i]);
if (0 == ((i+1) % 20)) {
printf("\r\n");
}
}
printf("\r\n");
return 0;
}
fopen(..., "rb")– pmg