EINVAL with gpfs_fcntl

1
votes

I'm trying to use prefetching hints on an IO-heavy code. I set up the code according to my understanding of the man page of gpfs_fcntl(), but do get an EINVAL. I'm a bit lost now what I'm doing wrong - any hints are appreciated.

Mount: /dev/scratch16 on /bgscratch type gpfs (rw,dev=cadmos-gss.gss1a:scratch16,ldev=scratch16)

Error: Prefetch using gpfs_fcntl failed: Invalid argument (22), 32768b at 7713095680 from /bgscratch/foo.dat

File: -rw-rw-r-- 1 delalond bbp 14739308544 Jul 25 2012 /bgscratch/foo.dat

Source:

void BufferedFile::prefetch( const uint64_t offset, const uint64_t size )
{
   if( file_.fd == -1 )
       file_.fd = ::open( filename.c_str(), O_RDONLY );
   if( file_.fd == -1 )
   {
       LBWARN << "open() failed: " << lunchbox::sysError << std::endl;
       return;
   }
   struct
   {
       gpfsFcntlHeader_t hdr;
       gpfsAccessRange_t acc;
   } arg;
   arg.hdr.totalLength = sizeof(arg);
   arg.hdr.fcntlVersion = GPFS_FCNTL_CURRENT_VERSION;
   arg.hdr.fcntlReserved = 0;
   arg.acc.structLen = sizeof(arg.acc);
   arg.acc.structType = GPFS_ACCESS_RANGE;
   arg.acc.start = offset;
   arg.acc.length = size;
   arg.acc.isWrite = 0;
   if( gpfs_fcntl( file_.fd, &arg ) != 0 )
        LBWARN << "Prefetch using gpfs_fcntl failed: " << lunchbox::sysError
               << ", " << size << "b at " << offset << " from " << filename
               << std::endl;
}

Edit: I could reproduce the problem in a standalone application, and the error happens once start in greater than 4GB, even though it is a long long and a 64 bit system.

3
c++cfilesystems
you should add the language tag so more people will see it. wasn't sure if it was c or c++ so i didn't added any – Rémi
Did you initialize file_.fd to -1 ? – Adrian Panasiuk
Yes, fd is initialized to -1 in the constructor. – eile
My guess would be that you need to open the file handle using something other than the bog standard "open" ... just a guess though, sorry :( – Goz
Looking at gpfsAccessRange_t the elements start and length are of type offset_t is this equivalent to uint64_t on your platform? – Timothy Brown

3 Answers

0
votes

You are trying to read after the end of file.

Given the fileSize, try something like

arg.acc.length = fileSize > offset + size ? size : fileSize - offset;
0
votes

Your offset and size are uint64_t, unsigned integers. While the documentation for the struct gpfsAccessRange_t has the elements start and length as being offset_t, which could possibly be signed integers. Do you know how offset_t is defined?

I also note HDF5 uses off_t.

0
votes

I could not help but notice /bgscratch: if you are on a blue gene /Q system gpfs_fcntl() won't work unless you are running V1R2M1 or newer. Older driver versions will not relay the gpfs_fcntl system call to the i/o nodes.