Following on with a previous project (Integer overflow when calculating the amount of memory to allocate) I have the program up and running and it is producing valid results on small data files creating small arrays (<2 GB of RAM). With larger project data files, however, the arrays are getting toward 10 GB. The read and processing of data progresses just fine.
But when it comes to writing the data back out to file, instead of writing to disk it fills the memory buffer, exhausts system memory (32 GB RAM), and then the machine locks up and restarts. This outcome is consistent between machines (laptops, desktops and VMs) and regardless of the storage device on which the process is completed (SSD, HDD, USB-HDD or network HDD).
All systems are ~12-18 months old, i7 processors, sufficient RAM and disk space, etc.
Google provided suggestions such as FLUSH, setting the environment variable GFORTRAN_UNBUFFERED_ALL to 1 (or 'y' or 'Y') and manual approaches such as closing the file and then opening it again with ACTION='append' to force the write.
Of these approaches, the close-n-open approach is the only one that clearly works, however it just results in memory filling more slowly than it otherwise would, and down goes the system again in the end.
Here is an example of the write without any interference:
program giant_array
use iso_fortran_env
implicit none
character(len=*), parameter :: csvfmt = '(*(f0.3,:,","))'
character(20) intval
character(200) line
integer(kind=int32) x, y, z, i, cnt
real(kind=real64), dimension(:,:,:,:), allocatable :: model
print *,
print *, "Allocating array and assigning values..."
print *,
call random_seed()
allocate(model(382,390,362,28))
call random_number(model)
print *, "Writing array to file..."
print *,
open(31, file="test.csv", status='replace', action='write')
cnt=0
! Write array to file:
do x = 1, 382
do y = 1, 390
do z = 1, 362
write(31, csvfmt) (model(x,y,z,i), i = 1, 28)
cnt=cnt+1
if((int(cnt/1000)*1000).eq.cnt) then
line = " Processing block grade "
write(intval,'(I12)') cnt
line = trim(line)//" "//trim(adjustl(intval))//"..."
write(*,'(A,A)', advance='no') achar(13), trim(line)
endif
enddo
enddo
enddo
close(31, status='keep')
end program
During execution you'll notice test.csv remains at size=0 until you kill the program.
Even with 'call SLEEP(1)' between the open and closing, the buffer fills more quickly than the disk write, and before the job is done the system crashes. It would also take forever to complete.
I've found reference to using fsync() to remedy this problem but can't get the code to compile (I think I'm stuffing the command line args). Code is as follows, from gcc.gnu.org:
! Declare the interface for POSIX fsync function
interface
function fsync (fd) bind(c,name="fsync")
use iso_c_binding, only: c_int
integer(c_int), value :: fd
integer(c_int) :: fsync
end function fsync
end interface
! Variable declaration
integer :: ret
! Opening unit 10
open (10,file="foo")
! ...
! Perform I/O on unit 10
! ...
! Flush and sync
flush(10)
ret = fsync(fnum(10))
! Handle possible error
if (ret /= 0) stop "Error calling FSYNC"
While others have come across this issue I can't find a solution anywhere. Comments and blog posts suggest that even the fsync() approach doesn't always do the trick.
The result is a system crash and self-restart every time.
I'm guessing there must be a way to write large files to disk in one go without excessive system specifications.
Many thanks.
Updated
Code updated as follows to test the C++ _commit statement to force from buffer to disk. Work about as well as the close-then-reopen method - still kills the machine. It's entirely possible there is still something wrong with my implementation...
program giant_array
use iso_fortran_env
use iso_c_binding
implicit none
! Declare the interface for WIN32 _commit function
interface
function commit (fd) bind(c,name="_commit")
use iso_c_binding, only: c_int
integer(c_int), value :: fd
integer(c_int) :: commit
end function commit
end interface
character(len=*), parameter :: csvfmt = '(*(f0.3,:,","))'
character(20) intval
character(200) line
integer(kind=int32) error
integer(kind=int32) var, x, y, z, i, cnt
real(kind=real64), dimension(:,:,:,:), allocatable :: model
print *,
print *, "Allocating array and assigning values..."
print *,
call random_seed()
allocate(model(382,390,362,28))
call random_number(model)
print *, "Writing array to file..."
print *,
open(31, file="test.csv", status='replace', action='write')
cnt=0
! Write array to file:
do x = 1, 382
do y = 1, 390
do z = 1, 362
write(31, csvfmt) model(x,y,z,:)
cnt=cnt+1
if((int(cnt/1000)*1000).eq.cnt) then
line = " Processing block grade "
write(intval,'(I12)') cnt
line = trim(line)//" "//trim(adjustl(intval))//"..."
write(*,'(A,A)', advance='no') achar(13), trim(line)
flush(31)
error=commit(fnum(31))
endif
enddo
enddo
enddo
close(31, status='keep')
end program
