Java - How to write a very large (20,000x20,000 px or larger) tif image

8
votes

I am working with extremely large tif images that I am composing into a large single image. I have a library that was created by a colleague of mine that generates the image pyramid and provides a very handy tool for visualizing the image pyramid. This visualizer is great for taking a peak at the large image and visually identifying points of interest, but the customers are more interested in image analysis on these large images.

Thus, it is necessary to export the very large image into a single file. I find this to be troublesome considering these images can be anywhere from 800 MB to multiple GBs in size. And just the task of loading this single image to memory is challenging, particularly when image analysis is being done.

I was wondering, if it were possible in java to write this large tiff image in a block or line-by-line fashion. Currently my application is running out of memory on small (8 GB RAM) machines.

The current method for composing these images is:

  1. Store the pixel values into a BufferedImage using a WritableRaster

    short[][]pixels = ...
BufferedImage image = new BufferedImage(width, height, type);
WritableRaster = image.getRaster();
for (int row = 0; row < height; row++)
{
    for (int col = 0; col < width; col++)
    {
       raster.setSample(col, row, 0, pixel[row][col]);
    }
}

  2. And then write the buffered image to disk. For this part I am using ImageJ to write the image as a tif. If there are better ways that support 16-bit grayscale tif images, then I will be happy to take a look

    // BufferedImage image; from above
...
ImagePlus img = new ImagePlus();
img.setImage(image);
FileSaver fs = new FileSaver(img);
fs.saveAsTiff(file.getAbsolutePath());

The problem with this method is it has too large of a memory footprint for an 8GB of RAM machine.

Ideally what i'd like is to just have a single short[][]pixels. This is mainly because I need to compute an average blending function, so some memory footprint will be there. Also in the future I will be adding a linear blend. The short[][]pixels should only take up ~765 MB of RAM for 20k x 20k pixel data, which I think is currently unavoidable, so for larger images for example 100k x 100k pixels, I would hope that the biologists would not want to export this image as it would take up 18GB of RAM.

Later I will modify the code to support exporting the extremely large images like the 100k x 100k. For now I am okay with assuming one piece of memory to store the initial pixel values.

So, what is a good method for writing portions of a tif image to disk so I can support writing out of core images such as the 100k x 100k images.

I did see this post: Write tiled output of TIFF, using ImageIO in Java

But it just discusses the TIFF 6.0 spec. But I will look into ImageOutputStreams. Tif is a beast though so I might just bite the bullet and encourage the biologists to only export the regions of interest.

EDIT: Found a viable solution:

WRITER: https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/out/TiffWriter.java

and

READER: https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/in/TiffReader.java

Main group page: https://github.com/openmicroscopy/bioformats

4
javaimagememorytiffimagej
Can you work with the image in sections, say 10k px squares then merge them all together? – ug_
I have no problem working 10k px squares, the problem is how to write them to a tif file. – Jameshobbs
You should perhaps divide the problem. 1. Write a large raw image data file. 2. Read that and save as tiff. Once that works, you can look into doing it in 1 pass. – hyde
Another suggestion, use memory mapped files of Java7 and 64 bit JVM, and save yourself a ton of headache. – hyde
For your blending filter, do you need the whole picture? Or just a portion? – Ortwin Angermeier

4 Answers

3
votes

With the SCIFIO project, we are generalizing the Bio-Formats image I/O framework to target scientific imaging in general, beyond just microscopy and the life sciences. The SCIFIO API is in beta now, and includes TIFF which can of course read and write in tiles. Feedback on API and bugs on the SCIFIO mailing list is always welcome!

2
votes

Ok I found a good solution to do this in Java.

Thanks to @bdares for pointing out BigTiff.

But packaged with FIJI, there is the bioformats group that implemented scifio.

They provide a number of supported reader/writers, one in which being a TiffReader/Writer

https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/in/TiffReader.java

and

https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/out/TiffWriter.java

I have edited my original posting. Thank you all for the comments, it helped me look in the right direction.

2
votes

Contrary to your comment, it's really not that hard to implement your own TIFF writer.

The spec is downloadable here. Specifically, pages 13-14 are (almost) all you need to understand what TIFF is, and how to write it.

Consider that according to the 6.0 spec (current as of September 2013), the maximum image size is 4GB. I'd suggest turning to BigTiff. The differences in specifications is listed on the second link.

0
votes

You have a double array of short. Instead of storing this in memory, you need to store it on disk and manipulate it on disk. BufferedImage is not going to help. You need a library (or to write a library) that allows the manipulation of an image on the disk without the need to fully load the image into memory.

You do not want to access each value individually. Instead, you will want to do something like this:

  1. read a block (maybe 4k, 8k, or 40k what ever makes sense).
  2. process the entire block.
  3. write the block to disk.
  4. goto step 1 until finished.