Resize huge jpeg using no memory

1
votes

I need to resize huge (up to 30000x30000) JPEG files using no RAM, speed doesn't matter, is there any way to do so? I tried different libraries (nativejpg and others) but they use all free RAM and and crash with errors like "Out of memory" or "Not enough storage is available to process this command". I even tried command line utility imagemagick, but it also uses gigabytes of memory.

3
delphiresizejpeg
@KenWhite No. That's not right. You can process the image in parts. You don't have to load the entire image into memory. – David Heffernan
@KenWhite You know what I say is true. Let's focus on the concepts. – David Heffernan
@Alex The problem with your question is that you seem to be asking for a library recommendation which is off topic. You ask if it is possible. Yes it is possible. Either you need to find a library, or do the image processing yourself. – David Heffernan
@KenWhite Also, ImageMagick can handle this: imagemagick.org/script/architecture.php – David Heffernan
@DavidHeffernan When resizing the segments you end up with a "border" around each segment. This happens because the resize doesn't take into account the neighboring pixels because they are on a different image. You would have to get around this by taking a large segment that what you need and then trim it back after the resize. These larger segments would overlap. – Graymatter

3 Answers

3
votes

I would suggest you have a look at vips. It is documented here.

I can create a 10000x10000 image of noise like this with ImageMagick

convert -size 10000x10000! xc:gray50 +noise poisson image.jpg

and check it is the correct size like this:

identify image.jpg
image.jpg JPEG 10000x10000 10000x10000+0+0 8-bit sRGB 154.9MB 0.000u

I can now use vips to resize the 10000x10000 image down to 2500x2500 like this

time vipsthumbnail image.jpg -s 2500 -o small.jpg --vips-leak
memory: high-water mark 20.48 MB

real    0m1.974s
user    0m2.158s
sys     0m0.096s

Note the memory usage peaked at just 20MB

Check the result like this with ImageMagick

identify result.jpg
result.jpg JPEG 2500x2500 2500x2500+0+0 8-bit sRGB 1.33MB 0.000u 0:00.000

Have a look at the Technical Note too, regarding performance and memory usage - here.

You can also call it from C as well as the command line.

2
votes

You can do this with imagemagick if you turn on libjpeg shrink-on-load. Try:

$ identify big.jpg 
big.jpg JPEG 30000x30000 30000x30000+0+0 8-bit sRGB 128MB 0.000u 0:00.000
$ time convert -define jpeg:size=2500x2500 big.jpg -resize 2500x2500 small.jpg
real    0m3.169s
user    0m2.999s
sys 0m0.159s
peak mem: 170MB

How this works: libjpeg has a great shrink-on-load feature. When you open an image, you can ask the library to downsample by x2, x4 or x8 during the loading process -- the library then just decodes part of each DCT block.

However, this feature must be enabled when the image is opened, you can't set it later. So convert needs a hint that when it opens big.jpg, it only needs to get an image of at least size 2500x2500 (your target size). Now all -resize is doing is shrinking a 3800x3800 pixel image down to 2500x2500, a pretty easy operation. You'll only need 1/64th of the CPU and memory.

As @mark-setchell said above, vipsthumbnail is even faster:

$ time vipsthumbnail big.jpg -s 2500 -o small.jpg --vips-leak
memory: high-water mark 29.93 MB
real    0m2.362s
user    0m2.873s
sys 0m0.082s

Though the speedup is not very dramatic, since both systems are really just resizing 3800 -> 2500.

If you try tif instead, you do see a large difference, since there's no shrink-on-load trick you can use:

$ identify 360mp.tif 
360mp.tif TIFF 18000x18000 18000x18000+0+0 8-bit sRGB 972MB 0.000u 0:00.000
$ time convert 360mp.tif -resize 2500 x.tif
peak mem: 2.8 GB
real    0m8.397s
user    0m25.508s
sys 0m1.648s
$ time vipsthumbnail 360mp.tif -o x.tif -s 2500 --vips-leak
memory: high-water mark 122.08 MB
real    0m2.583s
user    0m9.012s
sys 0m0.308s

Now vipsthumbnail is about 4x faster and needs only 1/20th of the memory.

1
votes

With built in Delphi Jpeg support you can load large jpeg image resampled to smaller size while loading without excessive usage of RAM.

Jpeg image Scale property can have following values jsFullSize, jsHalf, jsQuarter, jsEighth

procedure ScaleJpg(const Source, Dest: string);
var
  SourceImg, DestImg: TJPEGImage;
  Bmp: TBitmap;
begin
  Bmp := TBitmap.Create;
  try
    SourceImg := TJPEGImage.Create;
    try
      SourceImg.Scale := jsEighth;
      SourceImg.LoadFromFile(Source);
      Bmp.Width := SourceImg.Width;
      Bmp.Height := SourceImg.Height;
      Bmp.Canvas.Draw(0, 0, SourceImg);
    finally
      SourceImg.Free;
    end;
    DestImg := TJPEGImage.Create;
    try
      DestImg := TJPEGImage.Create;
      DestImg.Assign(Bmp);
      DestImg.SaveToFile(Dest);
    finally
      DestImg.Free;
    end;
  finally
    Bmp.Free;
  end;
end;

Once you have roughly rescaled image to size that can be comfortably processed in memory you can apply ordinary scaling algorithms to get the actual image size you want.