Easiest method:
Place ItemsControl inside list box item. Set ItemsControl's width and height as your layout algorithm tells you to. Set ItemsPanel to be a grid. In the data template, place an Image, set margins as your layout algorithm tells you to.
Images wont be virtualized, listbox items (containing the whole ItemsControl with images) will.
Most correct method (only required if you have many images and/or large images so that single item exceeds 2048px):
Inherit from e.g. VirtualizingStackPanel, override MeasureOverride and ArrangeOverride to implement your custom layout, then spend ~2-10 days fixing virtualization-related issues in your code. This way virtualizes everything, however the time is estimate (i never did it myself), and I can't guarantee you'll eventually succeed.
Intermediate method:
redesign your layout algorithm so it layout images in the rows. Use single virtualizing listbox or ItemsControl, split your item into smaller rows, each row containing single row of images.