Your way to determine the page size
PDFRenderer pdRender = new PDFRenderer(pdDoc);
BufferedImage singlePage = pdRender.renderImage(pgIdx-1);
singlePage.getWidth(); // pageWidth = 623
is determining the page width in pixel after rendering the page as bitmap using some default settings, in particular at some unknown resolution.
Your way to determine the image dimension
PDImageXObject image = (PDImageXObject) o;
image.getImage(); // imageWidth = 484
is determining the actual dimensions of the bitmap resource without consideration of how it is used on the page if at all.
Thus, those numbers are entirely unrelated.
If you want to compare sizes on a PDF page, the natural choice of units would be the default user space units of a PDF page. By default they equal 1/72 inch.
You can retrieve the page size of a PDPage page in user space units like this:
PDRectangle cropBox = page.getCropBox();
float width = cropBox.getWidth();
float height = cropBox.getHeight();
The dimensions of a bitmap on a PDF page are a bit more difficult because a bitmap is subject to an arbitrary affine transformation, the current transformation matrix (CTM) at the time it is drawn. Thus, you have to determine that CTM value. To do so you have to parse the page content up to the point at which the bitmap is drawn, and right then you have to read the CTM from the current transformation matrix.
The PDFBox example PrintImageLocations demonstrates this, the output "displayed size = XXX, YYY in user space units" is the one you're looking for.