Analyzing bitmaps produced by NSAffineTransform and CILineOverlay filters

Question

I am trying to manipulate an image using a chain of CIFilters, and then examine each byte of the resulting image (bitmap). Long term, I do not need to display the resulting image (bitmap) -- I just need to "analyze" it in memory. But near-term I am displaying it on screen, to help with debugging.

I have some "bitmap examination" code that works as expected when examining the NSImage (bitmap representation) I use as my input (loaded from a JPG file into an NSImage). And it SOMETIMES works as expected when I use it on the outputBitmap produced by the code below. More specifically, when I use an NSAffineTransform filter to create outputBitmap, then outputBitmap contains the data I would expect. But if I use a CILineOverlay filter to create the outputBitmap, none of the bytes in the bitmap have any data in them. I believe both of these filters are working as expected, because when I display their results on screen (via outputImageView), they look "correct." Yet when I examine the outputBitmaps, the one created from the CILineOverlay filter is "empty" while the one created from NSAffineTransfer contains data. Furthermore, if I chain the two filters together, the final resulting bitmap only seems to contain data if I run the AffineTransform last. Seems very strange, to me???

My understanding (from reading the CI programming guide) is that the CIImage should be considered an "image recipe" rather than an actual image, because the image isn't actually created until the image is "drawn." Given that, it would make sense that the CIimage bitmap doesn't have data -- but I don't understand why it has data after I run the NSAffineTransform but doesn't have data after running the CILineOverlay transform? Basically, I am trying to determine if creating the NSCIImageRep (ir in the code below) from the CIImage (myResult) is equivalent to "drawing" the CIImage -- in other words if that should force the bitmap to be populated? If someone knows the answer to this please let me know -- it will save me a few hours of trial and error experimenting!

Finally, if the answer is "you must draw to a graphics context" ... then I have another question: would I need to do something along the lines of what is described in the Quartz 2D Programming Guide: Graphics Contexts, listing 2-7 and 2-8: drawing to a bitmap graphics context? That is the path down which I am about to head ... but it seems like a lot of code just to force the bitmap data to be dumped into an array where I can get at it. So if there is an easier or better way please let me know. I just want to take the data (that should be) in myResult and put it into a bitmap array where I can access it at the byte level. And since I already have code that works with an NSBitmapImageRep, unless doing it that way is a bad idea for some reason that is not readily apparent to me, then I would prefer to "convert" myResult into an NSBitmapImageRep.

CIImage * myResult = [transform valueForKey:@"outputImage"]; 
NSImage *outputImage;
NSCIImageRep *ir = [NSCIImageRep alloc];
ir = [NSCIImageRep imageRepWithCIImage:myResult];
outputImage = [[[NSImage alloc] initWithSize:
                              NSMakeSize(inputImage.size.width, inputImage.size.height)]
                             autorelease];
                    [outputImage addRepresentation:ir];
[outputImageView setImage: outputImage];
NSBitmapImageRep *outputBitmap = [[NSBitmapImageRep alloc] initWithCIImage: myResult];

Thanks, Adam

Edit #1 -- for Peter H. comment: Sample code accessing bitmap data...

for (row = 0; row < heightInPixels; row++)
  for (column = 0; column < widthInPixels; column++) {
    if (row == 1340) {  //just check this one row, that I know what to expect
        NSLog(@"Row 1340 column %d pixel redByte of pixel is %d",column,thisPixel->redByte);
    }
}

Results from above (all columns contain the same zero/null value, which is what I called "empty")...

2010-06-13 10:39:07.765 ImageTransform[5582:a0f] Row 1340 column 1664 pixel redByte of pixel is 0
2010-06-13 10:39:07.765 ImageTransform[5582:a0f] Row 1340 column 1665 pixel redByte of pixel is 0
2010-06-13 10:39:07.766 ImageTransform[5582:a0f] Row 1340 column 1666 pixel redByte of pixel is 0

If I change the %d to %h nothing prints at all (blank rather than "0"). If I change it to %@ I get "(null)" on every line, instead of the "0" shown above. On the other hand ... when I run just the NSAffineTransform filter and then execute this code the bytes printed contain the data I would expect (regardless of how I format the NSLog output, something prints).

Adding more code on 6/14 ...

// prior code retrieves JPG image from disk and loads into NSImage
CIImage * inputCIimage = [[CIImage alloc] initWithBitmapImageRep:inputBitmap];
if (inputCIimage == nil) {
  NSLog(@"Bailing out.  Could not create CI Image");
  return;
}
NSLog (@"CI Image created.  working on transforms...");

Filter that rotates image.... this was previously in a method, but I have since moved it to be "in line" as I have been trying to figure out what is wrong...

// rotate imageIn by degreesToRotate, using an AffineTransform      
CIFilter *transform = [CIFilter filterWithName:@"CIAffineTransform"];
[transform setDefaults];
[transform setValue:inputCIimage forKey:@"inputImage"];
NSAffineTransform *affineTransform = [NSAffineTransform transform];
[affineTransform transformPoint: NSMakePoint(inputImage.size.width/2, inputImage.size.height / 2)]; 
//inputImage.size.width /2.0,inputImage.size.height /2.0)];
[affineTransform rotateByDegrees:3.75];
[transform setValue:affineTransform forKey:@"inputTransform"];
CIImage * myResult2 = [transform valueForKey:@"outputImage"]; 

Filter to apply CILineOverlay filter... (was also previously in a method)

CIFilter *lineOverlay = [CIFilter filterWithName:@"CILineOverlay"];
[lineOverlay setDefaults];
[lineOverlay setValue: inputCIimage forKey:@"inputImage"];
// start off with default values, then tweak the ones needed to achieve desired results
[lineOverlay setValue: [NSNumber numberWithFloat: .07] forKey:@"inputNRNoiseLevel"]; //.07 (0-1)
[lineOverlay setValue: [NSNumber numberWithFloat: .71] forKey:@"inputNRSharpness"]; //.71 (0-2)
[lineOverlay setValue: [NSNumber numberWithFloat: 1] forKey:@"inputEdgeIntensity"]; //1 (0-200)
[lineOverlay setValue: [NSNumber numberWithFloat: .1] forKey:@"inputThreshold"]; //.1 (0-1)
[lineOverlay setValue: [NSNumber numberWithFloat: 50] forKey:@"inputContrast"]; //50 (.25-200)
CIImage *myResult2 = [lineOverlay valueForKey:@"outputImage"];  //apply the filter to the CIImage object and return it

Finally ... the code that uses the results...

if (myResult2 == Nil)
    NSLog(@"Transformations failed");
else {
NSLog(@"Finished transformations successfully ... now render final image");
// make an NSImage from the CIImage (to display it, during initial development)
NSImage *outputImage;
NSCIImageRep *ir = [NSCIImageRep alloc];
// show the tranformed output on screen...
ir = [NSCIImageRep imageRepWithCIImage:myResult2];
outputImage = [[[NSImage alloc] initWithSize:
              NSMakeSize(inputImage.size.width, inputImage.size.height)]
             autorelease];
[outputImage addRepresentation:ir];
[outputImageView setImage: outputImage];  //rotatedImage

At this point the transformed image displays on screen just fine, regardless of which transform I apply and which one I leave commented out. It even works just fine if I "chain" together the transforms so that the output from #1 goes into #2. So, to me, this seems to indicates that the filters are working.

However ... the code that I really need to use is the "bitmap analysis" code that is examining the bitmap that is in (or "should be" in) Results2. And that code works only on the bitmap resulting from the CIAffineTransform filter. When I use it to examine the bitmap resulting from the CILineOverlay, the entire bitmap seems to contain only zeroes.

So here is the code used for that analysis...

// this is the next line after the [outputImageView ...] shown above
[self findLeftEdge :myResult2];

And then this is the code from the findLeftEdge method...

- (void) findLeftEdge :(CIImage*)imageInCI {
    // find the left edge of the input image, assuming it will be the first non-white pixel
    // because we have already applied the Threshold filter

    NSBitmapImageRep *outputBitmap = [[NSBitmapImageRep alloc] initWithCIImage: imageInCI];
    if (outputBitmap == nil)
        NSLog(@"unable to create outputBitmap");
    else 
        NSLog(@"ouputBitmap image rep created -- samples per pixel = %d", [outputBitmap samplesPerPixel]);

    RGBAPixel 
        *thisPixel, 
        *bitmapPixels = (RGBAPixel *)[outputBitmap bitmapData];  

    int 
    row, 
    column,
    widthInPixels = [outputBitmap pixelsWide], 
    heightInPixels = [outputBitmap pixelsHigh];

    //RGBAPixel *leftEdge [heightInPixels];
    struct { 
        int pixelNumber;
        unsigned char pixelValue;
    } leftEdge[heightInPixels];

    // Is this necessary, or does objective-c always intialize it to zero, for me?
    for (row = 0; row < heightInPixels; row++) {
        leftEdge[row].pixelNumber = 0;
        leftEdge[row].pixelValue = 0;
    }

    for (row = 0; row < heightInPixels; row++)
        for (column = 0; column < widthInPixels; column++)  {
            thisPixel = (&bitmapPixels[((widthInPixels * row) + column)]);

            //red is as good as any channel, for this test (assume threshold filter already applied)
            //this should "save" the column number of the first non-white pixel encountered
            if (leftEdge[row].pixelValue < thisPixel->redByte) {
                leftEdge[row].pixelValue = thisPixel->redByte;  
                leftEdge[row].pixelNumber = column;
            }
            // For debugging, display contents of each pixel
            //NSLog(@"Row %d column %d pixel redByte of pixel is %@",row,column,thisPixel->redByte);
            // For debugging, display contents of each pixel on one row
            //if (row == 1340) {
            //  NSLog(@"Row 1340 column %d pixel redByte of pixel is %@",column,thisPixel->redByte);
            //}

        }

    // For debugging, display the left edge that we discovered
    for (row = 0; row < heightInPixels; row++) {
        NSLog(@"Left edge on row %d was at pixel #%d", row, leftEdge[row].pixelNumber);
    }

    [outputBitmap release];  
}

Here is another filter. When I use it I do get data in the "output bitmap" (just like the Rotation filter). So it is just the AffineTransform that does not yield up its data for me in the resulting bitmap ...

- (CIImage*) applyCropToCI:(CIImage*) imageIn {
rectToCrop { 
    // crop the rectangle specified from the input image
    CIFilter *crop = [CIFilter filterWithName:@"CICrop"];
    [crop setDefaults];  
    [crop setValue:imageIn forKey:@"inputImage"];
//  [crop setValue:rectToCrop forKey:@"inputRectangle"];  //vector defaults to 0,0,300,300
    //CIImage * myResult = [transform valueForKey:@"outputImage"]; //this is the way it was "in-line", before putting this code into a method
    return [crop valueForKey:@"outputImage"];   //does this need to be retained?
}

Answer 1

for (row = 0; row < heightInPixels; row++)
  for (column = 0; column < widthInPixels; column++) {
    if (row == 1340) {  //just check this one row, that I know what to expect
        NSLog(@"Row 1340 column %d pixel redByte of pixel is %d",column,thisPixel->redByte);
  }
}

Aside from the syntax error, this code doesn't work because you never assigned to thisPixel. You are looping through indexes for nothing, since you never actually look up a pixel value at those indexes and assign it to thisPixel in order to inspect it.

Add such an assignment before the NSLog statement.

Furthermore, if the only row you care about is 1340, there's no need to loop through rows. Check using an if statement whether 1340 is less than the height, and if it is, then do only the columns loop. (Also, don't embed magic number literals like this in your code. Give that constant a name that explains the significance of the number 1340—i.e., why it's the only row you care about.)

Answer 2

You claim that the bitmap data contains “all zeroes”, but you're only looking at one byte per pixel. You're assuming that the first component is the red component, and you're assuming that the data is one byte per component; if the data is alpha-first or floating-point, one or both of these assumptions will be wrong.

Create a bitmap context in whatever format you want using a buffer you allocate, and render the image into that context. Your buffer will then contain the image in the format you expect.

You might also want to switch from structure-based access to byte-based access—i.e., pixels[(row*bytesPerRow)+col], incrementing col by the number of components per pixel. Endianness can easily become a headache when you use structures to access the components.