How to create a Racket mutable array from a C pointer to modify an OpenCV image?

Question

I am playing around with these OpenCV bindings for Racket, and I would like to be able to access the pixel values as an array, mostly so I can use row-column indexing. The examples only show how to call high-level OpenCV functions, and then accessing the raw data is only shown by a low-level access to a bytestring.

It all works great, I was quite impressed. For some applications, thought, we should be able to do some pixel-by-pixel processing, ideally utilizing whatever is the native multidimensional array object in Racket, which seems to be math:array. The question is how to create this mutable array from the IplImage object.

I was able to run the bytestring example, which is not ideal already because it wouldn't support other types like float. The following piece of code works creating a cvector out of the pointer to the image data:

(define xx (make-cvector* (IplImage-imageData frame) ffi:_uint8 307200))
(cvector-set! xx ii 255)
(set! ii (ii . + . 1) )

This paints a pixel moving at each frame inside a loop... This works. Now I would like to create an array out of this cvector. I came up with the following code:

(define (image->array image)
  (let ([data
         (make-cvector* (IplImage-imageData image)
                        ffi:_uint8
                        (IplImage-imageSize image))]
        [shape (vector (IplImage-height image) (IplImage-width image))])
    (array->mutable-array (list->array
                           shape
                           (cvector->list data)))))

This apparently works for reading though not for writing back. How do I make it actually mutable? And are there any performance concerns here? Is cvector->list perhaps copying all the data into a new Racket list? How do I create a Racket vector object that can be used by math:array?

what do you mean by "wouldn't support...float"? can you give an example of what you'd like to do with floats? — adatzer
cvector->list will copy the vector's contents. list->array will copy them again, and array->mutable-array will make one more copy. If you want accessors which will modify the bits of the original array, treating it as a two-dimensional object, I suspect you need to write them. — user5920214
@adatzer Many applications represent images with larger depths than 8 bits, and also both fixed-point and floating-point types. OpenCV supports many such types, not to mention the channels. — dividebyzero
@tfb well, the job of handling the dimensions is being done by the array functions, they just don't seem to accept a cvector instead of a vector, what would I need to implement this seemingly simple interface while preventing copies? — dividebyzero
As I said, you probably need to write them: math/array is very definitely not what you want for this, as these arrays expect to manage their own storage if they are mutable. — user5920214

adatzer adatzer · Accepted Answer · 2020-01-15T20:29:25

Your code does indeed return a mutable-array. Just that it works only for greyscale images, since list->array expects a list of length (* width height), while the cvector->list returns a list of length (* width height channels). Going all the way: cpointer >> cvector >> list >> immutable-array >> mutable-array just to have row-column indexing seems a bit too much, since you could define a function to "simulate" the access style you want.

For the performance concerns, see this StackOverflow question[1] (by the creator of racket-opencv), which explains the reasoning behind bytestrings and the IplImage-data function. You already mentioned that bytestrings are not ideal for you, but still, even if you have to go towards vectors (or math/array), try also the way: cpointer >> bytestring >> list >> vector (faster in my machine).

[1] Fast array access using Racket FFI

How to create a Racket mutable array from a C pointer to modify an OpenCV image?

1 Answers