Custom CIColorKernel in Xamarin IOS

Question

I'm trying to create a functionnality in Xamarin Forms that allow the application to change [1..N] colors to [1..N] of an images.

Example:

Change all blue & purple pixels to yellow & orange pixels

After some investigation, It seems that I need to create a custom CIColorKernel to achieve it.

The problem is that is very difficult to find examples and the documentation is light.

If someone have a tutorial or a basic example to start...

Thanks

EDIT :

I implemented the soutioion of @SushiHangover and I Call it in the second method of the code sample :

 private IImageSourceHandler GetHandler(ImageSource source)
        {
            IImageSourceHandler returnValue = null;
            if (source is UriImageSource)
            {
                returnValue = new ImageLoaderSourceHandler();
            }
            else if (source is FileImageSource)
            {
                returnValue = new FileImageSourceHandler();
            }
            else if (source is StreamImageSource)
            {
                returnValue = new StreamImagesourceHandler();
            }
            return returnValue;
        }

        public async Task<ImageSource> ChangeImageColor(ImageSource source, string oldColor, string newColor)
        {
            var handler = GetHandler(source);
            var uiImage = (UIImage)null;

            uiImage = await handler.LoadImageAsync(source);
            UIImage uiImageOutput = null;

            using (var context = new EAGLContext(EAGLRenderingAPI.OpenGLES3))
            using (var filter = new ColorReplaceFilter
            {
                InputImage = new CIImage(uiImage),
                MatchColor = CIColor.FromRgb(200, 200, 200),
                ReplaceWithColor = CIColor.RedColor,
                Threshold = 1f // Exact match, values >0 & <=1 to make a fuzzy match
            })
            {
                uiImageOutput = UIImage.FromImage(filter.OutputImage);
            }

            return Xamarin.Forms.ImageSource.FromStream(() => uiImageOutput.AsPNG().AsStream()); ;
        }

This two methods are in a class named BitmapHelper that is called in Xamarin Forms project with Dependecy Injection.

var bitmap = DependencyService.Get<IBitmapHelper>().ChangeImageColor(AmbiancePicture.Source, oldColor, newColor);
            AmbiancePicture.Source = bitmap.Result;

The result contains as expected the new Image but AmbiancePicture.Source but is not updated.

Here the image I try to change :

EDIT 2 :

If I set the AmbiancePicture.Source to null before the update, The image stay empty. The image seems to not be empty ( I see some correct properties in the stream ).

WORKING EDIT :

So after the error comes from the UIImage creation and convertion.

This is the working code :

  using (var context = new EAGLContext(EAGLRenderingAPI.OpenGLES3))
            using (var filter = new ColorReplaceFilter
            {
                InputImage = new CIImage(uiImage),
                MatchColor = CIColor.FromString(oldColor),
                ReplaceWithColor = CIColor.FromString(newColor),
                Threshold = 1f // Exact match, values >0 & <=1 to make a fuzzy match
            })
           {
            var output = context.CreateCGImage(filter.OutputImage, filter.OutputImage.Extent); // This line is slow...
            var img = UIImage.FromImage(output);
            jpegData = img.AsJPEG(1.0f);

        }
        return Xamarin.Forms.ImageSource.FromStream(() => jpegData.AsStream());

SushiHangover SushiHangover · Accepted Answer · 2017-11-16T11:52:35

Change all blue & purple pixels to yellow & orange pixels

Lets break that down into two distinct steps:

blue pixels to yellow pixels
purple pixels to orange pixels

This means we need just one CIFilter that changes a single color to another color and we can chain two (or more) of those filter together to change as many colors as needed.

In terms of a custom CIFilter, if we are changing only the color, we can use a CIColorKernel to process this on a GPU or a CPU vector unit (the OS will determine which one based on availability and kernel features requested). CIKernel subclasses use a modified version of GLSL (OpenGL Shading Language) as the language in a kernel (this code gets compiled at runtime as each device might have a different CPU and/or GPU).

So we need a CIColorKernel function that accepts the source "color" in RGA8 format as a vec4, a vec4 that represents the color to match, another vec4 to represent the color to change to if the source vec4 matches. Also we can supply a threshold that provides how "close" the original color needs to be (like chroma-keying). Taking all that and writing some GLSL, we get:

kernel vec4 main(__sample s, __color o, __color r, float threshold) {
    vec4 diff = s.rgba - o;
    float distance = length( diff );
    float alpha = compare( distance - threshold, 0.0, 1.0 );
     if (alpha == 0.0)
        return r;
     return s;
}

Now we need a CIFilter subclass to create/compile that kernel and provide the Core Image inputs and outputs to that kernel:

public class ColorReplaceFilter : CIFilter
{
    const string filterName = "colorReplace";
    const int numArgs = 4;
    const string coreImageShaderProgram =
        @"
            kernel vec4 main(__sample s, __color o, __color r, float threshold) {
                vec4 diff = s.rgba - o;
                float distance = length( diff );
                float alpha = compare( distance - threshold, 0.0, 1.0 );
                 if (alpha == 0.0)
                    return r;
                 return s;
            }
        ";

    NSObject[] arguments;
    CIColorKernel colorKernel;

    public ColorReplaceFilter() { Initializer(); }
    public ColorReplaceFilter(NSCoder coder) : base(coder) { Initializer(); }
    public ColorReplaceFilter(NSObjectFlag t) : base(t) { Initializer(); }
    public ColorReplaceFilter(IntPtr handle) : base(handle) { Initializer(); }

    public CIImage InputImage { get; set; }
    public CIColor MatchColor { get; set; }
    public CIColor ReplaceWithColor { get; set; }

    NSNumber _threshold;
    public nfloat Threshold
    {
        get { return _threshold.NFloatValue; }
        set { _threshold = NSNumber.FromNFloat(value); }
    }

    void Initializer()
    {
        arguments = new NSObject[numArgs];
        colorKernel = CIColorKernel.FromProgramSingle(coreImageShaderProgram);
        MatchColor = CIColor.WhiteColor;
        ReplaceWithColor = CIColor.WhiteColor;
        _threshold = new NSNumber(0.2f);
    }

    public override string Name
    {
        get => filterName;
    }

    public override CIImage OutputImage
    {
        get => CreateOutputImage();
    }

    CIImage CreateOutputImage()
    {
        if (InputImage != null) // Avoid object creation to allow fast filter chaining
        {
            arguments[0] = InputImage as NSObject;
            arguments[1] = MatchColor as NSObject;
            arguments[2] = ReplaceWithColor as NSObject;
            arguments[3] = _threshold as NSObject;
            var ciImage = colorKernel.ApplyWithExtent(InputImage.Extent, arguments);
            return ciImage;
        }
        return null;
    }

    protected override void Dispose(bool disposing)
    {
        base.Dispose(disposing);
        arguments = null;
        InputImage = null;
        MatchColor = null;
        ReplaceWithColor = null;
        colorKernel.Dispose();
    }
}

A simple CIContext example:

var uiImageInput = inputVIew.Image;
UIImage uiImageOutput;
using (var context = CIContext.Create())
using (var filter = new ColorReplaceFilter
{
    InputImage = new CIImage(uiImageInput),
    MatchColor = CIColor.BlueColor,
    ReplaceWithColor = CIColor.RedColor,
    Threshold = 0.0f // Exact match, values >0 & <=1 to make a fuzzy match
})
{
    uiImageOutput  = UIImage.FromImage(filter.OutputImage);
}
// Do something with your new uiImageOutput

Note: There are EAGLContext based contexts also if you have special needs for real-time image/video processing.

Note: As creating a context has overhead, you do not need the CIContext if you are rendering that CIImage directly to a UIImage holder that already has a context. A UIImageView would have a context since it is displaying an image on the screen, so remove the context creation and create/assign the CIImage backed UIImage directly to the UIImageView:

aUIImageViewObject.Image = UIImage.FromImage(replaceFilter.OutputImage);

In chaining filters, you take the CIImage output of one filter and apply that as the CIImage input to the next filter.

var uiImageInput = inputVIew.Image;
UIImage uiImageOutput;
using (var context = CIContext.Create())
using (var filter1 = new ColorReplaceFilter
{
    InputImage = new CIImage(uiImageInput),
    MatchColor = CIColor.BlueColor,
    ReplaceWithColor = CIColor.RedColor,
    Threshold = 0
})
using (var filter2 = new ColorReplaceFilter
{
    InputImage = filter1.OutputImage,
    MatchColor = CIColor.WhiteColor,
    ReplaceWithColor = CIColor.BlackColor,
    Threshold = 0
})
{
    uiImageOutput = UIImage.FromImage(filter2.OutputImage);
}
// Do something with your new UIImage

As Apple puts it, a CIImage object is a “recipe” for producing an image and since we do not "render" a UIImage|CGImage until the end of the chain above, this chained filter rendering happens in one pass as we are only passing CIImages through the chain.

Note: If you are processing a number of images with different color replacements, create the CIFilters once and keep changing the CIImage input and replacement colors to process each image reusing the filters. When you are done remember to dispose the CIFilter(s). This way the GLSL kernel only has to be compiled once per filter for an unlimited number of input/output images.

Here is an example where the input colors of the filters are being changed. The filters are created and maintained a class level objects of the UIViewController and disposed of when the view controller is closed.

Another example using the threshold to interactively replace "blue" pixels with red.

Re:Core Image Kernel Language Re: CIContext Re: CIColorKernel

Custom CIColorKernel in Xamarin IOS

2 Answers