CameraImage conversion returns red Image

[collo54]

https://github.com/brendan-duncan/image/issues/654#issuecomment-2270202591

When converting CameraImage object from camera package using the Image package I get a red image instead of the correct image color as shown below (expected vs red error image). Code below image.

Uint8List captureJpeg(CameraImage image) {
 try {
      // img represents "import 'package:image/image.dart' as img;"
      // image parameter is type "ImageFormatGroup.yuv420" from camera package pub.dev "contoller.startImageStream((Image){})"
      // ImageFormatGroup.yuv420 description from android developer docs is "Images in this format are always represented by three separate buffers of data, one for each color plane".
      final imageData = img.Image.fromBytes(
        width: image.width,
        height: image.height,
        bytes: image.planes[0].bytes.buffer,
        order: img.ChannelOrder.bgra,
        numChannels: 1,
         );

      final jpeg = Uint8List.fromList(img.encodeJpg(imageData));
      return jpeg;
    } catch (e) {
      print('Error capturing image: $e');
      return Uint8List(0);
    }
  }

[brendan-duncan]

You mentioned before the camera image is in yuv420 colors, which isn't the same as bgra colors you're telling Image. Perhaps you can try converting the yuv420 colors, like in #535 (comment).

[collo54]

@brendan-duncan Thank you for the reply. However, all the solutions discussed in #535 (comment) don't work. I got an Index Range error. If possible could you try and modify the capturejpeg function in the main.dart code in the provided repository https://github.com/collo54/vision_edit/blob/d0c9d11d81c8150de5d00e2437101ea1bf382da2/lib/main.dart#L143 to show the fix.
I'll maintain the repo as public so other developers can reference it in the future. I hope this isn't too much of a bother.

[brendan-duncan]

I can certainly help but I have very little experience with Flutter, so I can offer suggestions but I'm not set up to do any testing to see if any of this works at the moment. I wish I had some more time.

CameraImage has a format that defines how its image data is stored, https://pub.dev/documentation/camera/latest/camera/CameraImage-class.html.

The format can be one of the ones defined in https://pub.dev/documentation/camera/latest/camera/ImageFormatGroup.html. So the full function to convert a CameraImage to an img.Image would need to be based on the format. I believe on iOS the CameraImage format is bgra888, and on Android it will be yuv420, which will need additional conversion like the #535 issue suggests.

For the image you posted here, my suspicion is you're testing on iOS and the format of your CameraImage is bgra8888. That makes sense since the images you're showing look intact, they're just red.

So why is it red? You are setting the numChannels of the Image to 1, so the image will only have a red channel. If you set numChannels to 4, then it would be an rgba image. Then, specifying the Image channel order as img.ChannelOrder.bgra, it should copy the bytes in the correct order.

Something you might possibly run into is a shifting of the pixels. If you don't notice that, you can skip the following.

There was another bug reported a while back related to bgra8888 CameraImage, #599. With that bug, they converted the CameraImage to an Image similar to what you are trying to do.

Their code was

CameraImage image; // from cameraController.startImageStream((image) { ... });
final img = imglib.Image.fromBytes(
  width: image.width,
  height: image.height,
  bytes: image.planes.first.bytes.buffer,
  rowStride: image.planes.first.bytesPerRow,
  numChannels: 4,
  order: imglib.ChannelOrder.bgra,
);

The issue they had reported was that the pixels are shifted. It turns out there seems to be some sort of header with the CameraImage bytes. So if the above code works for you but the pixels seem offset, you can try their suggestion,

CameraImage image; // from cameraController.startImageStream((image) { ... });
final img = imglib.Image.fromBytes(
  width: image.width,
  height: image.height,
  bytes: image.planes.first.bytes.buffer,
  byteOffset: 28, // <---- offset the buffer bytes
  rowStride: image.planes.first.bytesPerRow,
  numChannels: 4,
  order: imglib.ChannelOrder.bgra,
);

[collo54]

@brendan-duncan The CameraImage format is yuv420. I am testing on android. The code provided above #669 (comment) doesn't work. When I change the numChannels to 4 I get a new error shown below. Only when I use numChannels: 1 that's when I get any sort of image output (red image). Nothing else works.

I/flutter ( 6008):  Bad state: Too few elements
E/FlutterJNI( 6008): Failed to decode image
E/FlutterJNI( 6008): android.graphics.ImageDecoder$DecodeException: Failed to create image decoder with message 'unimplemented'Input contained an error.

I suppose its all boils down to "Android it will be yuv420, which will need additional conversion like the #535 issue suggests" as you said. If you have updated info on how to do this I am sure it will work.Thank you.

[collo54]

I've found a partial solution. Partial because the solution is slow so a different isolate should be configured to handle the process. First follow the solution in the link

First Get Y, U, V from the planes,

CameraImage availableImage;
int imageWidth = availableImage.width;
int imageHeight = availableImage.height;
int imageStride = availableImage.planes[0].bytesPerRow;
List<Uint8List> planes = [];
for (int planeIndex = 0; planeIndex < 3; planeIndex++) {
          Uint8List buffer;
          int width;
          int height;
          if (planeIndex == 0) {
            width = availableImage.width;
            height = availableImage.height;
          } else {
            width = availableImage.width ~/ 2;
            height = availableImage.height ~/ 2;
          }

          buffer = Uint8List(width * height);

          int pixelStride = availableImage.planes[planeIndex].bytesPerPixel!;
          int rowStride = availableImage.planes[planeIndex].bytesPerRow;
          int index = 0;
          for (int i = 0; i < height; i++) {
            for (int j = 0; j < width; j++) {
              buffer[index++] = availableImage
                  .planes[planeIndex].bytes[i * rowStride + j * pixelStride];
            }
          }

          planes.add(buffer);
      
}

Second Convert YUV420 to RGBA8888

Uint8List yuv420ToRgba8888(List<Uint8List> planes, int width, int height) {
  final yPlane = planes[0];
  final uPlane = planes[1];
  final vPlane = planes[2];

  final Uint8List rgbaBytes = Uint8List(width * height * 4);

  for (int y = 0; y < height; y++) {
    for (int x = 0; x < width; x++) {
      final int yIndex = y * width + x;
      final int uvIndex = (y ~/ 2) * (width ~/ 2) + (x ~/ 2);

      final int yValue = yPlane[yIndex] & 0xFF;
      final int uValue = uPlane[uvIndex] & 0xFF;
      final int vValue = vPlane[uvIndex] & 0xFF;

      final int r = (yValue + 1.13983 * (vValue - 128)).round().clamp(0, 255);
      final int g =
          (yValue - 0.39465 * (uValue - 128) - 0.58060 * (vValue - 128))
              .round()
              .clamp(0, 255);
      final int b = (yValue + 2.03211 * (uValue - 128)).round().clamp(0, 255);

      final int rgbaIndex = yIndex * 4;
      rgbaBytes[rgbaIndex] = r.toUnsigned(8);
      rgbaBytes[rgbaIndex + 1] = g.toUnsigned(8);
      rgbaBytes[rgbaIndex + 2] = b.toUnsigned(8);
      rgbaBytes[rgbaIndex + 3] = 255; // Alpha value
    }
  }

  return rgbaBytes;
}

Uint8List data = yuv420ToRgba8888(planes, imageWidth, imageHeight);

Third Create a ui.Image from RGBA8888

import 'dart:ui' as ui;
Future<ui.Image> createImage(
    Uint8List buffer, int width, int height, ui.PixelFormat pixelFormat) {
  final Completer<ui.Image> completer = Completer();

  ui.decodeImageFromPixels(buffer, width, height, pixelFormat, (ui.Image img) {
    completer.complete(img);
  });

  return completer.future;
}

Lastly Convert the Flutter UI Image to the Dart Image Library

  Future<Uint8List> convertFlutterUiToImage(ui.Image uiImage) async {
    final uiBytes = await uiImage.toByteData();
    if (uiBytes == null) {
      throw Exception('Failed to convert UI image to ByteData');
    }

    final image = img.Image.fromBytes(
      width: uiImage.width,
      height: uiImage.height,
      bytes: uiBytes.buffer,
      numChannels: 4,
    );

    final rotatedImage = img.copyRotate(image, angle: 90);

    final uint8list = Uint8List.fromList(img.encodeJpg(
      rotatedImage,
      chroma: img.JpegChroma.yuv420,
    ));

    return uint8list;
  }
}

The Uint8List can then be rendered using Flutter widgets like Image.memory

Image.memory(uint8Listdata);

@brendan-duncan Thank you for the leads provided. Cheers.

[brendan-duncan]

Seems to me you should be able to combine all those steps into a single step. This should work for yuv420 format CameraImage's. bgra8888 would be converted as described above.

CameraImage fromImage;
int width = fromImage.width;
int height = fromImage.height;
int halfWidth = width ~/ 2;
int halfHeight = height ~/ 2;
int pixelStride0 = fromImage.planes[0].bytesPerPixel!;
int rowStride0 = fromImage.planes[0].bytesPerRow;
int pixelStride1 = fromImage.planes[0].bytesPerPixel!;
int rowStride1 = fromImage.planes[0].bytesPerRow;

// img.Image will have a default numChannels of 3, RGB. Camera images won't have transparency so an alpha isn't necessary.
img.Image toImage = img.Image(width: width, height: height);

// Get a Pixel iterator for the first pixel
var toPixel = toImage.getPixel(0, 0);

for (int i = 0; i < height; i++) {
    for (int j = 0; j < width; j++) {
        int y = fromImage.planes[0].bytes[i * rowStride0 + j * pixelStride0];
        int u = fromImage.planes[1].bytes[i * rowStride1 + j * pixelStride1];
        int v = fromImage.planes[2].bytes[i * rowStride1 + j * pixelStride1];
        
        final int r = (y + 1.13983 * (v - 128)).round().clamp(0, 255);
        final int g = (y - 0.39465 * (u - 128) - 0.58060 * (v - 128)).round().clamp(0, 255);
        final int b = (y + 2.03211 * (u - 128)).round().clamp(0, 255);
        
        toPixel.setRgb(r, g, b);
        
        // Go to the next pixel in the toImage
        toPixel.moveNext();
    }
}

[abhinovpankaj]

I tried this to stream the imagedata over websocket, the data transmits well but there is a lag of few milliseconds,.
which is managble.But there is flickering problem on the receiving device.
Can anyone suggest why a white flash keeps coming.