How to pass image frames camera to a function in wasm (C++)?

问题

I'm trying to build a C++ function and compile it to Wasm using Emscripten.
What this function will do is receive an image and do some process on it and return a result.
My first POC was successful, the user upload image using file input and I pass the data of the image using FileReader API:

const fileReader = new FileReader();
fileReader.onload = (event) => {
   const uint8Arr = new Uint8Array(event.target.result);
   passToWasm(event.target.result);
};

fileReader.readAsArrayBuffer(file); // I got this `file` from `change` event of the file input.

But when I implemented the camera feed and started to get frames to pass it to Wasm, I started to get exceptions in C++ side, and here's the JS implementation:

let imageData = canvasCtx.getImageData(0, 0, videoWidth, videoHeight);
var data=imageData.data.buffer;
var uint8Arr = new Uint8Array(data);
passToWasm(uint8Arr);

This one throws an exception in C++ side.

Now passToWasm implementation is:

function passToWasm(uint8ArrData) {
   // copying the uint8ArrData to the heap
   const numBytes = uint8ArrData.length * uint8ArrData.BYTES_PER_ELEMENT;
   const dataPtr = Module._malloc(numBytes);
   const dataOnHeap = new Uint8Array(Module.HEAPU8.buffer, dataPtr, numBytes);
   dataOnHeap.set(uint8ArrData);

   // calling the Wasm function
   const res = Module._myWasmFunc(dataOnHeap.byteOffset, uint8ArrData.length);
}

While the C++ implementation will be something like this:

void EMSCRIPTEN_KEEPALIVE checkImageQuality(uint8_t* buffer, size_t size) {
   // I'm using OpenCV in C++ to process the image data
   // So I read the data of the image
   cv::Mat raw_data = cv::Mat(1, size, CV_8UC1, buffer);

   // Then I convert it
   cv::Mat img_data = cv::imdecode(raw_data, cv::IMREAD_COLOR | cv::IMREAD_IGNORE_ORIENTATION);

   // in one of the following steps I'm using cvtColor function which causes the issue for some reason
}

The exception I'm getting because of the camera implementation says:

OpenCV(4.1.0-dev) ../modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

What is the difference between using file input and getting the data to pass it, and getting the data from a canvas as long as both of them are convert it to Uint8Array

回答1:

I found a solution for this (maybe suits my case only).
When you're trying to get an image data from canvas you get it as 4 channels (RGBA like in PNG), and depending on your image processing code you need to deal with it.
My code was considering that the image should be 3 channels (RGB like in jpeg) so I had to convert it using this code:

canvasBuffer.toBlob(function (blob) {
  passToWASM(blob);
},'image/jpeg');

来源：https://stackoverflow.com/questions/55845033/how-to-pass-image-frames-camera-to-a-function-in-wasm-c

标签

javascript

c++

OpenCV

webassembly