Rotate camera preview to Portrait Android OpenCV Camera

Question

I am trying to use OpenCV 2.4.3.2 to create a camera app and do some opencv processing. I would like it to be able to have multiple UI orientations, not just Landscape.

Answer 1

Thanks to @Kaye Wrobleski for his answer. I have extended it to allow both landscape and portrait orientation. This is basically just a little extra code to allow easily switching between the default code that gives landscape orientation, and his code for portrait.

Insert his code as a new method in CameraBridgeViewBase.java

protected void deliverAndDrawFramePortrait(CvCameraViewFrame frame) {
        Mat modified;

        if (mListener != null) {
            modified = mListener.onCameraFrame(frame);
        } else {
            modified = frame.rgba();
        }

        boolean bmpValid = true;
        if (modified != null) {
            try {
                Utils.matToBitmap(modified, mCacheBitmap);
            } catch(Exception e) {
                Log.e(TAG, "Mat type: " + modified);
                Log.e(TAG, "Bitmap type: " + mCacheBitmap.getWidth() + "*" + mCacheBitmap.getHeight());
                Log.e(TAG, "Utils.matToBitmap() throws an exception: " + e.getMessage());
                bmpValid = false;
            }
        }

        if (bmpValid && mCacheBitmap != null) {
            Canvas canvas = getHolder().lockCanvas();
            // Rotate canvas to 90 degrees
            canvas.rotate(90f, canvas.getWidth()/2, canvas.getHeight()/2);
            if (canvas != null) {
                canvas.drawColor(0, android.graphics.PorterDuff.Mode.CLEAR);
                Log.d(TAG, "mStretch value: " + mScale);

                if (mScale != 0) {
                    // Resize
                    Bitmap bitmap = Bitmap.createScaledBitmap(mCacheBitmap, canvas.getHeight(), canvas.getWidth(), true);
                    // Use bitmap instead of mCacheBitmap
                    canvas.drawBitmap(bitmap, new Rect(0,0,bitmap.getWidth(), bitmap.getHeight()), new Rect(
                            (int)((canvas.getWidth() - mScale*bitmap.getWidth()) / 2),
                            (int)((canvas.getHeight() - mScale*bitmap.getHeight()) / 2),
                            (int)((canvas.getWidth() - mScale*bitmap.getWidth()) / 2 + mScale*bitmap.getWidth()),
                            (int)((canvas.getHeight() - mScale*bitmap.getHeight()) / 2 + mScale*bitmap.getHeight())), null);
                } else {
                    Bitmap bitmap = Bitmap.createScaledBitmap(mCacheBitmap, canvas.getHeight(), canvas.getWidth(), true);
                    // Use bitmap instead of mCacheBitmap
                    canvas.drawBitmap(bitmap, new Rect(0,0,bitmap.getWidth(), bitmap.getHeight()), new Rect(
                            (int)((canvas.getWidth() - bitmap.getWidth()) / 2),
                            (int)((canvas.getHeight() - bitmap.getHeight()) / 2),
                            (int)((canvas.getWidth() - bitmap.getWidth()) / 2 + bitmap.getWidth()),
                            (int)((canvas.getHeight() - bitmap.getHeight()) / 2 + bitmap.getHeight())), null);
                }

                if (mFpsMeter != null) {
                    mFpsMeter.measure();
                    mFpsMeter.draw(canvas, 20, 30);
                }
                getHolder().unlockCanvasAndPost(canvas);
            }
        }
    }

Then modify JavaCameraView.java

Add a new variable to track whether we are in portrait or landscape mode

private boolean portraitMode;

Add two methods to set the orientation mode

public void setLandscapeMode() {
        portraitMode = false;
    }
    public void setPortraitMode() {
        portraitMode = true;
    }

Now replace these lines in the JavaCameraView CameraWorkerClass, run() method

if (!mFrameChain[1 - mChainIdx].empty())
                        deliverAndDrawFrame(mCameraFrame[1 - mChainIdx]);

With these lines:

if (!mFrameChain[1 - mChainIdx].empty()) {
                        if (!portraitMode) {
                            deliverAndDrawFrame(mCameraFrame[1 - mChainIdx]);
                        } else {
                            deliverAndDrawFramePortrait(mCameraFrame[1 - mChainIdx]);
                        }
                    }

To switch between orientations, simply call either setLandscapeMode() or setPortraitMode() on your JavaCameraView object.

Please note that reverse portrait and reverse landscape orientations will still be upside-down. You will need to rotate them 180 degrees to get them right-side up, which is easily done with OpenCV's warpAffine() method. Note when using the back camera (LENS_FACING_BACK), portrait mode will flip the images upside down.

Answer 2

Modify your code in JavaCameraView.java as outlined on this page

It is really easy to fix.

Before

Log.d(TAG, "startPreview");

mCamera.startPreview();

After

Log.d(TAG, "startPreview");

setDisplayOrientation(mCamera, 90);

mCamera.setPreviewDisplay(getHolder());

mCamera.startPreview();

Answer 3

Unfortunately Opencv4Android does not support portrait camera. But there's a way how to overcome it. 1)Write your custom Camera and set it's orientation to portrait. 2)Register for it's preview callback. 3)In onPreviewFrame(byte[]data, Camera camera) create Mat of preview bytes:

Mat mat = new Mat(previewSize.height, previewSize.width, CvType.CV_8UC1);
mat.put(0, 0, data);

Core.transpose(mat, mat);
Core.flip(mat, mat, -1); // rotates Mat to portrait

CvType depends on a preview format your camera is using.

PS. do not forget to release all the Mat instances you've created when you're done.

PPS. it's good to manage your camera on a separate thread in order not to overload UI thread while doing some detection.

Answer 4

You have to consider a few things:

• onPreviewFrame() always delivers the raw camera data in its assambled rotation
• getSupportedPreviewSizes() gives corresponding aspect ratios
• Algorithm needs to analyze the frame in portrait to detect objects correct.
• the Bitmap created (Java-side) to store the resulting frame also needs the correct aspect ratio

So, for a fast and high-resolution solution i changed JavaCameraView.java and my JNI-part. in JavaCameraView.java:

...

      if (sizes != null) {
         /* Select the size that fits surface considering maximum size allowed */
         Size frameSize;
         if(width > height)
         {
            frameSize = calculateCameraFrameSize(sizes, new JavaCameraSizeAccessor(), width, height);
         }else{
            frameSize = calculateCameraFrameSize(sizes, new JavaCameraSizeAccessor(), height, width);
         }
...

         mCamera.setParameters(params);
         params = mCamera.getParameters();

         int bufFrameWidth, bufFrameHeight;
         bufFrameWidth = params.getPreviewSize().width;
         bufFrameHeight = params.getPreviewSize().height;

         if(width > height) {
             mFrameWidth = params.getPreviewSize().width;
             mFrameHeight = params.getPreviewSize().height;
         }else{
             mFrameWidth = params.getPreviewSize().height;
             mFrameHeight = params.getPreviewSize().width;
         }
...

         mFrameChain = new Mat[2];
         mFrameChain[0] = new Mat(bufFrameHeight + (bufFrameHeight/2), bufFrameWidth, CvType.CV_8UC1);
         mFrameChain[1] = new Mat(bufFrameHeight + (bufFrameHeight/2), bufFrameWidth, CvType.CV_8UC1);

         AllocateCache();

         mCameraFrame = new JavaCameraFrame[2];
         mCameraFrame[0] = new JavaCameraFrame(mFrameChain[0], bufFrameWidth, bufFrameHeight);
         mCameraFrame[1] = new JavaCameraFrame(mFrameChain[1], bufFrameWidth, bufFrameHeight);

With these changes, we made sure we are using the highest resultion available for portrait (switches height/width in calculateCameraFrameSize). We are still handling landscape as input from onPreviewFrame() but created a Bitmap to draw in portrait (AllocateCache).

Last, we need to give the algorithm the portrait-frame in order to let him detect "standing" objects and return it for saving and rendering the bitmap. So following modifications to your Activity:

public Mat rot90(Mat matImage, int rotflag){
    //1=CW, 2=CCW, 3=180
    Mat rotated = new Mat();
    if (rotflag == 1){
        rotated = matImage.t();
        flip(rotated, rotated, 1); //transpose+flip(1)=CW
    } else if (rotflag == 2) {
        rotated = matImage.t();
        flip(rotated, rotated,0); //transpose+flip(0)=CCW
    } else if (rotflag ==3){
        flip(matImage, rotated,-1);    //flip(-1)=180
    } else if (rotflag != 0){ //if not 0,1,2,3:
       Log.e(TAG, "Unknown rotation flag("+rotflag+")");
    }
    return rotated;
}

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

    mRgba = rot90(inputFrame.rgba(), 1);
    mGray = rot90(inputFrame.gray(), 1);
...

Answer 5

I`ve got portrait orientation with CameraBridgeViewBase, but I had to change JavaCameraView.java inside the OpenCV :( The idea is next: after camera init, do next

setDisplayOrientation(mCamera, 90);
mCamera.setPreviewDisplay(getHolder());

and setDisplayOrientation method

protected void setDisplayOrientation(Camera camera, int angle){
    Method downPolymorphic;
    try
    {
        downPolymorphic = camera.getClass().getMethod("setDisplayOrientation", new Class[] { int.class });
        if (downPolymorphic != null)
            downPolymorphic.invoke(camera, new Object[] { angle });
    }
    catch (Exception e1)
    {
    }
}