How to correctly use ImageReader with YUV_420_888 and MediaCodec to encode video to h264 format?

Question

I\'m implementing a camera application on Android devices. Currently, I use Camera2 API and ImageReader to get image data in YUV_420_888 format, but I don\'t kn

Answer 1

You are right, YUV_420_888 is a format that can wrap different YUV 420 formats. The spec carefully explains that the arrangement of U and V planes is not prescribed, but there are certain restrictions; e.g. if the U plane has pixel stride 2, same applies to V (and then the underlying byte buffer can be NV21).

COLOR_FormatYUV420Flexible is a synonym of YUV_420_888, but they belong to different classes: MediaCodec and ImageFormat, respectively.

The spec explains:

All video codecs support flexible YUV 4:2:0 buffers since LOLLIPOP_MR1.

COLOR_FormatSurface is an opaque format that can deliver best performance for MediaCodec, but this comes at price: you cannot directly read or manipulate its content. If you need to manipulate the data that goes to the MediaCodec, then using ImageReader is an option; whether it will be more efficient than ByteBuffer, depends on what you do and how you do it. Note that for API 24+ you can work with both camera2 and MediaCodec in C++.

The invaluable resource of details for MediaCodec is http://www.bigflake.com/mediacodec. It references a full example of 264 encoding.