Android MediaCodec appears to buffer H264 frames

Question

I'm manually reading a RTP/H264 stream and pass the H264 frames to the Android MediaCodec. I use the "markerBit" as a border for the frames. The MediaCodec is tied to a OpenGL Texture (SurfaceTexture). In general everything works fine. But the Decoder appears to buffer frames. If I put a frame in the decoder it is not rendered immediately to the texture. After I put 2-3 frames more in the decoder the first frame is rendered to the texture.

I'm implementing against Android 4.4.4.

private static final int INFINITE_TIMEOUT = -1;
private static final int TIMEOUT_OUTPUT_BUFFER_MEDIA_CODEC = 1000;
...
int bufferIndex = codec.dequeueInputBuffer(INFINITE_TIMEOUT);
if (bufferIndex < 0) {
  throw new RuntimeException("Error");
}

ByteBuffer inputBuffer = inputBuffers[bufferIndex];
inputBuffer.clear();

// Copy H264 data to inputBuffer
h264Frame.fill(inputBuffer);

codec.queueInputBuffer(bufferIndex, 0, inputBuffer.position(), 0, 0);
drainOutputBuffers();
...

and

private boolean drainOutputBuffers() {
MediaCodec.BufferInfo buffInfo = new MediaCodec.BufferInfo();

int outputBufferIndex = codec.dequeueOutputBuffer(buffInfo, TIMEOUT_OUTPUT_BUFFER_MEDIA_CODEC);

if (outputBufferIndex >= 0) {
  codec.releaseOutputBuffer(outputBufferIndex, true);
  return true;
}

switch (outputBufferIndex) {
  case MediaCodec.INFO_TRY_AGAIN_LATER:
    LOG.debug("Could not dequeue output buffer. Try again later");
    break;
  case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
    LOG.warn("The output format has changed.");
    break;
  case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
    LOG.warn("The output buffers has changed.");
    break;
  default:
    LOG.warn("The output buffer index was negative: {}", outputBufferIndex);
}
return false;
}

On the rendering side I use the "onFrameAvailable" callback for checking if I have to update the texture on the openGl Thread. The flag I use for checking is guarded by a lock (synchronized).

I suspect that the presentation timestamp may influence the rendering. But I set it to 0. Thus I assume the frame should be rendered without a delay.

I'd like to have the frame rendered to the texture without having to put additional frames.

Answer 1

From the MediaCodec documentation

The Executing state has three sub-states: Flushed, Running and End-of-Stream. Immediately after start() the codec is in the Flushed sub-state, where it holds all the buffers. As soon as the first input buffer is dequeued, the codec moves to the Running sub-state, where it spends most of its life. When you queue an input buffer with the end-of-stream marker, the codec transitions to the End-of-Stream sub-state. In this state the codec no longer accepts further input buffers, but still generates output buffers until the end-of-stream is reached on the output. You can move back to the Flushed sub-state at any time while in the Executing state using flush().

You need to "queue an input buffer with the end-of-stream marker". Do this with the first frame you feed to the decoder (make sure it is a keyframe).

This point is to tell the decoder not to expect anymore frames and therefore begin playback immediately. Otherwise it's normal to feed 3 or 4 frames before seeing anything. This an expectation of all MPEG decoders and is not Android-related.

Answer 2

Mediacodec decoder buffers 6-7 frames before outputting first decoded output frame. It seems to flaw in the mediacodec. This will be problem in the streaming application. So far my debugging shows decoding H264 with mediacodec has 6-7 frames delay during start of the stream.