MediaCodec crash on high quality stream

Question

I am decoding a h264 video stream with the following code (original guide):

public void configure(Surface surface, int width, int height, ByteBuffer csd0) {
         


        

Answer 1

For my Android h.264 decoder i do it slightly different to your setup. I think your using more modern api level. But for me it looks more like this:

public void startDecoder() {
    // Initilize codec
    mediaCodec = MediaCodec.createDecoderByType("video/avc");
    mediaFormat = MediaFormat.createVideoFormat("video/avc", 0, 0);
    bufferInfo = new MediaCodec.BufferInfo();

    // STOPS unit-tests from crashing here from mocked out android
    if (mediaCodec != null) {
        mediaCodec.configure(mediaFormat, targetSurface, null, 0);
        mediaCodec.start();
        decoderThread = new Thread(this);
        decoderThread.start();
    }
}

// Decoder Thread refers to this class which does the decoder/render loop:

public void run() {
    //mediaCodec input + output dequeue timeouts
    long kInputBufferTimeoutMs = 50;
    long kOutputBufferTimeoutMs = 50;

    while (running && mediaCodec != null) {
        synchronized (mediaCodec) {
            // stop if not running.
            if (!running || mediaCodec == null)
                break;

            // Only push in new data if there is data available in the queue
            if (naluSegmentQueue.size() > 0) {
                int inputBufferIndex = mediaCodec.dequeueInputBuffer(kInputBufferTimeoutMs);
                if (inputBufferIndex >= 0) {
                    NaluSegment segment = naluSegmentQueue.poll();
                    codecInputBufferAvailable(segment, mediaCodec, inputBufferIndex);
                }
            }

            // always check if output is available.
            int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, kOutputBufferTimeoutMs);
            if (outputBufferIndex >= 0) {
                // Try and render first
                codecOuputBufferAvailable(mediaCodec, outputBufferIndex, bufferInfo);
            } else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
                // Subsequent data will conform to new format.
                // Can ignore if using getOutputFormat(outputBufferId)
                mediaFormat = mediaCodec.getOutputFormat();
            }
        }
    }
}

To put data into the decoder including the parameters. I don't bother with trying to use the csd-0/1 network streams can have changing format descriptions and its easier to just let it be picked up dynamically.

private void codecInputBufferAvailable(NaluSegment segment, MediaCodec codec, int index) {
    int flags = (segment.getType() == NaluType.SPS
            || segment.getType() == NaluType.PPS
            || segment.getType() == NaluType.SUPP_ENHANCEMENT) ?
            MediaCodec.BUFFER_FLAG_CODEC_CONFIG : MediaCodec.BUFFER_FLAG_SYNC_FRAME;

    ByteBuffer[] buffers = codec.getInputBuffers();
    ByteBuffer buffer = buffers[index];
    // Can throw buffer overflow exception when buffer sizes are too small.
    try {
        buffer.put(segment.getBuffer());
        codec.queueInputBuffer(index, 0, segment.getBufferSize(), 0, flags);
    } catch(Exception e) {
        Log.e(TAG, "Failed to push buffer to decoder");
    }
}

IMPORTANT: buffer.put(segment.getBuffer()); getBuffer() here always returns a 4 byte annexb buffer. The android decoders do not understand 3 byte nal units. So if you have a 3 byte nal unit turn it into 4 bytes magic sequence with length + 1 and 0x00, 0x00, 0x00, 0x01 as the start magic sequence the rest of the buffer should be &buffer[headerLength].

Notice the try-catch here this doesn't give a compiler warning but it can throw a buffer overflow exception here if your have a very big payload and the byte-buffer is too small.

So long as your parse out your NAL units correctly this should work for you. But for my case i noticed that the NAL units can be 3 or 4 bytes for the magic header.

/**
 * H264 is comprised of NALU segments.
 *
 * XXXX Y ZZZZZZZZ -> XXXX Y ZZZZZZZZ -> XXXX Y ZZZZZZZZ
 *
 * Each segment is comprised of:
 *
 * XXXX   -> Magic byte header (0x00, 0x00, 0x00, 0x01) NOTE: this can be either 3 of 4 bytes
 * Y      -> The Nalu Type
 * ZZZ... -> The Payload
 *
 * Notice there is no nalu length specified. To parse an nalu, you must
 * read until the next magic-byte-sequence AKA the next segment to figure
 * out the full nalu length
 **/
public static List<NaluSegment> parseNaluSegments(byte[] buffer) throws NaluBufferException {
    List<NaluSegment> segmentList = new ArrayList<>();
    if (buffer.length < 6) {
        return segmentList;
    }

    int lastStartingOffset = -1;
    for (int i = 0; i < buffer.length - 10; ++i) {
        **if (buffer[i] == 0x00 && buffer[i+1] == 0x00 && buffer[i+2] == 0x01)** {
            int naluType = (buffer[i+3] & 0x1F);
            NaluSegment segment = new NaluSegment(naluType, 3, i);

            **if (i > 0 && buffer[i-1] == 0x00)** {
                // This is actually a 4 byte segment
                int currentSegmentOffset = segment.getOffset();
                segment.setHeaderSize(4);
                segment.setOffset(currentSegmentOffset - 1);
            }
...

Create your own nalu-segment objects and don't forget the trailing NAL.

I hope this helps.