Does RTMP support the Display Orientation SEI Message in h264 streams?

问题

I'm streaming video h264 video and AAC audio over RTMP on Android using the native MediaCodec APIs. Video and audio look great, however while the video is shot in potrait mode, playback on the web or with VLC is always in landscape.

Having read through the h264 spec, I see that this sort of extra metadata can be specified in Supplemental Enhancement Information (SEI), and I've gone about adding it to the raw h264 bit stream. My SEI NAL unit for this follows this rudimentary format, I plan to optimize later:

val displayOrientationSEI = {
    val prefix = byteArrayOf(0, 0, 0, 1)
    val nalHeader = byteArrayOf(6) // forbidden_zero_bit:0; nal_ref_idc:0; nal_unit_type:6 

    val display = byteArrayOf(47 /* Display orientation type*/, 3 /*payload size*/)

    val displayOrientationCancelFlag = "0" // u(1); Rotation information follows
    val horFlip = "1" // hor_flip; u(1); Flip horizontally
    val verFlip = "1" // ver_flip; u(1); Flip vertically
    val anticlockwiseRotation = "0100000000000000" // u(16); value / 2^16 -> 90 degrees
    val displayOrientationRepetitionPeriod = "010" // ue(v); Persistent till next video sequence
    val displayOrientationExtensionFlag = "0" // u(1); No other value is permitted by the spec atm
    val byteAlignment = "1"

    val bitString = displayOrientationCancelFlag +
            horFlip +
            verFlip +
            anticlockwiseRotation +
            displayOrientationRepetitionPeriod +
            displayOrientationExtensionFlag +
            byteAlignment

    prefix + nalHeader + display + BigInteger(bitString, 2).toByteArray()
}()

Using Jcodec's SEI class, I can see that my SEI message is parsed properly. I write out these packets to the RTMP stream using an Android JNI wrapper for LibRtmp.

Despite this, ffprobe does not show the orientation metadata, and the video when played remains in landscape.

At this point I think I'm missing a very small detail about how FLV headers work when the raw h264 units are written out by LibRtmp. I have tried appending this displayOrientationSEI NAL unit:

To the initial SPS and PPS configuration only.
To each raw h264 NAL units straight from the encoder.
To both.

What am I doing wrong? Going through the source of some RTMP libraries, like rtmp-rtsp-stream-client-java, it seems the SEI message is dropped when creating FLV tags.

Help is much, much appreciated.

回答1:

Does RTMP support the Display Orientation SEI Message in h264 streams?

RTMP is unaware of the very concept. from RTMPs perspective, the SEI is just a series of bytes it copys. It never looks at them, it never parses them.

The thing that needs to support it, is the h.264 decoder (which RTMP is also unaware of) and the player software. If it is not working for you, you must check the player, or the validity of the encoded SEI, not the transport.

来源：https://stackoverflow.com/questions/62089397/does-rtmp-support-the-display-orientation-sei-message-in-h264-streams

标签

android

video-streaming

h.264

rtmp