Render Bitmap frames to Surface for encoding

Question

My goal is to take in a M4V video file, decode a segment of the video as PNG frames, modify these frames, and re-encode the trimmed video (also to M4V).

The workflow

Answer 1

Note that original STextureRender renders external texture from SurfaceTexture. If you would like it to render your texture (created from bitmap) you have to make the following changes:

1. change target parameters GLES11Ext.GL_TEXTURE_EXTERNAL_OES to GLES20.GL_TEXTURE_2D

2. change this line in shader definition "uniform samplerExternalOES sTexture;\n" to “uniform sampler2D sTexture;\n”

3. remove st.getTransformMatrix(mSTMatrix); from drawFrame

This solution worked for me.

Answer 2

You can try Intel INDE Media Pack, it allows to modify frames, cut segments, join files and much more. The are several sample effects for frames modifications: colors modifications, text overlays an so on, and you can easily modify or add new effects. It has a nice samples set and tutorials how to build and run app: https://software.intel.com/en-us/articles/intel-inde-media-pack-for-android-tutorials-running-samples

Frames modifications are gl shaders based, like this, for example for Sepia:

@Override
protected String getFragmentShader() {
    return "#extension GL_OES_EGL_image_external : require\n" +
            "precision mediump float;\n" +
            "varying vec2 vTextureCoord;\n" +
            "uniform mat3 uWeightsMatrix;\n" +
            "uniform samplerExternalOES sTexture;\n" +
            "void main() {\n" +
            "  vec4 color = texture2D(sTexture, vTextureCoord);\n" +
            "  vec3 color_new = min(uWeightsMatrix * color.rgb, 1.0);\n" +
            "  gl_FragColor = vec4(color_new.rgb, color.a);\n" +
            "}\n";

}

where uWeightsMatrix is set to shader via getAttributeLocation and glUniformMatrix3fv

protected float[] getWeights() {
    return new float[]{
            805.0f / 2048.0f, 715.0f / 2048.0f, 557.0f / 2048.0f,
            1575.0f / 2048.0f, 1405.0f / 2048.0f, 1097.0f / 2048.0f,
            387.0f / 2048.0f, 344.0f / 2048.0f, 268.0f / 2048.0f
    };
}

Answer 3

I was able to render bitmap frames to a surface for encoding. I used MediaCodec + MediaMuxer to encode bitmap frames using the InputSurface and rendering the bitmaps using OpenGL.

Looks like all you are missing is open gl commands to render the texture

To fix this issue I also added some additional open gl commands to render the texture using a vertex shader.

See this project and util class for more details on rendering a texture using open gl https://github.com/rsri/Pic2Fro/blob/b4fe69b44343dab2515c3fd6e769f3370bf31312/app/src/main/java/com/pic2fro/pic2fro/util/Util.java

Calling the renderTexture(..) with the texture handle and appropriate width and height after GLUtils.texImage2D(..) in your snippet will fix all bitmap rendering issues.

See also my related answer here https://stackoverflow.com/a/49331295/7602598