I need to be able to rotate a video on screen, so I created a custom TextureView which provides a convenience layer over a MediaPlayer similar to how the current implementat
You can see this article
SurfaceView and TextureView fill similar roles, but have very different implementations. To decide which is best requires an understanding of the trade-offs. Because TextureView is a proper citizen of the View hierarchy, it behaves like any other View, and can overlap or be overlapped by other elements. You can perform arbitrary transformations and retrieve the contents as a bitmap with simple API calls.
The main strike against TextureView is the performance of the composition step. With SurfaceView, the content is written to a separate layer that SurfaceFlinger composites, ideally with an overlay. With TextureView, the View composition is always performed with GLES, and updates to its contents may cause other View elements to redraw as well (e.g. if they're positioned on top of the TextureView). After the View rendering completes, the app UI layer must then be composited with other layers by SurfaceFlinger, so you're effectively compositing every visible pixel twice. For a full-screen video player, or any other application that is effectively just UI elements layered on top of video, SurfaceView offers much better performance.
Yes, it is expected with TextureView. TextureView causes the video to go through normal view-compositing for rendering, unlike SurfaceView which is composited directly in the GPU (the decoding pipeline is rendering directly to the area of the screen where you place the SurfaceView). While the TextureView rendering is hardware-accelerated, it still is going through more steps for the additional flexibility, and there is a definite performance hit. Additionally, any code running on the UI-thread may affect TextureView unlike SurfaceView.
Additional information: