What I want to do is next: We need to protect the video content(encrypt it) and be able to play it.
Tricky part is that we do not want to load everything into memory
There is a way for Windows using file virtualization. The idea is to create a pseudofile those content is encrypted and can be decrypted on demand. E.g. with the help of BoxedApp SDK you create a virtual file that it is based on IStream using BoxedAppSDK_CreateVirtualFileBasedOnIStream. Then you pass the path of the virtual file to a player (an external application, or media player activex). When the player needs next part of decrypted data, this request passes to IStream, and its implementation decrypts data and returns it back. Thus entire file is not decrypted entirely at any moment. Also there is a large step-by-step tutorial.