Using VLC imem to play an h264 video file from memory but receiving error “main stream error: cannot pre fill buffer”

Question

I have an h264 video file that is loaded into memory, and I attempt to play it with imem using the parameter "imem-cat=4" so that vlc will use an access module to demux the video, and vlc starts and receives my imem parameters successfully:

[0x7f38a0000e28] access_imem demux debug: Using get(0x404e1d), release(0x404e91), data(0x7fff5b4a9430), cookie("IMEM")

This category also means I don't have to provide DTS and PTS. The imem module for VLC isn't well documented, but I've found hints in several places e.g.

https://forum.videolan.org/viewtopic.php?t=111917

https://forum.videolan.org/viewtopic.php?f=32&t=93842

Play video using libVLC from memory in python

My imem-get function simply sets the buffer pointer to the video data on the first call, returning 0, on any further calls it returns 1 to indicate that there is no more data:

int MyImemGetCallback (void *data,
                   const char *cookie,
                   int64_t *dts,
                   int64_t *pts,
                   unsigned *flags,
                   size_t * bufferSize,
                   void ** buffer)
{

ImemData* imem = (ImemData*)data;
cookie = imem->cookieString;

if(imem == NULL || imem->allBuffered==true) //indicate all data has been get()ted
    return 1;

*buffer = (void*) imem->video;
bufferSize = (size_t*) &(imem->bytes);
imem->allBuffered=true;

return 0;
}

Unfortunately after the first call I receive the following errors:

[0x189cb18] main input debug: Creating an input for 'imem://'
[0x189cb18] main input debug: using timeshift granularity of 50 MiB, in path '/tmp'
[0x189cb18] main input debug: `imem://' gives access `imem' demux `' path `'
[0x189cb18] main input debug: creating demux: access='imem' demux='' location='' file='(null)'
[0x7f2808000e28] main demux debug: looking for access_demux module matching "imem": 20 candidates
[0x7f2808000e28] access_imem demux debug: Using get(0x404e1d), release(0x404e91), data(0x7ffe0da3b940), cookie("h264")
[0x7f2808000e28] main demux debug: no access_demux modules matched
[0x189cb18] main input debug: creating access 'imem' location='', path='(null)'
[0x7f2808001958] main access debug: looking for access module matching "imem": 25 candidates
[0x7f2808001958] access_imem access debug: Using get(0x404e1d), release(0x404e91), data(0x7ffe0da3b940), cookie("h264")
[0x7f2808001958] main access debug: using access module "access_imem"
[0x7f2808000e28] main stream debug: Using block method for AStream*
[0x7f2808000e28] main stream debug: starting pre-buffering
[0x7f2808000e28] main stream error: cannot pre fill buffer
[0x7f2808001958] main access debug: removing module "access_imem"
[0x189cb18] main input warning: cannot create a stream_t from access
[0x17d7298] main libvlc debug: removing all interfaces
[0x17d7298] main libvlc debug: exiting
[0x17d7298] main libvlc debug: no exit handler
[0x17d7298] main libvlc debug: removing stats

For some reason it seems vlc cannot access the video data, but the error messages aren't very helpful and normally refer to network streams rather than memory locations.

Has anyone successfully used imem in this way or have any ideas as to what the problem could be? The video plays in VLC perfectly from disk. Thanks for any help.

Edit

It looks like the item interface may not actually support playing in this way. However, libVLC provides libvlc_media_t and livblc_media_new_callbacks which may allow me achieve what I want. I'll report back if I get it working.

Answer 1

So I couldn't get Imem to work, however on the VLC forums I was pointed towards a new API available in version 3.0.0. I had to remove my current installs of vlc and libvlc-dev and add the VLC daily builds PPA to the Ubuntu install, then install those versions.

The API is libvlc_media_new_callbacks:

LIBVLC_API libvlc_media_t *     libvlc_media_new_callbacks     (libvlc_instance_t *instance, libvlc_media_open_cb open_cb, libvlc_media_read_cb read_cb, libvlc_media_seek_cb seek_cb, libvlc_media_close_cb close_cb, void *opaque)

You must implement each of those callbacks to give VLC access to the in-memory stream. Although the documentation states that implementing the seek() callback is unnecessary, I couldn't get h264 video to play without it.

The open() callback should pass a pointer to your video data, I recommend a container class so you can store the index of the last byte read along side it.

The read() callback is used to read len bytes into a buffer for which a pointer is passed. Here you can write len or less bytes to the buffer and return the number of bytes copied, block until you have some bytes ready, return 0 for EOF or -1 for error.

The seek() callback is used to set the byte index at which the next read() will take place.

Finally close() doesn't return anything and is used to tidy up.

Here is an example of a read() implementation:

class MemVideoData
{
public:
MemVideoData(char *data, int data_bytes) : video(data), bytes(data_bytes), lastByteIndex(0) {}   //init
~MemVideoData() {}
char* video;   //  pointer to video in memory
int bytes;
int lastByteIndex;
};

ssize_t memVideo_read(void *opaque, unsigned char* buf, size_t len)
{
//TODO: block if not end of stream but no bytes available

MemVideoData *mVid = (MemVideoData*) opaque;    //cast and give context
int bytesToCopy=0;
int bytesSoFar = mVid->lastByteIndex;
int bytesRemaining = mVid->bytes - mVid->lastByteIndex;

if(bytesRemaining >= len)   //at least as many bytes remaining as requested
{
    bytesToCopy = len;
}
else if (bytesRemaining < len)    //less that requested number of bytes remaining
{
    bytesToCopy = bytesRemaining;
}
else
{
    return 0;   // no bytes left to copy
}

char *start = mVid->video;
std::memcpy(buf,&start[mVid->lastByteIndex], bytesToCopy);    //copy bytes requested to buffer.
mVid->lastByteIndex = mVid->lastByteIndex + bytesToCopy;    //increment bytes read count

return bytesToCopy;
}

As requested here is an example of an Open callback:

int VideoPlayer::memVideo_open(void* opaque, void** datap, uint64_t* sizep)
{
   //cast opaque to our video state struct
   MemVideoData *mVid = static_cast<MemVideoData*> (opaque);    

   //TODO: get mutex on MemVideoData object pointed to by opaque
   *sizep = (uint64_t) mVid->bytesTotal;    //set stream length
   *datap = mVid;   /*point to entire object. Think this was intended to 
  point to the raw video data but we use the MemVideoData object in read() 
  and seek()*/

   mVid->lastByteReadIndex=0;

   return 0;
}

Answer 2

Take a look at my Qt example below, it works. Actually I didn't want to read the whole file and store it in the ram. So I implemented the MediaDescriptor because I will implement a decryption logic in it to read from encrypted files. Btw I used libvlc 3.0.6 x64 pre-built libraries, looks working very well.

MediaDescriptor.h

#pragma once

#include <QObject>
#include <fstream>

class MediaDescriptor : public QObject
{
    Q_OBJECT

public:
    MediaDescriptor(QString mediaFilePath);
    ~MediaDescriptor();

    bool tryOpen();
    uint64_t getMediaLength();
    uint64_t getMediaBytes(unsigned char *buffer, uint64_t length);
    void setSeek(uint64_t seek);

private:
    QString m_mediaFilePath;
    std::ifstream *m_mediaFile;
    uint64_t m_mediaLength;
    uint64_t m_seek;
};

MediaDescriptor.cpp

#include "MediaDescriptor.h"

MediaDescriptor::MediaDescriptor(QString mediaFilePath)
    : m_mediaFilePath(mediaFilePath), m_mediaFile(nullptr), m_mediaLength(0), m_seek(0)
{
}

MediaDescriptor::~MediaDescriptor()
{
    if (m_mediaFile)
    {
        m_mediaFile->close();
        delete m_mediaFile;
    }
}

bool MediaDescriptor::tryOpen()
{
    m_mediaFile = new std::ifstream(m_mediaFilePath.toStdString().c_str(), std::ios::binary | std::ios::ate);

    if (m_mediaFile->is_open())
    {
        m_mediaFile->seekg(0, m_mediaFile->end);
        m_mediaLength = m_mediaFile->tellg();
        return true;
    }

    delete m_mediaFile;
    return false;
}

uint64_t MediaDescriptor::getMediaLength()
{
    return m_mediaLength;
}

uint64_t MediaDescriptor::getMediaBytes(unsigned char *buffer, uint64_t length)
{
    // to do: decrytion logic
    if (m_mediaFile->is_open())
    {
        uint64_t len = length;
        if (m_seek + len > m_mediaLength)
            len = (size_t)(m_mediaLength - m_seek);

        if (len > 0)
        {
            m_mediaFile->seekg(m_seek);
            m_mediaFile->read((char*)&buffer[0], len);
            m_seek += len;
        }

        return len;
    }

    return -1;
}

void MediaDescriptor::setSeek(uint64_t seek)
{
    m_seek = seek;
}

VLCHelper.h

#pragma once

#include <QObject>
#include <QWidget>
#include <QTime>
#include <mutex>
#include "vlc/vlc.h"
#include "MediaDescriptor.h"

class VLCHelper : public QObject
{
    Q_OBJECT

public:
    ~VLCHelper();
    static VLCHelper& getInstance();

    int vlcMediaOpenCallback(void* opaque, void** datap, uint64_t* sizep);
    int vlcMediaReadCallback(void *opaque, unsigned char* buf, size_t len);
    int vlcMediaSeekCallback(void *opaque, uint64_t offset);
    void vlcMediaCloseCallback(void *opaque);

    void initMedia(MediaDescriptor &mediaDescriptor, QWidget *output = nullptr, int volume = 100, bool repeat = false);
    void destroyMedia();
    bool isMediaValid();

    Q_INVOKABLE void playPauseMedia(bool play);
    bool isMediaPlaying();
    Q_INVOKABLE void stopMedia();

    void setRepeatMedia(bool repeat);
    bool getRepeatMedia();

    void setMediaPosition(float position);
    float getMediaPosition();

    QTime getMediaTime();
    QTime getMediaTotalTime();

    void setMediaVolume(int volume);
    int getMediaVolume();

signals:
    void mediaEOFReached();
    void error(QString error);

private:
    VLCHelper();

    std::mutex m_callbackMutex;
    libvlc_instance_t *m_vlcInstance;
    libvlc_media_t *m_vlcMedia;
    libvlc_media_player_t *m_vlcMediaPlayer;
    bool m_repeat;
    bool m_stopRequested;
    MediaDescriptor *m_mediaDescriptor;
    QWidget *m_output;
};

VLCHelper.cpp

#include "VLCHelper.h"

#pragma region Callback Wrappers
extern "C" int vlcMediaOpenCallbackGateway(void* opaque, void** datap, uint64_t* sizep)
{
    return VLCHelper::getInstance().vlcMediaOpenCallback(opaque, datap, sizep);
}

extern "C" int vlcMediaReadCallbackGateway(void *opaque, unsigned char* buf, size_t len)
{
    return VLCHelper::getInstance().vlcMediaReadCallback(opaque, buf, len);
}

extern "C" int vlcMediaSeekCallbackGateway(void *opaque, uint64_t offset)
{
    return VLCHelper::getInstance().vlcMediaSeekCallback(opaque, offset);
}

extern "C" void vlcMediaCloseCallbackGateway(void *opaque)
{
    VLCHelper::getInstance().vlcMediaCloseCallback(opaque);
}
#pragma endregion

VLCHelper::VLCHelper()
    : QObject(nullptr),
    m_vlcInstance(nullptr),
    m_vlcMedia(nullptr),
    m_vlcMediaPlayer(nullptr),
    m_repeat(false),
    m_stopRequested(false)
{
}

VLCHelper::~VLCHelper()
{
}

VLCHelper& VLCHelper::getInstance()
{
    static VLCHelper ins;
    return ins;
}

void VLCHelper::initMedia(MediaDescriptor &mediaDescriptor, QWidget *output, int volume, bool repeat)
{
    m_mediaDescriptor = &mediaDescriptor;
    m_output = output;
    m_repeat = repeat;

    m_vlcInstance = libvlc_new(0, NULL);

    m_vlcMedia = libvlc_media_new_callbacks(
        m_vlcInstance,
        vlcMediaOpenCallbackGateway,
        vlcMediaReadCallbackGateway,
        vlcMediaSeekCallbackGateway,
        vlcMediaCloseCallbackGateway,
        0
    );

    m_vlcMediaPlayer = libvlc_media_player_new_from_media(m_vlcMedia);

#if defined(Q_OS_WIN)
    libvlc_media_player_set_hwnd(m_vlcMediaPlayer, output ? (void*)output->winId() : nullptr);
#elif defined(Q_OS_MAC)
    libvlc_media_player_set_nsobject(m_vlcMediaPlayer, output ? (void*)output->winId() : nullptr);
#elif defined(Q_OS_LINUX)
    libvlc_media_player_set_xwindow(m_vlcMediaPlayer, output ? (void*)output->winId() : nullptr);
#endif

    libvlc_audio_set_volume(m_vlcMediaPlayer, volume);

    m_mediaDescriptor->setSeek(0);
}

void VLCHelper::destroyMedia()
{
    if (!m_vlcInstance)
        return;

    if (m_vlcMediaPlayer)
    {
        libvlc_media_player_stop(m_vlcMediaPlayer);
        libvlc_media_player_release(m_vlcMediaPlayer);
        m_vlcMediaPlayer = nullptr;
    }

    libvlc_release(m_vlcInstance);
    m_vlcInstance = nullptr;
}

bool VLCHelper::isMediaValid()
{
    return m_vlcInstance && m_vlcMedia && m_vlcMediaPlayer;
}

void VLCHelper::playPauseMedia(bool play)
{
    m_stopRequested = false;

    if (isMediaValid())
        play ? libvlc_media_player_play(m_vlcMediaPlayer) : libvlc_media_player_pause(m_vlcMediaPlayer);
    else
        emit error("TO DO");
}

bool VLCHelper::isMediaPlaying()
{
    if (isMediaValid())
        return libvlc_media_player_is_playing(m_vlcMediaPlayer);

    return false;
}

void VLCHelper::stopMedia()
{
    m_stopRequested = true;

    if (isMediaValid())
        libvlc_media_player_stop(m_vlcMediaPlayer);
    else
        emit error("TO DO");
}

void VLCHelper::setRepeatMedia(bool repeat)
{
    m_repeat = repeat;
}

bool VLCHelper::getRepeatMedia()
{
    return m_repeat;
}

void VLCHelper::setMediaPosition(float position)
{
    if (isMediaValid())
        libvlc_media_player_set_position(m_vlcMediaPlayer, position);
    else
        emit error("TO DO");
}

float VLCHelper::getMediaPosition()
{
    if (isMediaValid())
        return libvlc_media_player_get_position(m_vlcMediaPlayer);
    else
        emit error("TO DO");

    return -1.0;
}

QTime VLCHelper::getMediaTime()
{
    if (isMediaValid())
        return QTime::fromMSecsSinceStartOfDay((int)libvlc_media_player_get_time(m_vlcMediaPlayer));
    else
        emit error("TO DO");

    return QTime();
}

QTime VLCHelper::getMediaTotalTime()
{
    if (isMediaValid())
        return QTime::fromMSecsSinceStartOfDay((int)libvlc_media_player_get_length(m_vlcMediaPlayer));
    else
        emit error("TO DO");

    return QTime();
}

void VLCHelper::setMediaVolume(int volume)
{
    if (isMediaValid())
        libvlc_audio_set_volume(m_vlcMediaPlayer, volume);
    else
        emit error("TO DO");
}

int VLCHelper::getMediaVolume()
{
    if (isMediaValid())
        return libvlc_audio_get_volume(m_vlcMediaPlayer);
    else
        emit error("TO DO");

    return -1;
}

int VLCHelper::vlcMediaOpenCallback(void* opaque, void** datap, uint64_t* sizep)
{
    std::lock_guard<std::mutex> lock(m_callbackMutex);

    // optional, if comment out libvlc will trigger the 'vlcMediaReadCallback' as more often but for less buffer length
    *sizep = m_mediaDescriptor->getMediaLength();

    return 0;
}

int VLCHelper::vlcMediaReadCallback(void *opaque, unsigned char* buf, size_t len)
{
    std::lock_guard<std::mutex> lock(m_callbackMutex);

    return m_mediaDescriptor->getMediaBytes(buf, len);
}

int VLCHelper::vlcMediaSeekCallback(void *opaque, uint64_t offset)
{
    std::lock_guard<std::mutex> lock(m_callbackMutex);

    // optional, but important for some media types which holds meta data end of the file, for example: .mp4
    m_mediaDescriptor->setSeek(offset);

    return 0;
}

void VLCHelper::vlcMediaCloseCallback(void *opaque)
{
    std::lock_guard<std::mutex> lock(m_callbackMutex);

    m_mediaDescriptor->setSeek(0);

    if (!m_stopRequested)
    {
        emit mediaEOFReached();
        QMetaObject::invokeMethod(&getInstance(), "stopMedia");

        if(m_repeat)
            QMetaObject::invokeMethod(&getInstance(), "playPauseMedia", Q_ARG(bool, true));
    }
}

And the usage in the player widget:

    VLCHelper::getInstance().initMedia(*m_mediaDescriptor, ui.frame_video);
    connect(&VLCHelper::getInstance(), SIGNAL(mediaEOFReached()), this, SLOT(vlcMediaEOFReached()));
    connect(&VLCHelper::getInstance(), SIGNAL(error(QString)), this, SLOT(vlcError(QString)));

...

void PlayerWidget::on_pushButton_media_play_pause_clicked()
{
      VLCHelper::getInstance().playPauseMedia(!VLCHelper::getInstance().isMediaPlaying());
}

void PlayerWidget::on_pushButton_media_stop_clicked()
{
    VLCHelper::getInstance().stopMedia();
}

void PlayerWidget::timer_timeout()
{
    bool isValid = VLCHelper::getInstance().isMediaValid();

    if (isValid)
    {
        if (VLCHelper::getInstance().isMediaPlaying())
        {
            // update media position
            ui.horizontalSlider_media_position->blockSignals(true);
            ui.horizontalSlider_media_position->setValue((int)(VLCHelper::getInstance().getMediaPosition() * 1000.0f));
            ui.horizontalSlider_media_position->blockSignals(false);

            // update media volume
            ui.horizontalSlider_media_volume->blockSignals(true);
            ui.horizontalSlider_media_volume->setValue(VLCHelper::getInstance().getMediaVolume());
            ui.horizontalSlider_media_volume->blockSignals(false);

            // update media time
            ui.label_media_time->setText(VLCHelper::getInstance().getMediaTime().toString());

            // update media total time
            ui.label_media_time_total->setText(VLCHelper::getInstance().getMediaTotalTime().toString());
        }
    }

    ui.horizontalSlider_media_volume->setEnabled(isValid);
    ui.pushButton_media_stop->setEnabled(isValid);
    ui.pushButton_media_repeat->setEnabled(isValid);
}