Segmentation fault while avcodec_encode_video2

Question

I have some problems while trying to encode a AVFrame to a packet.

Before reading the whole code, the input stuff is working, I tested it. The output stuff is from an example here. I think there is the problem. But the segmentation fault occurs in the loop near the end.

Here is my reduced code:

void nmain() {

  // input stuff
  AVFormatContext *formatCtxIn=0;
  AVInputFormat *formatIn=0;
  AVCodecContext *codecCtxIn=0;
  AVCodec *codecIn;
  AVPacket *pktIn;

  av_register_all();
  avdevice_register_all();
  avcodec_register_all();


  formatIn = av_find_input_format("dshow");

  if(!formatIn)
    return;


  AVDictionary *avoption=0;
  av_dict_set(&avoption, "rtbufsize", "1000000000", NULL);

  if(avformat_open_input(&formatCtxIn, "video=Integrated Camera", formatIn, &avoption)!=0)
    return;

  if(avformat_find_stream_info(formatCtxIn, NULL)<0)
    return;

  codecCtxIn = formatCtxIn->streams[0]->codec;
  codecIn = avcodec_find_decoder(codecCtxIn->codec_id);

  if(avcodec_open2(codecCtxIn, codecIn, NULL)<0)
    return;


  // end input stuff  
//------------------------------------------------------------------------------
  // output stuff

  AVOutputFormat *formatOut=0;
  AVFormatContext *formatCtxOut=0;
  AVStream *streamOut=0;
  AVFrame *frame=0;
  AVCodec *codecOut=0;
  AVPacket *pktOut;

  const char *filename = "test.mpeg";

  formatOut = av_guess_format(NULL, filename, NULL);
  if(!formatOut)
    formatOut = av_guess_format("mpeg", NULL, NULL);
  if(!formatOut)
    return;

  formatCtxOut = avformat_alloc_context();
  if(!formatCtxOut)
    return;

  formatCtxOut->oformat = formatOut;

  sprintf(formatCtxOut->filename, "%s", filename);

  if(formatOut->video_codec != AV_CODEC_ID_NONE) {
    AVCodecContext *ctx;

    codecOut = avcodec_find_encoder(formatOut->video_codec);
    if(!codecOut)
      return;

    streamOut = avformat_new_stream(formatCtxOut, codecOut);
    if(!streamOut)
      return;

    ctx = streamOut->codec;

    ctx->bit_rate = 400000;
    ctx->width    = 352;
    ctx->height   = 288;
    ctx->time_base.den = 25;
    ctx->time_base.num = 1;
    ctx->gop_size      = 12;
    ctx->pix_fmt       = AV_PIX_FMT_YUV420P;

    if(ctx->codec_id == AV_CODEC_ID_MPEG2VIDEO)
      ctx->max_b_frames = 2;
    if(ctx->codec_id == AV_CODEC_ID_MPEG1VIDEO)
      ctx->mb_decision = 2;


    if(formatCtxOut->oformat->flags & AVFMT_GLOBALHEADER)
      ctx->flags |= CODEC_FLAG_GLOBAL_HEADER;
  }

  if(streamOut) {
    AVCodecContext *ctx;
    ctx = streamOut->codec;

    if(avcodec_open2(ctx, codecOut, NULL) < 0)
      return;
  }

  if(!(formatCtxOut->flags & AVFMT_NOFILE))
    if(avio_open(&formatCtxOut->pb, filename, AVIO_FLAG_WRITE) < 0)
      return;

  avformat_write_header(formatCtxOut, NULL);


  // doit

  pktIn = new AVPacket;
  pktOut = new AVPacket;
  av_init_packet(pktOut);
  pktOut->data = 0;

  frame = avcodec_alloc_frame();
  if(!frame)
    return;

  for(;;) {
    if(av_read_frame(formatCtxIn, pktIn) >= 0) {
      av_dup_packet(pktIn);

      int fff;
      if(avcodec_decode_video2(codecCtxIn, frame, &fff, pktIn) < 0)
        std::cout << "bad frame" << std::endl;

      if(!fff)
        return;  // ok

      static int counter=0;
      SaveFrame(frame, codecCtxIn->width, codecCtxIn->height, counter++);  // work fine

      // here a segmentation fault is occured.
      if(avcodec_encode_video2(streamOut->codec, pktOut, frame, &fff) < 0)
        std::cout << "bad frame" << std::endl;
    }
  }
}


// only for testing
// add to ensure frame is valid
void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) {
  FILE *pFile;
  char szFilename[32];
  int y;

  // Open file
  sprintf(szFilename, "frame%d.ppm", iFrame);
  pFile=fopen(szFilename, "wb");
  if(pFile==NULL)
      return;

  // Write header
  fprintf(pFile, "P6\n%d %d\n255\n", width, height);

  // Write pixel data
  for(y=0; y<height; y++)
      fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);

  // Close file
  fclose(pFile);
}

What am i doing wrong?

While debugging i didn't found any problems with the parameters. streamOut->codec is filled. pktOut is allocated and frame is filled with the picture encoded before. I think the problem is while creating the output codec but watching the example and looking to the doxypages it seems to be correct.

The trace route is from QT using msvc11 and framework 5.

I also tried to run with dr. memory and get this:

Error #26: UNADDRESSABLE ACCESS: reading 0x00000000-0x00000004 4 byte(s)
# 0 replace_memcpy                      [d:\derek\drmemory\withwiki\trunk\drmemory\replace.c:203]
# 1 avcodec-54.dll!ff_dct_quantize_c   +0xd463   (0x6a482364 <avcodec-54.dll+0x3c2364>)
# 2 avcodec-54.dll!avcodec_encode_video2+0xb7     (0x6a56a5b8 <avcodec-54.dll+0x4aa5b8>)
# 3 nmain                               [d:\prg\tests\recording system-qt\libav\recsys\main.cpp:610]
# 4 main                                [d:\prg\tests\recording system-qt\libav\recsys\main.cpp:182]
Note: @0:00:06.318 in thread 5312
Note: instruction: mov    (%edx) -> %ebx

It seems like the reading process while memcpy is going wrong.

---

Version:

I've forgot to mention the version of libav/ffmpeg i'm using:

libavutil      51. 76.100 / 51. 76.100
libavcodec     54. 67.100 / 54. 67.100
libavformat    54. 33.100 / 54. 33.100
libavdevice    54.  3.100 / 54.  3.100
libavfilter     3. 19.103 /  3. 19.103
libswscale      2.  1.101 /  2.  1.101
libswresample   0. 16.100 /  0. 16.100
libpostproc    52.  1.100 / 52.  1.100

---

Addendum:

Function SafeFrame is copied from tutorial 1.

Answer 1

Finally i solved my problem.

The problem is (apart from the documentation of libav) avpacket is not a (real) copy of the picture in the packet. it just points to the data of the packet. You have to make a copy, or better you have to let it libav do.

So first i created a new avframe for the output and a buffer on which the output avframe is pointing to.

AVFrame *outpic = avcodec_alloc_frame();
nbytes = avpicture_get_size(codecCtxOut->pix_fmt, codecCtxOut->width, codecCtxOut->height);
uint8_t* outbuffer = (uint8_t*)av_malloc(nbytes);

This buffer is used for the conversion from input to output. Then in the loop i have to fillup the outpic (avframe) with the buffer. I have found in the code that this function is filling up the plane pointers with the buffer. see here

avpicture_fill((AVPicture*)outpic, outbuffer, AV_PIX_FMT_YUV420P, codecCtxOut->width, codecCtxOut->height);

Then i converted the inpic to outpic using sws_scale. But first you have to setup the swscontext.

SwsContext* swsCtx_ = sws_getContext(codecCtxIn->width, codecCtxIn->height,
                                     codecCtxIn->pix_fmt,
                                     codecCtxOut->width, codecCtxOut->height,
                                     codecCtxOut->pix_fmt,
                                     SWS_BICUBIC, NULL, NULL, NULL);

sws_scale(swsCtx_, inpic->data, inpic->linesize, 0, codecCtxIn->height, outpic->data, outpic->linesize);

Then you can encode the outpic into pktout (avpacket for output). But first do free the output packet, otherwise you will get an error and a leak... see here

av_free_packet(pktOut);

if(avcodec_encode_video2(streamOut->codec, pktOut, outpic, &fff) < 0) {
  std::cout << "shit frame" << std::endl;
  continue;
}
// and write it to the file
formatOut->write_packet(formatCtxOut, pktOut);

So now it works (nearly fine) for me. Still a small memory leak, but this i can spot later.

Answer 2

I see at least two problems with that transcoding loop:

1) You're not checking whether the decoder produced a frame. Many decoders have a delay between input and output, so a decode call won't necessarily produce a frame even if no error occurs. You just have to keep passing packets to the decoder until it starts returning frames (and then flush it with NULL packets at the end, as described in the documentation).

The result is that you're passing an uninitialized frame to the encoder, which is probably the reason for the crash.

2) Another problem I see is that you're initing the ouput packet only once. As the documentation says

The user can supply an output buffer by setting avpkt->data and avpkt->size prior to calling the function, but if the size of the user-provided data is not large enough, encoding will fail. All other AVPacket fields will be reset by the encoder using av_init_packet(). If avpkt->data is NULL, the encoder will allocate it. The encoder will set avpkt->size to the size of the output packet. The returned data (if any) belongs to the caller, he is responsible for freeing it.

So if you only initialize it once before starting the transcode loop, on each iteration after the first it will contain the old data. The encoder will think you want to use that buffer for encoding and overwrite it. This will end in tears if you've already passed that packet to a muxer or something like that. So make sure to init packet data and size to NULL/0 before each encode call.