How do I pass and retrieve memory stream from my Application to/from DLL?

Question

Suppose I have a TMemoryStream I need to pass to my DLL and get back TMemoryStream (Bitmap stream) from the DLL.

I was thinking my DLL would ha

Answer 1

The InBuff is easy (I think). I pass TMemoryStream.Memory and TMemoryStream.Size.

Yes.

Question is how do I allocate the OutBuff in the DLL, and the caller application can convert it back to TMemoryStream and later free that memory (By the caller application)?

Given the signature of the DLL function you showed, you would not allocate the memory in the DLL at all. The caller has to allocate it. The caller can call Process() once to get the needed size, then allocate it, then call Process() again to fill it. This way, the caller is responsible for both allocating and freeing the memory. For example:

procedure Process(InBuff: Pointer; InBuffSize: Integer; OutBuff: Pointer; var OutBuffSize: Integer); stdcall;
begin
  //...
  if (OutBuf <> nil) then
  begin
    // copy no more than OutBuffSize bytes into OutBuf, and then
    // update OutBuffSize with the number of bytes actually copied...
    Move(..., OutBuf^, ...);
    OutBuffSize := ...;
  end else begin
    // update OutBuffSize with the number of bytes needed for OutBuff...
    OutBuffSize := ...;
  end;
  //...
end;

var
  InStream: TMemoryStream;
  OutStream: TMemoryStream;
  BuffSize: Integer;
begin
  InStream := TMemoryStream.Create;
  try
    // fill InStream as needed...

    BuffSize := 0;
    Process(InStream.Memory, InStream.Size, nil, BuffSize);

    OutStream := TMemoryStream.Create;
    try
      OutStream.Size := BuffSize;
      Process(InStream.Memory, InStream.Size, OutStream.Memory, BuffSize);
      // use OutStream as needed...
    finally
      OutStream.Free;
    end;
  finally
    InStream.Free;
  end;
end;

If you actually want the DLL to allocate the memory, you have to change the signature of your DLL function to make OutBuff be a var parameter. You also have to export an additional function so the DLL can free the memory that the DLL allocated. The benefit of this approach is that the caller would only have to call Process() once, and the DLL can decide how it wants to allocate and free the memory. For example:

procedure Process(InBuff: Pointer; InBuffSize: Integer; var OutBuff: Pointer; var OutBuffSize: Integer); stdcall;
begin
  //...
  OutBuffSize := ...;
  GetMem(OutBuf, OutBuffSize);
  Move(..., OutBuf^, OutBuffSize);
  //...
end;

procedure FreeProcessBuff(InBuff: Pointer); stdcall;
begin
  FreeMem(InBuff);
end;

type
  TMemoryBufferStream = class(TCustomMemoryStream)
  public
    constructor Create(APtr: Pointer; ASize: NativeInt);
  end;

procedure TMemoryBufferStream.Create(APtr: Pointer; ASize: NativeInt);
begin
  inherited Create;
  SetPointer(APtr, ASize);
end;

...

var
  InStream: TMemoryStream;
  OutStream: TMemoryBufferStream;
  Buff: Pointer;
  BuffSize: Integer;
begin
  InStream := TMemoryStream.Create;
  try
    // fill InStream as needed...

    Buff := nil;
    BuffSize := 0;
    Process(InStream.Memory, InStream.Size, Buff, BuffSize);
    try
      OutStream := TMemoryBufferStream.Create(Buff, BuffSize);
      try
        // use OutStream as needed...
      finally
        OutStream.Free;
      end;
    finally
      FreeProcessBuff(Buff);
    end;
  finally
    InStream.Free;
  end;
end;