问题
I'm constantly hitting an OutOfMemoryException inside a method that creates and processes some byte arrays. The code looks like this:
- Create MemoryStream to get some data (about 60MB).
- Create byte array (same size as MemoryStream, about 60MB)
- Fill the array with bytes from memory stream
- Close MemoryStream
- Process data from byte array
- Leave method
When this method is called like 20-30 times I get OutOfMemoryException right where the byte array is allocated. But I don't think it's the system memory issue. Application memory usage is around 500MB (private working set) and the test machine is 64-bit with 4GB of RAM.
Is it possible that memory used by the byte array or MemoryStream is not released after method finishes? But then, it does not look like this memory is allocated for the process as private working set is only 500MB or so.
What may cause the OutOfMemoryException when creating large byte array (60MB) beside physical memory shortage?
[Edited to add code sample] Source comes from PdfSharp lib
Exception is thrown at line byte[] imageBits = new byte[streamLength]; It indeed looks like LOH fragmentation issue.
/// <summary>
/// Reads images that are returned from GDI+ without color palette.
/// </summary>
/// <param name="components">4 (32bpp RGB), 3 (24bpp RGB, 32bpp ARGB)</param>
/// <param name="bits">8</param>
/// <param name="hasAlpha">true (ARGB), false (RGB)</param>
private void ReadTrueColorMemoryBitmap(int components, int bits, bool hasAlpha)
{
int pdfVersion = Owner.Version;
MemoryStream memory = new MemoryStream();
image.gdiImage.Save(memory, ImageFormat.Bmp);
int streamLength = (int)memory.Length;
if (streamLength > 0)
{
byte[] imageBits = new byte[streamLength];
memory.Seek(0, SeekOrigin.Begin);
memory.Read(imageBits, 0, streamLength);
memory.Close();
int height = image.PixelHeight;
int width = image.PixelWidth;
if (ReadWord(imageBits, 0) != 0x4d42 || // "BM"
ReadDWord(imageBits, 2) != streamLength ||
ReadDWord(imageBits, 14) != 40 || // sizeof BITMAPINFOHEADER
ReadDWord(imageBits, 18) != width ||
ReadDWord(imageBits, 22) != height)
{
throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format");
}
if (ReadWord(imageBits, 26) != 1 ||
(!hasAlpha && ReadWord(imageBits, 28) != components * bits ||
hasAlpha && ReadWord(imageBits, 28) != (components + 1) * bits) ||
ReadDWord(imageBits, 30) != 0)
{
throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format #2");
}
int nFileOffset = ReadDWord(imageBits, 10);
int logicalComponents = components;
if (components == 4)
logicalComponents = 3;
byte[] imageData = new byte[components * width * height];
bool hasMask = false;
bool hasAlphaMask = false;
byte[] alphaMask = hasAlpha ? new byte[width * height] : null;
MonochromeMask mask = hasAlpha ?
new MonochromeMask(width, height) : null;
int nOffsetRead = 0;
if (logicalComponents == 3)
{
for (int y = 0; y < height; ++y)
{
int nOffsetWrite = 3 * (height - 1 - y) * width;
int nOffsetWriteAlpha = 0;
if (hasAlpha)
{
mask.StartLine(y);
nOffsetWriteAlpha = (height - 1 - y) * width;
}
for (int x = 0; x < width; ++x)
{
imageData[nOffsetWrite] = imageBits[nFileOffset + nOffsetRead + 2];
imageData[nOffsetWrite + 1] = imageBits[nFileOffset + nOffsetRead + 1];
imageData[nOffsetWrite + 2] = imageBits[nFileOffset + nOffsetRead];
if (hasAlpha)
{
mask.AddPel(imageBits[nFileOffset + nOffsetRead + 3]);
alphaMask[nOffsetWriteAlpha] = imageBits[nFileOffset + nOffsetRead + 3];
if (!hasMask || !hasAlphaMask)
{
if (imageBits[nFileOffset + nOffsetRead + 3] != 255)
{
hasMask = true;
if (imageBits[nFileOffset + nOffsetRead + 3] != 0)
hasAlphaMask = true;
}
}
++nOffsetWriteAlpha;
}
nOffsetRead += hasAlpha ? 4 : components;
nOffsetWrite += 3;
}
nOffsetRead = 4 * ((nOffsetRead + 3) / 4); // Align to 32 bit boundary
}
}
else if (components == 1)
{
// Grayscale
throw new NotImplementedException("Image format not supported (grayscales).");
}
FlateDecode fd = new FlateDecode();
if (hasMask)
{
// monochrome mask is either sufficient or
// provided for compatibility with older reader versions
byte[] maskDataCompressed = fd.Encode(mask.MaskData);
PdfDictionary pdfMask = new PdfDictionary(document);
pdfMask.Elements.SetName(Keys.Type, "/XObject");
pdfMask.Elements.SetName(Keys.Subtype, "/Image");
Owner.irefTable.Add(pdfMask);
pdfMask.Stream = new PdfStream(maskDataCompressed, pdfMask);
pdfMask.Elements[Keys.Length] = new PdfInteger(maskDataCompressed.Length);
pdfMask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
pdfMask.Elements[Keys.Width] = new PdfInteger(width);
pdfMask.Elements[Keys.Height] = new PdfInteger(height);
pdfMask.Elements[Keys.BitsPerComponent] = new PdfInteger(1);
pdfMask.Elements[Keys.ImageMask] = new PdfBoolean(true);
Elements[Keys.Mask] = pdfMask.Reference;
}
if (hasMask && hasAlphaMask && pdfVersion >= 14)
{
// The image provides an alpha mask (requires Arcrobat 5.0 or higher)
byte[] alphaMaskCompressed = fd.Encode(alphaMask);
PdfDictionary smask = new PdfDictionary(document);
smask.Elements.SetName(Keys.Type, "/XObject");
smask.Elements.SetName(Keys.Subtype, "/Image");
Owner.irefTable.Add(smask);
smask.Stream = new PdfStream(alphaMaskCompressed, smask);
smask.Elements[Keys.Length] = new PdfInteger(alphaMaskCompressed.Length);
smask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
smask.Elements[Keys.Width] = new PdfInteger(width);
smask.Elements[Keys.Height] = new PdfInteger(height);
smask.Elements[Keys.BitsPerComponent] = new PdfInteger(8);
smask.Elements[Keys.ColorSpace] = new PdfName("/DeviceGray");
Elements[Keys.SMask] = smask.Reference;
}
byte[] imageDataCompressed = fd.Encode(imageData);
Stream = new PdfStream(imageDataCompressed, this);
Elements[Keys.Length] = new PdfInteger(imageDataCompressed.Length);
Elements[Keys.Filter] = new PdfName("/FlateDecode");
Elements[Keys.Width] = new PdfInteger(width);
Elements[Keys.Height] = new PdfInteger(height);
Elements[Keys.BitsPerComponent] = new PdfInteger(8);
// TODO: CMYK
Elements[Keys.ColorSpace] = new PdfName("/DeviceRGB");
if (image.Interpolate)
Elements[Keys.Interpolate] = PdfBoolean.True;
}
}
回答1:
I hope you are using MemoryStream.GetBuffer() and that you are not copying to a new array.
Your main problem is not a direct lack of memory but fragmentation of the LOH. That can be a tricky problem, the main issue is allocating large buffers of varying size. Items on the LOH are collected but not compacted.
Solutions might be:
- first make sure you are not blocking anything from being collected. use a profiler.
- try to reuse your buffer(s).
- round allocations up to a set of fixed numbers.
The last 2 both require you to work with oversized arrays, may take some work.
回答2:
Disposing has been suggested, but only for the MemoryStream, this will do you nothing. Also, GC.Collect has been suggested. This may not help, as it seems your memory is not substantially growing. Be careful, calling GC.Collect can be an expensive operation.
Fragmentation
It really looks like you're hitting the infamous Large Object Heap fragmentation issue. This may well be caused by often allocating and freeing chunks of 60MB of memory. If the LOH gets fragmented, it stays fragmented. This is a major issue with long-running .NET applications and a reason why ASP.NET is often configured to restart at intervals.
- Large object fragmentation (Stackoverflow)
- Excellent explanation and easy to follow code on Codeproject.
- Reported to Microsoft, but nothing will be done yet.
Prevent OutOfMemoryException
See the above CodeProject article on how to do this. The trick is, use MemoryFailPoint and catch the InsufficientMemoryException. This way, you can gracefully degrade and your application will not become unstable.
Possible general solution
Make sure your large objects live as long as possible. Reuse the buffer. Allocate it once with sufficient size and zero the buffer when you need it again. This way you won't run into any other memory issues. When your objects stay below 85k in size, they generally don't run into the LOH and won't clutter.
64 bit machines should not have this issue
EDIT: According to this post (workaround tab) and this post (see open comment), this issue should not appear on 64 bit machines. Since you say you run your code on 64 bit machines, perhaps you compiled with the configuration set to x86?
回答3:
Your Heap memory is throwing this exception, try calling GC.Collect() at the end to free up resources. You can also MemoryProfiler to find out the heap memory usage, it comes with 14 days trial
回答4:
Try enclosing it in a using(MemoryStream x = ...) { } block, which will dispose the object for you.
Although Close is supposed to Dispose the object, according to .NET guidelines, perhaps it's different in MemoryStream.
回答5:
I have same problem in code like below:
ImageData = (byte[])pDataReader.qDataTable.Rows[0][11];
if (ImageData != null)
{
ms = new MemoryStream(ImageData);
button1.BackgroundImage = Image.FromStream(ms);
}
ImageData = null;
ImageData = (byte[])pDataReader.qDataTable.Rows[0][12];
if (ImageData != null)
{
ms = new MemoryStream(ImageData);
button1.BackgroundImage = Image.FromStream(ms);
}
ImageData = null;
ms.Close();
Removing ms.Close(); solves the problem.
I think the problem rise because you define MemoryStream memory = new MemoryStream(); outside of if block and close it in the if block, same as me!
回答6:
First of all, it is not recommended that you read/write large FILESTREAM data through TSQL. The recommeded approach is by using the Win32/DOTNET APIs provided by SQL server. The code that I posted above shows how to access the FILESTREAM data using the SqlFileStream() .net class. It also shows how to send the data in smaller chunks.
Providing that your total memory is sufficient, you can prevent Out of memory exceptions resulting from LOH fragmentation by creating a bunch of smaller arrays, and wrapping them in a single IList, or some other indexed interface.
来源：https://stackoverflow.com/questions/9889925/outofmemoryexception-when-creating-multiple-byte-arrays