1.2GB memory exception

I read about memory limit

I have an application which works with huge images which needs to be streamed. Like in a video processing with single frames. The application has about 40 plugins, each of them can contain database, image processing and WPF GUI.

The application also has 2 plugins which uses older DotNet Winforms.

All works well except the application goes over about 1.2GB in RAM. Then on unusual locations in the plugins where new memory is allocated I receive the "Out of Memory exception".

I am working on a 64Bit system compiled as 32Bit. I have no more idea what to do and how to search for any fault.

Is there a limit or can I catch them?

Answers


It is very difficult to write a 32-bit program that consumes all of the available virtual memory space. You'll hit the wall well below 2 gigabytes, what you run out of first is a chunk of virtual memory that's large enough to fit the requested size. You can only get up to the 2GB limit by making small allocations, small enough to fit in the holes.

That wall hits early in a program that manipulates bitmaps. They can consume a big chunk of VM to store the bitmap pixels and it needs to be a contiguous allocation. They are stored in an array, not a tree. It's an unmanaged memory allocation, typical .NET memory profilers tend to be a bit helpless to show you the problem.

There isn't anything reasonable you can do about address space fragmentation, the notion that consuming all available VM should be possible is just wrong. You can get more breathing space on a 64-bit operating system by running editbin.exe in a post build event and use its /LARGEADDRESSAWARE command line option. That allows the process to use the available 4 gigabytes of VM, an option that's specific to the 64-bit version of Windows and possible because Windows doesn't need the upper 2GB. And of course, changing the platform target to AnyCPU is a quick and easy way to get gobs of virtual memory.


A 32Bit application running on windows (even if the OS is 64Bit) has a 4Gb Address space, but this is split into 2Gb Application/2Gb System (this can be changed to 3/1 with a different startup switch).

It is quite likely that the total memory you are using is actually 2Gb rather than 1.2Gb, how are you determining this 1.2Gb figure, Have you looked at the application using the process explorer tool?

If you change your application to ANYCPU or 64Bit you should find that this limitation disappears (well moves to a massively larger value) on a 64Bit OS.


To become more deterministic you should write some integration tests to check where your memory does end up. You can do it now with WMemoryProfiler. I would first load the images in 1500x1500 size, clean everything up and then mark all objects as known. Then I would relod the big images and check which new objects were allocated and have a sharp look at how many of them are there and who does own them.

You say that there are many external modules used. Perhaps you should drop some of them due to unwise usage of memory and replace them with something better. Now you can check.

If you are reaching the limit you still can unload some images and load them on demand if you and your plugins do support lazy structures such as IEnumerable<Image> where you as provider can decide when to load an images and how long to keep it in a cache until you get rid of the reference to help freeing up some memory.

[Test]
public void InstanceTracking()
{
   using (var dumper = new MemoryDumper())  // if you have problems use to see the debugger windows true,true))
   {
      TestWith1500x1500();
      dumper.MarkCurrentObjects();
      TestWith3000x3000();
      ILookup<Type, object> newObjects = dumper.GetNewObjects()
                                               .ToLookup( x => x.GetType() );

      // here we do find out which objects are holding most of the memory
      MemoryStatistics statOld = dumper.GetMemoryStatistics();
      foreach (var typeInfo in statOld.ManagedHeapStats
                                   .OrderByDescending(x => x.Value.Count))
      {
            Console.WriteLine("Type {0} has {1} instances of total size {2:N0} bytes", 
                           typeInfo.Key, 
                           typeInfo.Value.Count,
                           typeInfo.Value.TotalSize);
      }

      // then check with the info from above who is holding the most interesting new objects. 
      Console.WriteLine("New Strings:"); // just an example perhaps you should have a look at the images.
      foreach (var newStr in newObjects[typeof(string)] )
      {
          Console.WriteLine("Str: {0}", newStr);
      }
   }
}

Need Your Help

Example to understand scipy basin hopping optimization function

python scipy mathematical-optimization

I came across the basin hopping algorithm in scipy and created a simple problem to understand how to use it but it doesnt seem to be working correctly for that problem. May be I'm doing something

Can I serialize an ExpandoObject in .NET 4?

c# .net serialization dynamic expandoobject

I'm trying to use a System.Dynamic.ExpandoObject so I can dynamically create properties at runtime. Later, I need to pass an instance of this object and the mechanism used requires serialization.