How to Solve Gen2 Heap Fragmentation

Question

I am running a C# application that services HTTP requests. I have noticed recently that it's taking up more memory then I expect. I grabbed some dumps, popped them in Windbg, and found that most of the memory was marked as Free:

!dumpheap -stat
...
00007ffde4783630   681599     65433504 System.Threading.Tasks.TaskFactory+CompleteOnInvokePromise
00007ffde47cc988   167885     76872908 System.Byte[]
00007ffde47c6948   521353     80352802 System.String
0000007e3a16c2d0  1870425   1415374334      Free

So the dump is ~3GB so about half of it is free memory. Looking at the heaps I see this:

!heapstat
Heap             Gen0         Gen1         Gen2          LOH
Heap0        82248472      7354560    987275056    178834656
Heap1        93146552      6382864    857470096    129435960
Total       175395024     13737424   1844745152    308270616

Free space:                                                 Percentage
Heap0        40969256       146456    640426720     54829792 SOH: 63% LOH: 30%
Heap1        75943736        94448    550812312     54825216 SOH: 65% LOH: 42%
Total       116912992       240904   1191239032    109655008

So the my small object heaps are very fragmented, specifically Gen2. On the server I can see that gen2 collections are happening (using performance counters) but even though they are the it looks like the gen2 heap is not being compacted. Even when there is only 1-2% of RAM available on the server the gen2 heap is not being compacted.

To me it looks like I am suffering this memory pressure because the heap is fragmented. However I can't figure out why the fragmentation is happening or why gen2 is unable to be compacted. Some of the free spaces are 6MB in size so I would think it would for sure compact those spaces away.

Can anyone give me some ideas on how to figure out why my heap is so fragmented? Am I even barking up the right tree here?

Any help will be greatly appreciated, thanks!

EDIT 1:

The breakdown of !gchandles is:

Handles:
Strong Handles:       4507
Pinned Handles:       58
Async Pinned Handles: 977
Ref Count Handles:    1
Weak Long Handles:    6087
Weak Short Handles:   724

Answer 1

The next step would be to use !gchandles and look for pinned handles. This might identify objects that are locked to a specific memory position, because some native code (e.g. C++) needs to access it. While garbage colelction may move objects around in memory, the C++ pointer will not be updated by .NET, so pinning is the only option.

Even when there is only 1-2% of RAM available on the server the gen2 heap is not being compacted.

You're talking about physical RAM here. From an operating system, I expexct that it uses the available resources as good as possible, so I expect the RAM to be 100% used at all the time. If it is "not used", I expect the OS to use it for caching. Due to this, the physical RAM usage cannot really be an indicator for garbage collection.

Also, I would not worry too much. If the memory is not used by .NET, it will be swapped to disk by the OS and thus free physical RAM for other purposes.