I am creating a distributed database application in Golang. The application is running inside a K8 pod with a memory limit of 10GB. I am tracking the Max RSS size of the process and Go Heap Memory stats within a separate go routine. As I increase the load the process gets killed by OOM. Before OOM happens, I see Max RSS hits the 10 GB, but Go Heap Memory goes up and down but never goes beyond 8GB. Questions.
- How do I determine that if there a memory leak?
- The account of permanent data structures (like map, global data structures, etc) usage shouldnt not go beyond 2 GB according to calculations. Where is all the memory going?
- How can I avoid OOM kills for my process?