NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.
I don't see how GOMAXPROCS alone can help here though. You would have to use Topology Manager (single-node policy) to avoid cross-NUMA allocations. This is in addition to other managers - Memory and CPU Manager.
CPU Pinning (via CPU Manager's Static Policy) will also be required to ensure your processes don't just get a CFS quota/share but are actually pinned onto specific CPU cores.
Intel suffers just as much when NUMA enters the picture, even prior to CCD style architecture. That extra latency hop across to the other core to get at memory is absolutely crippling, especially in a hot loop. It requires very careful handling, while being this kind of invisible element (unless you know to look for it, nothing will draw your attention to it)
Heck, we saw crazy performance degradation with redis when its memory usage exceeded a single NUMA block. Not much to be done about that at the k8s level when redis is single-threaded. Have to be super conscious of the underlying hardware at that point.
In that case I run one redis instance per NUMA domain. On my home server I essentially split machine in two and treat it as two distinct machines. PCIe devices attached to a proper domain etc.
Some lessons are never lost it seems, back when Windows NT was recent, we had to lock threads/processes to specific CPUs on SMP machines (affinity), exactly for similar reasons.
