What's the practical point of running containers on VM?

To expand on my personal use-case, I don't want or need a whole stack of physical servers sitting around just for a test k8s environment. There are some things that you can only really test properly in a real multi-node environment rather than single-node solutions like minikube: failovers, shared storage, some networking particularities, etc.

But there are reasons to run containers on VMs in production too. Hypervisors and container orchestration tools solve very different problems. Depending on what problems you are trying to solve, it might be useful to leverage both.

Can you please elaborate a little bit more on the reason to run containers on VMs in production?

Here's a reason if your production platform is Kubernetes:

---

The default pod limit per node 110. This is by design and considered a reasonable upper limit for the kubelet to reliably monitor and manage everything without falling over into NotReady/PLEG status.

If your node has a ton of cpu and memory, then 110 pods will not come close to utilizing all the metal. You can go down the path of tuning and increasing the pod limit, but this is risky and often triggers other issues on components that are designed for more sane defaults.

It also means that if your node goes NotReady (not a hardware failure), it's now a much bigger deal because you have fewer total nodes and many more pods to re-schedule at once.

This is solved by splitting up these massive nodes into smaller nodes via virtualization.

It's also nice having an api-driven layer to manage and upgrade the vms versus shoehorning a bare-metal solution. I would argue it also encourages immutable infrastructure by making it much more accessible.

There are bare-metal solutions but it's often more complicated and slower than launching/destroying vms.

Not the person you replied to, but we do this.

The main reason being on premise deployments. Our workload is primarily machine learning models, as such they are well suited to docker containers due to good support and being able to quickly build images, deploy and tear down.

Additionally, whilst the task would be well suited to a Kubernetes deployment which is ideal in the cloud, on prem we're likely running on a single box running multiple VM's. Hence we use containers within the VM.

It isn't the perfect solution and definitely has it's downsides, but it is good enough and better than the alternatives when considering everything possible.

Lets say you have a complicated distributed application that needs to run on multiple cloud providers, and maybe it needs to run on-prem sometimes.

You could write your application multiple times, and publish multiple images for the same application across different infrastructures.

Or, just build your application once to run in containers, and run those containers on k8s everywhere.

This sounds like every talk given by Bryan Cantrill regarding public clouds running containers.