Here at Datawire, we’ve been hacking on some cloud applications and tools on Kubernetes. As our development team added services we found there was often a need for instant on-demand and ephemeral Kubernetes clusters that run in the cloud. We need these in a variety of situations:
- As part of our CI workflow where we want to do a clean install and test of our services. The popular Minikube fails us here because you cannot run a VM inside of another container or VM.
- Quickly standup a specific version of our app to reproduce a problem but do it in the cloud rather than locally. Often so we can share it easily with other folks on the team.
- Hacking around for fun and profit!
We tried using Google Container Engine. Our developers use Google as their primary development environment, so we figured it would be a natural fit. However, we found that automatically destroying clusters when you had completed a job was not what GKE was designed to do and getting engineers to remember to do it was often a challenge not to mention sometimes automated test code fails and the cleanup does not happen.
To solve this problem we wrote a web service and accompanying command-line tool that hands out short-lived Kubernetes clusters. While handing out Kubernetes clusters is easy it is quite difficult to do it instantly, so we spent a bunch of time optimizing this flow. Behind the scenes we manage a pool of Kubernetes clusters and when you do a
kubernaut claim you are actually given the keys to a pre-allocated cluster. Every cluster is fresh and when you are done with the cluster we destroy the underlying machines so nothing is preserved between claims or leaked to other users.
We wrote Kubernaut for internal usage, but thought it might be interesting to other folks so I wanted to share it to see if anyone else might find it useful. I’m keeping the backing pool for Kubernaut small for now since this is an experimental project so if you receive a Kubernaut is at capacity type error then this is the reason.