Moving Kubernetes to the mainstream

Many PaaS and IaaS providers are diving into Kubernetes and providing turnkey setups. They are betting on the Kubernetes API as a specification for deploying application workloads in their service. Examples include Google's Kubernetes engine, Azure Kubernetes Service, and last but certainly not least, Amazon Elastic Container Service for Kubernetes (Amazon EKS). These are great to get you started, but what about if/when you move steady workloads back on premises due to cost or security concerns?

Finally, beware of the limitations tied to PaaS solutions. If you use a PaaS Kubernetes management plane, you may be limited to using the PaaS provider's CNI plugin and their implementation may limit your options. As an example, if you are running Kubernetes on AWS, the networking implementation may require one virtual IP/pod, but you only get a limited number of virtual IPs per instance type. Subsequently, you might need to move up to a bigger, more expensive instance type to support more pods, even though you don't really need any more/better CPUs, network, or storage.