bbrundert.com

With the recent announcement of VMware and Bitnami joining forces, I wanted to revisit the kubeapps project on Enterprise PKS earlier today. I followed the community documentation but ran into some smaller issues (see my GitHub comments here) that were coming up in the MongoDB deployment initially.

UPDATE: At first I thought you needed to enable privileged containers in PKS but actually you don’t have to do that! There was a typo in my configuration which led to an unknown flag for the MongoDB deployment. I used the flag „mongodb.securityContext.enable=false“ when deploying the Helm chart but it should have been „mongodb.securityContext.enabled=false“. Thanks to Andres from the Bitnami team for catching this! The instructions below have been updated!

Install Helm

Add the bitnami repo:

helm repo add bitnami https://charts.bitnami.com/bitnami

Add a „kubeapps“ namespace to deploy into

kubectl create namespace kubeapps

Add a Service Account to Tiller

vi rbac-config-tiller.yaml
---
apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: tiller
   namespace: kube-system
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
   name: tiller
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: cluster-admin
 subjects:
 kind: ServiceAccount
 name: tiller
 namespace: kube-system 
---
kubectl create -f rbac-config-tiller.yaml

Leverage newly created service account for Tiller:

helm init --service-account tiller

Create Service account for kubeapps-operator

kubectl create serviceaccount kubeapps-operator 

kubectl create clusterrolebinding kubeapps-operator \
--clusterrole=cluster-admin \
--serviceaccount=default:kubeapps-operator

kubectl get secret $(kubectl get serviceaccount kubeapps-operator -o jsonpath='{.secrets[].name}') -o jsonpath='{.data.token}' | base64 --decode

Copy the secret for use in the kubeapps dashboard later on.

Since NSX-T brings an out-of-the-box capability for exposing kubeapps to an external IP address, we can use LoadBalancer and skip the port-forwarding section of the documentation. Following what I found in another bug, I set some extra flags for disabling IPv6:

helm install --name kubeapps --namespace kubeapps bitnami/kubeapps \
--set frontend.service.type=LoadBalancer \
--set mongodb.securityContext.enabled=false \
--set mongodb.mongodbEnableIPv6=false

After a few minutes, the deployed services & deployments should be up and running:

Follow then part three of the instructions to access the dashboard.

I have installed VMware PKS in a variety of environments and I typically show something that helps Platform Operators running PKS to dive even deeper into the status of PKS components beyond the pks cli. One of the key lifecycle components in PKS is called BOSH. BOSH deploys Kubernetes masters and workers and performs a number of other tasks. So how do you get access to BOSH in the easiest way?

Step 1)

• Login to the Ops Manager VM via ssh: ssh ubuntu@your.opsmanager.com  

Step 2)

• Open Ops Manager and click on the BOSH Director tile: 
  
• Click on the „Credentials“ Tab and search for „BOSH Commandline Credentials“:
  
• You will see an output similar like this one:

  {"credential":"BOSH_CLIENT=ops_manager 
  BOSH_CLIENT_SECRET=ABCDEFGhijklmnopQRSTUVWxyz 
  BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate 
  BOSH_ENVIRONMENT=192.168.1.100 bosh "}

• Copy and paste that line and reformat it the following way:

  BOSH_CLIENT=ops_manager 
  BOSH_CLIENT_SECRET=ABCDEFGhijklmnopQRSTUVWxyz 
  BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate 
  BOSH_ENVIRONMENT=192.168.1.100

   

• Easiest way to get started every time is to make it part of your .bashrc configuration by doing the following:
  • edit your .bashrc and append the outputs from above like this:

  • export BOSH_CLIENT=ops_manager 
    export BOSH_CLIENT_SECRET=ABCDEFGhijklmnopQRSTUVWxyz 
    export BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate 
    export BOSH_ENVIRONMENT=192.168.1.100

  • logout and login again (or just run the export commands on the CLI manually once)

• Some example commands on how to interact with BOSH (and a nice cheat sheet at https://github.com/DennyZhang/cheatsheet-bosh-A4):

• bosh deployments
  PKS=$(bosh deployments | grep ^pivotal | awk '{print $1;}')
  bosh -d $PKS vms
  bosh -d $PKS instances
  bosh -d $PKS tasks
  bosh -d $PKS tasks -ar
  bosh -d $PKS task 724
  bosh -d $PKS task 724 --debug
  
  CLUSTER=$(bosh deployments | grep ^service-instance | awk '{print $1;}')
  bosh -d $CLUSTER vms
  bosh -d $CLUSTER vms --vitals
  bosh -d $CLUSTER tasks --recent=9
  bosh -d $CLUSTER task 2009 --debug
  bosh -d $CLUSTER ssh master/0
  bosh -d $CLUSTER ssh worker/0
  bosh -d $CLUSTER logs
  bosh -d $CLUSTER cloud-check

   

• Advanced users: you can also install the BOSH CLI on and admin VM and run from there:
  • Download from https://github.com/cloudfoundry/bosh-cli/releases
  • Copy the certificate from the Ops Manager VM (/var/tempest/workspaces/default/root_ca_certificate) to your admin VM and edit the .bashrc environment variables accordingly

Over the past months, I had multiple conversations on why you would want to virtualize containers or Kubernetes. The „containers are somewhat providing virtualization – why should I do it at the server level as well?“ myth has been around for some time now. Before I start addressing this, let me take a quick step back here. When I started my career roughly 10 years ago in datacenter operations, virtualization wasn’t mainstream in many environments. I learned a lot about operating physical machines before I got to work on virtual infrastructures at scale. I also worked with multiple vendors and used several „Lights Out Management“ solutions and their basic automation capabilities to get my hardware up and running. But it was always a „yes, it’s getting easier from now on“ moment when vSphere was ready for configuration. While I enjoyed working in operations, I was always happy to set something up without plugging cables in or working on a server in the datacenters. 

I have worked with customers that fully embraced virtualization and have been 100% virtualized for years. They have benefited so much from this move and were able to simplify so many of their operational tasks while doing this. Even if they chose a 1:1 mapping for few extremly demanding VMs to a given host, this was still the better option. Having a consistent infrastructure and operational framework outpaces the potential drawbacks or „virtualization overhead“ (another myth) if you look at the bigger picture. Even though I haven’t been working in operations for some time now, I still remember what it means to be called during the night or deal with spontaneous changes in plans/projects all the time. And businesses and therefore IT are only moving faster – automation, „software-defined“ and constant improvements should be part of everyone’s daily business in operations.

For me, this applies to all workloads – from your traditional legacy applications to modern application runtime frameworks such as Kubernetes or event-driven architectures that are leveraging Functions-as-a-Service capabilities. Most of them co-exist all the time and it’s not a one-or-the-other conversation but an AND conversation. Even highly demanding workloads such as core telco applications are put on virtual infrastructure these days, enabled by automation and Open Source API definitions. All of these can be operated on a consistent infrastructure layer with a consistent operational model. Infrastructure silos have been broken down over the past decade and VMware has invested a lot to make vSphere a platform for all workloads. So when someone mentions bare-metal these days all I can ask myself is „why would I ever want to go back“? I sometimes wonder if all the challenges that virtualization took away have simply been forgotten – it just ran too well.

So what are my personal reasons to run containers on a virtual infrastructure & vSphere in specific?

1. Agility, Independence & Abstraction: scale, repair, lifecycle & migrate underlying components independently from your workloads; if you ever worked in operations, this is daily business (datacenter move, new server vendor selected, major storage upgrades, … there are tons of reasons why this is still a thing)
2. Density: run multiple k8s clusters/tenants on same hardware cluster, avoid idle servers e.g. due to N+1 availability concepts
3. Availability and QoS: you can plan for failures without compromising density, you can even ensure SLOs/SLAs by enforcing policies (networking, storage, compute, memory) that will also be enforced during outages (NIOC, SIOC, Resource Pools, Reservations, Limits, …)
4. Performance: better-than-physical performance & resource management (core ESXi scheduling, DRS & vMotion, vGPUs, …)
5. Infrastructure as Code: automate all the things on an API-driven Software Defined Datacenter stack
6. Security & Isolation: yep, still a thing
7. Fun fact: even Google demoes K8s on vSphere as part of their „GKE on-prem“ offering 😉 

There has been a ton of material published around this topic recently (and some awesome foundational work by Michael Gasch incl. his KubeCon talk), I want to list a few of the public resources here:

• Why Choose VMware Virtualization for Kubernetes and Containers (Blogpost, January 2021)
• vSphere with Tanzu Supports 6.3 Times More Container Pods than Bare Metal (Blogpost, August 2021)
  • Full Study/Paper (PDF, August 2021)
• The Value of vSphere in a Kubernetes World (Blogpost)
• Containers on Virtual Machines or Bare-Metal? (Whitepaper) 
• Performance of Enterprise Web Applications in Docker Containers on VMware vSphere 6.5 (Blogpost and link to Whitepaper)
• VMs and Containers – Friends or Enemies (Slidedeck by Simone Morellato)
• VMworld 2018: The Value of Running Kubernetes on vSphere (video) (shout out to my friends Michael Gasch and Frank Denneman)

Introducing: #vK8s

So, no matter what your favorite Kubernetes framework is these days – I am convinced it runs best on a virtual infrastructure and of course even better on vSphere. Friends don’t let friends run Kubernetes on bare-metal. And what hashtag could summarize this better than something short and crips like #vK8s ? I liked this idea so much that I created some „RUN vK8s“ images (inspired by my colleagues Frank Denneman and Duncan Epping – guys, it’s been six years since RUN DRS!) that I want to share with all of you. You can find the repository on GitHub – feel free to use them whereever you like.