Exploring Sealed Secrets in minikube

K8s-Sealed-Secrets

The idea behind writing this story is to be able to test and understand sealed secretes using minikube setup.

We all know default secrets in K8s are based on base64 encoding, which can be decoded easily, therefore we need to use a strong encryption for our secrets. Sealed secrets provides a mechanism to add another layer of security on top of base64 encoding, therefore making it much more secure.

Pre-requisite

Before we dive into sealed secrets, we need to setup some pre-requisites:

1. minikube
2. base64

Let’s get started.

Once you’ve installed minikube, let’s kick

minikube start --cpus 4 --memory 8192

Once minikube is up, you can explore the dashboard by running:

minikube dashboard

Before we create sealed secrets, we need to install the controller for managing the sealed secrets. We can do that by following command:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/controller.yaml

Once this is done, you would see the deployment for controller:

Along with controller pod, this would also deploy the certificate and private key, that can be used to seal/unseal sealed secrets:

You can access the certificate and key by navigating to secrets section under kube-system namespace

We’ll need the certificate in order to create a sealed secret so let’s bring it to our local filesystem:

mkdir -p /tmp/k8s
kubectl get secret -n kube-system | grep 'sealed-secrets-key' | awk '{print$1}' | xargs -I {} kubectl get secret/{} -n kube-system  -o yaml | grep tls.crt | awk -F: 'NR==1{print $2}' | base64 --decode >> /tmp/k8s/mycert.pem

This command basically gets a copy of certificate from local k8s cluster under kube-system namespace and saves it to mycert.pem file.

You can also access it from k8s dashboard and copy-paste it manually in the file as well :).

Let’s create the sealed secrets now.

/tmp/k8s/createSecrets.sh

This file first creates base64 secret and saves it in a file and then based on base64 secret file, generates file for creating sealed-secrete. This file expects secrets in a file in plain-text file (secrets.txt) in key value pair format, where key is secret name and value is the actual secret.

#!/usr/bin/env bash
# create secret directory
secretFile=$1
secretDir=/tmp/k8s/secrets
mkdir -p "$secretDir"
while
# shellcheck disable=SC2162
read  key value
do
   echo "$key" "$value"
   # Generate plain text file containing secrets
   printf "$value" > "$secretDir/$key.txt"
   kubectl delete secret "$key-secret"
   kubectl create secret generic "$key-secret" --from-file="$secretDir/$key.txt"
   kubectl get secret "$key-secret" -o yaml > "$secretDir/$key-secret-base64.yaml"
   kubeseal --scope cluster-wide --cert "mycert.pem" --format yaml < "$secretDir/$key-secret-base64.yaml" > "$secretDir/$key-secret-sealed-minikube.yaml"
done < "$secretFile"

/tmp/k8s/secrets.txt

dbsecretname k8s!sfun

Usage

cd /tmp/k8s
→  bash createSecrets.sh secrets.txt
dbsecretname k8s!sfun
secret "dbsecretname-secret" deleted
secret/dbsecretname-secret created

Once script execution is finished, we should see another directory under /tmp/k8s/ with name secrets

→  ls -lrt
total 24
-rw-r--r--  1 sharad.mishra  wheel     8  9 Mar 22:49 dbsecretname.txt
-rw-r--r--  1 sharad.mishra  wheel   557  9 Mar 22:49 dbsecretname-secret-base64.yaml
-rw-r--r--  1 sharad.mishra  wheel  1452  9 Mar 22:49 dbsecretname-secret-sealed-minikube.yaml
→ secrets (master) ✔
→  pwd

Here

dbsecretname.txt is the file which contains secret in plain-text.

k8s!sfun

dbsecretname-secret-base64.yaml file contains our secret in base64 encoding.

dbsecretname.txt: azhzIXNmdW4=

dbsecretname-secret-sealed-minikube.yaml file contains our secret encoded by sealed secrets.

dbsecretname.txt: AgCHY9Q4vugEu0b3Rhl1mqt88W0bMNCoP3hsk5eVLSh4l+kTTnmbt8YWQ2u/zPjlquUW58UIuUmqdWwrLXGoaalZpo+e70piPv90HuSA708mdrgT9rEl6xX6eJhaASMDKBUyxpaa2b9ufls8vMmbCwqX1h31rloOk
      mjSBk9vxLTYWLO8ALAb5GiaPwXVuJN3CGFypgNLvroBRz5gKaRJKN7EcI9xWPGGaKd0AFfD0QBuyuc1IW8u7VGo6Jkd8Rbyio07kQlakyT5ulL4fKH3ZvtIVu5RWWjli38aRp3p1dg8pcd0WfmgwcZ7NPVjglmYgGDcFx74C4mTbBHHuvcaKCKk      i7ftMxIE4JKAeayzXbe+PyNapUuTl5ZC+/fkQjXLvGFYq0aNVUCNtKt3wjMDB6FFnHjFcdkNqazGtrQuYKV4J/vakBpgSHh7drfSWC2OXuxL3Z+G41jj/Uxc5z39NdQ7G9R+V3G8xZAgxp1qr8IlfxZDFeLbSqtEpNdaGNyLpIHNIM/02uvB+Fy      ejxSz7du2Mm6QTdCRrGS6dRlWcspPcYOS3KyxslbQiE/UcMDMdiSwtBWzSAEIWv0WwXKODpNCawzt57W03lBwsPjzn+siQZl8/wuW4s1Fo4/nf9WlG97MImeUiYe5iqNzJe+A/YHgW04EDVlJbdV0H6cIRj7q/tJI5nFNGWT3CyyFoBrI6pIjT4      noGu5DWw==

Once our sealed secret is created we can install in our minikube and see if our controller is able to decrypt it successfully.

Before we install let’s rename our sealed-secret to a different name so that we can differentiate it with base64 secret:

sed 's/dbsecretname-secret/dbsecretname-secret-sealed/g' dbsecretname-secret-sealed-minikube.yaml > dbsecretname-secret-sealed-minikube-v1.yaml

To install sealed-secret, we can run following command:

→  kubectl apply -f dbsecretname-secret-sealed-minikube-v1.yaml

Once installed, we can see our secret in k8s dashboard under default namespace

We can see that our sealed-secret was successfully decrypted by controller:

We can also check the logs of controller pod and debug what’s going on:

I hope it helps.

Let me know in the comment section, if you encounter any issues or have any suggestions.

References:

1. https://github.com/imsharadmishra/sealed-secrets
2. https://github.com/bitnami-labs/sealed-secrets