Logo

Armand.nz

Home / About / Linkedin / Github

Getting Started with Sealed Secrets

#sealed-secrets #kubernetes #devops |

The importance of securely managing secrets within you Kubernetes infrastructure cannot be overstated. Sealed-secrets are a handy tool that simplifies secret management in Kubernetes clusters and eliminates the risk of commit secrets in a code repo . This blog will walk you through the key considerations, the process of bringing your own certificates, installation using Helm, and how to seal your first secret.

Nowadays, we often use GitOps for application deployment, putting all the application’s information and configuration on Git. But can we do the same with Kubernetes Secrets YAML files on Git? The answer is a definite NO. The Secret’s file contains base64 encoded values of sensitive information like passwords, keys, certificates, tokens, etc., which should never be exposed in a repository.

For example, this is a sample YAML file for a Kubernetes Secret:

apiVersion: v1  
kind: Secret  
metadata:  
  name: mysecret  
type: Opaque  
data:  
  username: YXJtYW5k
  password: YXNpbGx5cGFzc3dvcmQ=

We can easily decode the sensitive information elements username and password(!) using a base64 decode command like so:

# echo -n YXNpbGx5cGFzc3dvcmQ= | base64 -d

Sealed Secrets

Sealed Secrets is a fantastic open-source tool from Bitnami that combines a Kubernetes controller and a client-side CLI to tackle the tricky issue of “storing secrets in Git.” Using asymmetric cryptographic encryption, it offers a practical solution. When paired with RBAC configurations to prevent non-admins from accessing secrets, Sealed Secrets effectively addresses this challenge. It’s a reliable and community-friendly approach to managing secrets securely.

How it works

Image source: Civo: - Sealed Secrets: Securely Storing Kubernetes Secrets in Git

Here’s how it works:

  1. Start by encrypting the secret on your machine using a public key and kubeseal CLI. This step encodes the encrypted secret into a Kubernetes Custom Resource Definition (CRD), “sealed-secret”
  2. Deploy the sealed-secret CRD to your target cluster.
  3. The sealed-secrets-controller on the target cluster will then decrypt the secret using a private key, transforming it into a standard Kubernetes secret.

The private key is only accessible to the Sealed Secrets controller within the cluster, while the public key is available for developers. This setup ensures your secrets, and only the cluster can decrypt the secrets, keeping them secure while developers handle the encryption.

This approach secures your secrets and streamlines the deployment process, making it easier to manage sensitive information across your Kubernetes environments. Securing your secrets is crucial, and using tools like Sealed Secrets can make this task much more manageable. Happy encrypting!

Considerations

When diving into sealed-secrets, here are some important considerations to keep in mind:

Namespace

If you run a sealed secrets controller in a namespace other than kube-system, you must pass the --controller-namespace flag every time you run the kubeseal commands or alternatively, you can set the environment variable SEALED_SECRETS_CONTROLLER_NAMESPACE.

Deploying Sealed Secrets outside the kube-system namespace can enhance security by isolating it from critical cluster components and offer more granular control through namespace-specific configurations and RBAC policies. Despite this, for my homelab use case i simply kept it in the kube-system namespace

sealed-secrets-controller

The helm chart by default installs the controller with the name sealed-secrets, while the kubeseal command line interface (CLI) tries to access the controller with the name sealed-secrets-controller. You can explicitly pass --controller-name to the CLI:

kubeseal --controller-name sealed-secrets <args>

Alternatively, you can set fullnameOverride when installing the chart to customize the name. Keep in mind that kubeseal defaults to assuming the controller is installed in the kube-system namespace. If you want to use the kubeseal CLI without needing to specify the controller name and namespace each time, you should install the Helm Chart like this:

helm install sealed-secrets \
	-n kube-system \
	--set-string fullnameOverride=sealed-secrets-controller \
	sealed-secrets/sealed-secrets

Regularly back up all sealed-secrets keys or re-encrypt secrets

The key certificate (public key portion) is essential for sealing secrets and must be accessible wherever kubeseal is used. While the certificate isn’t confidential, you must ensure you use the correct one.

By default, sealed-secret certificates renew automatically, which creates a new sealing key and corresponding certificate (default period: 30 days), and kubeseal uses the latest key to encrypt new secrets. However, sealed secrets aren’t rotated automatically, and old keys aren’t deleted when new keys are generated. This means old sealed secrets can still be decrypted.

This setup works well for a single Kubernetes cluster. But imagine if your cluster becomes unreachable and your secrets are stored in a git repository encrypted with kubeseal. Without access to the plain secrets, you face a significant challenge: recreate all secrets from scratch in your new cluster and update your deployments.

Although this represents a worst-case scenario, it’s essential to be prepared for it. Being proactive about these potential pitfalls can save you a lot of hassle in the long run.

Solution 1 - regularly back up your sealed-secrets keys

It’s important to regularly back up all your sealed-secrets keys and store them securely. This way, you can easily restore these keys in a new cluster when needed. To help you with this, here’s a helpful command to export all active sealed secret keys in the kube-system namespace.

$ kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml | kubectl neat > allsealkeys.yml

Solution 2 - Disable automatic key renewal (insecure)

While this method is somewhat less secure, you can adjust the secret rotation interval when creating the sealed-secrets controller. The value field is defined using golang duration flag (default: 720h30m). A value of 0 will disable automatic key renewal:

Solution 3 - re-encrypt secrets

When sealed-secrets generate new certificates, it’s so important to re-encrypt your existing secrets. This ensures that your secrets in the Git repository are protected with the latest keys. Don’t forget to back up the most recent key. For more details, check out the [Sealed-Secrets re-encryption guide]( https://github.com/bitnami-labs/sealed-secrets#re-encryption-advanced.

Before you discard any old sealing keys, take a moment to re-encrypt your SealedSecrets with the latest private key:

kubeseal --re-encrypt <my_sealed_secret.json >tmp.json \
  && mv tmp.json my_sealed_secret.json

The command above creates a newly sealed secret file, freshly encrypted with the latest key, ensuring the secrets stay secure within the cluster and don’t reach the client. You can then safely store this file in your version control system. Just a heads-up: kubeseal --re-encrypt won’t update the in-cluster object.

Currently, old keys are not garbage collected automatically.

It’s a good idea to periodically re-encrypt your SealedSecrets. But as mentioned above, don’t lull yourself in a false sense of security: you must assume the old version of the SealedSecret resource (the one encrypted with a key you think of as dead) is still potentially around and accessible to attackers. I.e. re-encryption is not a substitute for periodically rotating your actual secrets.

Bring your own certificates

The controller generates its own certificates when is deployed for the first time, it also manages the renewal for you. But you can also bring your own certificates so the controller can consume them as well.

The controller consumes certificates contained in any secret labeled with sealedsecrets.bitnami.com/sealed-secrets-key=active, the secret has to live in the same namespace as the controller . There can be multiple such secrets.

Below you can find all the steps needed to create and consume your own certificates.

  1. Set your vars to your existing certificates
export PRIVATEKEY="sealed-secret.key"
export PUBLICKEY="sealed-secret.crt"
export NAMESPACE="kube-system"
export SECRETNAME="sealed-secret-armand"
  1. Generate a new RSA key pair (certificates), e.g. 1 year validity:
openssl req -x509 -days 3650 -nodes -newkey rsa:4096 -keyout "$PRIVATEKEY" -out "$PUBLICKEY" -subj "/CN=sealed-secret/O=sealed-secret"
  1. Create a tls kubernetes secret, using your recently created RSA key pair
kubectl -n "$NAMESPACE" create secret tls "$SECRETNAME" \
	--cert="$PUBLICKEY" \
	--key="$PRIVATEKEY"

kubectl -n "$NAMESPACE" label secret "$SECRETNAME" \
	sealedsecrets.bitnami.com/sealed-secrets-key=active
  1. Deleting the controller Pod will force to pick the new keys
kubectl get pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

kubectl get pod sealed-secrets-controller-54449b494f-vwl5s -n kube-system
  1. Your private key was saved to disk as sealed-secret.key, additionally Get the decryption key using the commands given below:
sudo  kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secret-MASTER.yaml
  1. Store these keys (sealed-secret.key and sealed-secret-MASTER.yaml) at the secure location like data key vault.

Check the expiry date of your secrets

To check the expiry date of a certificate,to whom it is issued and the issuer: 

export cert="sealed-secret.crt"
openssl x509 -in $cert -issuer -noout -subject -dates

openssl x509 -in $cert -issuer -noout -subject -dates
issuer=CN = sealed-secret, O = sealed-secret
subject=CN = sealed-secret, O = sealed-secret
notBefore=Jul  5 16:21:39 2024 GMT
notAfter=Jul  3 16:21:39 2025 GMT

How to reuse SealedSecret Key

Sealed secret has one problem that a key pair is generated only once, so a sealed secret generated for one cluster will not be usable in another cluster. This problem can be resolved by reusing the decryption key. To do it follow the steps given below:

  1. Get the decryption key using the command given below:
sudo  kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > MASTER.yaml

  1. Store this key at the secure location like data key vault.

  2. Now this key can be used in another cluster using the command below. Make sure the key is being created in the correct namespace.

sudo kubectl apply -f MASTER.yaml
  1. When the SealedSecret controller starts it will scan the namespace to check whether a key already exists or not. Deleting the controller Pod will force to pick the new keys
kubectl get pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

kubectl delete pod sealed-secrets-controller-54449b494f-vwl5s -n kube-system

Understanding scopes

Before we start encrypting secrets, we need to understand the concept of scopes.

The scope is nothing but the context or visibility of a sealed secret within a Kubernetes cluster. The scope of a Sealed Secret relates to where and how the Sealed Secret can be decrypted and used within your cluster.

There are 3 types of scopes in Sealed Secrets:

Scope Type Description
strict The name and namespace of the secret are included in the encrypted data. Therefore, you must seal the secret using the same name and namespace.
namespace-wide You can freely rename the sealed secret within a given namespace.
cluster-wide You have the flexibility to unseal the secret using any name and in any namespace.

By default, strict scope is selected unless you pass the --scope flag to kubeseal CLI with a different value, i.e.:

# for json
kubeseal --scope cluster-wide <secret.yaml >sealed-secret.json
# for yaml
kubeseal --format=yaml --scope cluster-wide < secret.yaml > sealed-secret.yaml

Setup kubeseal

Kubeseal client

Installing the kubeseal Client

To install the client-tool on Linux x86_64 systems, execute the following command to place it in /usr/local/bin:

KUBESEAL_VERSION='' # Set this to, for example, KUBESEAL_VERSION='0.23.0'

curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION:?}/kubeseal-${KUBESEAL_VERSION:?}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION:?}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Or, get the latest (curl and jq required):

# Fetch the latest sealed-secrets version using GitHub API
KUBESEAL_VERSION=$(curl -s https://api.github.com/repos/bitnami-labs/sealed-secrets/tags | jq -r '.[0].name' | cut -c 2-)

# Check if the version was fetched successfully
if [ -z "$KUBESEAL_VERSION" ]; then
    echo "Failed to fetch the latest KUBESEAL_VERSION"
    exit 1
fi

curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

For MacOS systems, the client-tool is installed as follows:

brew install kubeseal

Installing the Custom Controller and CRD for SealedSecret using helm

Our current deployment process can be done manually using the helm install command or kubectl on the targeted cluster:

$ helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
$ helm dependency update sealed-secrets


export VERSION="2.16.0"
helm install sealed-secrets \
	-n kube-system \
	--set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets \
	--version $VERSION

Check the status of the controller pod:

kubectl get pods -n kube-system | grep sealed-secrets-controller

sealed-secrets-controller-859b878fff-j4lng   1/1     Running   0              12m

The logs printed by the controller reveal the name of the Secret that it created in its namespace, kube-system, and which contains the private key pair used by the controller for unsealing SealedSecrets deployed to the cluster. Note that the name of the controller pod will be different in your cluster.

kubectl logs sealed-secrets-controller-859b878fff-j4lng  -n kube-system


time=2024-07-05T15:06:27.733Z level=INFO msg="Starting sealed-secrets controller" version=v0.27.0
time=2024-07-05T15:06:27.736Z level=INFO msg="Searching for existing private keys"
time=2024-07-05T15:06:36.351Z level=INFO msg="New key written" namespace=kube-system name=sealed-secrets-keykbw9g
time=2024-07-05T15:06:36.351Z level=INFO msg="
-----BEGIN CERTIFICATE-----\nMIIEzDCCArSgAwIBAgIQCiLgiG8FpP9agrEq+AqfXzANBgkqhkiG9w0BAQsFADAA\nMB4XDTI0MDcwNTE1MDYzNloXDT....d\n/JXCXE/hw/OOocG/eWjtF2yptg/ttcKxjnMsXWreZCB4qPFX6jmUYOtS7+hwOYDW\n+kXOECVjooojoWjBuzwEPsWX5oz1Z6mPMtJfQIrcy4g=\n
-----END CERTIFICATE-----
time=2024-07-05T15:06:36.361Z level=INFO msg="HTTP server serving" addr=:8080
time=2024-07-05T15:06:36.361Z level=INFO msg="HTTP metrics server serving" addr=:8081

As seen from the logs of the controller, it searches for a Secret with the label sealedsecrets.bitnami.com/sealed-secrets-key in its namespace. If it does not find one, it creates a new one in its namespace and prints the public key portion of the key pair to its output logs. View the contents of the Secret which contais the public/private key pair in YAML format as follows:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml

Output:

apiVersion: v1
items:
- apiVersion: v1
  data:
    tls.crt: LS0tLS1CRUd
    tls.key: LS0tLS1CRUd
  kind: Secret
  metadata:
    creationTimestamp: "2024-07-05T15:36:20Z"
    labels:
      sealedsecrets.bitnami.com/sealed-secrets-key: active
    name: sealed-secret-armand
    namespace: kube-system
    resourceVersion: "85055615"
    uid: 000aa670-9793-4b02-a30f-ffe44955bdb2
  type: kubernetes.io/tls
kind: List
metadata:

Converting Secret to Sealed Secret

Option 1. Seal using the sealed-secret-controller

kubeseal can encrypt the Secret using the public key that it fetches at runtime from the controller running in the Kubernetes cluster

  1. Create a Secret. A example Kubernetes Secret manifest is given below, with username and passwords are base64 encoded. For example, save the manifest in a file name secrets.yaml.
apiVersion: v1
kind: Secret
metadata:
  name: test-secret
  namespace: test
data:
  password: asd3qrt3q2c=        # base64 encoded username
  username: asd3qrt3q2c=        # base64 encoded password
  1. Run the kubeseal command

Use the kubeseal command to convert the secret.yaml file to a Sealed Secret. The -f flag specifies the input file, and the -w flag writes the output to sealedsecret.yaml.

kubeseal -f secret.yaml -w sealedsecret.yaml

This command will generate a new file, sealedsecret.yaml, containing the encrypted secret.

  1. Review the Sealed Secret file

Open the sealedsecret.yaml file to review the contents. It should look similar to the example below:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: test-secret
  namespace: test
spec:
  encryptedData:
    password: AgB7KNC0rRElWiW...
    username: AgBz4WWuA5z/LBE...
  template:
    metadata:
      creationTimestamp: null
      name: test-secret
      namespace: test

This file is safe to store in version control systems such as your git repository as it is encrypted.

Option 2. Seal using the offline public key

If a user does not have direct access to the cluster, then a cluster administrator may retrieve the public key from the controller logs and make it accessible to the user.

A SealedSecret CRD is then created using kubeseal as follows using the public key file:

kubeseal --format=yaml --cert=sealed-secret.crt < secret.yaml > sealed-secret.yaml

Obfuscate keys in a a sealed secret file

Note that the keys in the original Secret — namely, username and password — are not encrypted in the SealedSecret, only their values are. You may change the names of these keys, if necessary, in the SealedSecret YAML file and still be able to deploy it successfully to the cluster. However, you cannot change the name and namespace of the SealedSecret. Doing so will invalidate the SealedSecret because the name and namespace of the original Secret are used as input parameters in the encryption process.

Apply the Secret

The YAML manifest for the Secret is no longer needed and can be deleted. We’ll only be deploying the SealedSecret to the cluster as shown below:

rm secret.yaml
kubectl apply -f sealed-secret.yaml

Once the SealedSecret CRD is created in the cluster, the controller understands this and uses the private key to unseal the underlying Secret, deploying it to the same namespace. You can verify this by checking the controller’s logs.

kubectl logs sealed-secrets-controller-859b878fff-j4lng  -n kube-system

References:

comments powered byDisqus

Copyright © Armand