Cert-manager

Cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters and simplifies obtaining, renewing, and using those certificates.

It can issue certificates from various supported sources, including Let’s Encrypt, HashiCorp Vault, Venafi, and private PKI.

This tool supports certificate issuance from various sources, including Let’s Encrypt, HashiCorp Vault, Venafi, and private PKI systems.

Lets Encrypt

In this guide, we will focus on using Let’s Encrypt, a renowned, free, automated, and open Certificate Authority that issues digital certificates for HTTPS encryption. Known for its straightforward setup and broad browser compatibility, Let’s Encrypt is operated by a non-profit organization and advocates for secure web connections, although it does impose rate limits on certificate issuance.

ACME

We will configure cert-manager to procure a TLS certificate from Let’s Encrypt employing the ACME protocol. The ACME protocol facilitates Let’s Encrypt in verifying your authority over the domain names listed in your certificate request. This is done through a “challenge” process that necessitates specific verifiable modifications either on your website or DNS records

DNS-01 challenge - you need to create a DNS TXT record for your domain. This record should contain a unique token and a fingerprint of your account key, and it’s placed at _acme-challenge.<YOUR_DOMAIN>.
HTTP-01 challenge - either you or your ACME client must generate a file on your web server. This file should include a unique token and a fingerprint of your account key. Successfully doing this proves your control over the website to the Certificate Authority (CA). A key benefit of the HTTP-01 challenge is its ease of automation on widely used web server platforms like Nginx, which is part of our setup process.

**In this deployment, we will be configuring the HTTP-01 - You can follow along the guide on most cloud-managed Kubernetes platforms where External Load Balancer services are provided

Alternatively, you can configure the DNS-01 Ingress solver. This is not covered here, and you can see my written instructions to do that here in Automating HTTPS Certificates in Kubernetes Clusters Using Cert-Manager an DNS01

Installation

Install cert-manager with helm

Follow the Cert-Manager Helm install instructions for in-depth instructions. Here is a rundown:

Add the helm repo for cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

Install cert-manager using helm (you can specify a specific version or omit the version flag for the latest)

export VERSION=v1.10.0
export NAMESPACE=cert-manager

helm install cert-manager jetstack/cert-manager \
    --namespace $NAMESPACE \
    --create-namespace \
    --version $VERSION \
    --debug \
    --set installCRDs=true

Watch and verify the pods are running before moving on

export NAMESPACE=cert-manager
watch kubectl get all -n cert-manager

Create `ClusterIssuer` for Issuing certificates

Issuers and clusterissuers are the backbone for supplying certificates to your cluster. Right now, with just the basic Cert-Manager setup we’ve done, it’s not yet ready to start creating certificates. So by hooking up Issuers and clusterissuers configured for Let’s Encrypt, we can get our cluster to automatically grab new certificates for our services. This way, we’re gearing up for some real action in certificate management.

Before we get started, here are some recommendations and notes:

Staging vs Produciton ClusterIssuer: It’s highly recommended to test against Let’s Encrypt’s staging environment before using our production environment. This will allow you to get things right before issuing trusted certificates and reduce the chance of your running up against rate limits. That said we can create both ClusterIssuers for Production and Staging:
- ClusterIssuer for Staging: Used for testing and development, obtaining untrusted certificates from a CA’s test environment. Useful for avoiding production rate limits and for testing configurations.
- ClusterIssuer for Production: Used to obtain real, trusted TLS certificates from a certificate authority (CA) for live, user-facing environments. Subject to strict rate limits.
Namespaces: Consider using ClusterIssuer (which we do here) as opposed to Issuer (Feel free to change this up), while they both identify which Certificate Authority cert-manager will be used to issue a certificate, Issuer is a namespaced resource allowing you to use different CAs in each namespace, and ClusterIssuer is used to issue certificates in any namespace.
ClusterIssuer Naming convention: In the steps below, we will be naming the ClusterIssuer letsencrypt-staging and letsencrypt-prod, we can then reference this name in the cert-manager.io/cluster-issuer annotation in your ingress deployment

Create ClusterIssuer for Staging

Create a manifest file to create the ClusterIssuer for Staging. You can use this template, also shown below:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: ${EMAIL} # CHANGE ME
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            class: nginx # Or any other ingress class available

For example,

# download the template into a temp file
curl https://gist.githubusercontent.com/armsultan/226dc1f47d5e421f75739e31f7ee788a/raw/183c6bf61aad9e9f92c071117bcb9c1b05304fce/staging-clusterissuer-http01.yaml -o  staging-clusterissuer-http01.yaml.temp 

# Then, substitute the placeholder with your API token
export [email protected]

envsubst < staging-clusterissuer-http01.yaml.temp   \
             > staging-clusterissuer-http01.yaml 

# delete the temp file
rm staging-clusterissuer-http01.yaml.temp

Apply the manifest file to create the ClusterIssuer for Staging.

kubectl apply -f staging-clusterissuer-http01.yaml -n cert-manager

ClusterIssuer for Production

Create a manifest file to create the ClusterIssuer for Production. You can use this template, also shown below:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${EMAIL} # CHANGE ME
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx # Or any other ingress class available

For example,

# download the template into a temp file
curl https://gist.githubusercontent.com/armsultan/30e1d9af27ae8fb6b7ba0dbbd5ff0eab/raw/edf837d6b39f052e33fe8111b73117b9d0837edd/prod-clusterissuer-http01.yaml -o prod-clusterissuer-http01.yaml.temp 

# substitute the placeholder with your API token
export [email protected]

envsubst < prod-clusterissuer-http01.yaml.temp  \
             > prod-clusterissuer-http01.yaml

# delete the temp file
rm prod-clusterissuer-http01.yaml.temp

Apply the manifest file to create the ClusterIssuer for Staging.

kubectl apply -f prod-clusterissuer-http01.yaml -n cert-manager

Check they have been deployed

Check both letsencrypt-staging and letsencrypt-prod are ready

kubectl get clusterissuer -n cert-manager

# Example output
NAME                  READY   AGE
letsencrypt-staging   True    5m43s
letsencrypt-prod      True    69s

When we view all CertificateRequest resources we see that an Order has been created and is not yet READY

 kubectl get certificaterequest -A
NAMESPACE             NAME                          APPROVED   DENIED   READY   ISSUER                 REQUESTOR                                         AGE
verify-cert-manager   moon-tls-6xbqg                True                False   letsencrypt-prod       system:serviceaccount:cert-manager:cert-manager   11m
verify-cert-manager   sun-tls-dg92k                 True                False   letsencrypt-staging    system:serviceaccount:cert-manager:cert-manager   11m

When we run a description on those CertificateRequest resources, we see information about failures in the process. In our case, we are still waiting on “certificate issuance”:

kubectl describe CertificateRequest  moon-tls-6xbqg  -n verify-cert-manager 


# Example output:
# ...etc...
Events:
  Type    Reason           Age   From          Message
  ----    ------           ----  ----          -------
  Normal  cert-manager.io  14m   cert-manager  Certificate request has been approved by cert-manager.io
  Normal  OrderCreated     14m   cert-manager  Created Order resource verify-cert-manager/moon-tls-6xbqg-2438443713
  Normal  OrderPending     14m   cert-manager  Waiting on certificate issuance from order verify-cert-manager/moon-tls-6xbqg-2438443713: ""

Validation

Create and Verify a Let’s Encrypt certificate with a Test Application.

If you have an Ingress Controller set up in your cluster, we can validate our Let’s Encrypt staging and production certificates by directing traffic to a demo application. Typically, in cloud-managed Kubernetes environments like AWS, deploying a load balancer service automatically assigns an external IP address, usually pointing to an Elastic Load Balancer (ELB) or a Network Load Balancer (NLB). To get your certificate validated by Let’s Encrypt, it’s essential to set up a CNAME record that maps your domain (the hostname for which you’re requesting the certificate) to this publicly available IP address. This step ensures that Let’s Encrypt can successfully verify your domain ownership during the HTTP-01 ACME challenge process and issue a valid certificate.

The HTTP01 validation method requires port 80 to be opened on the server (or your load balancer) as per RFC8555. If you can’t use this port, you can’t use this method.

We will do that here…

Create the application and get External IP Address

Apply this manifest for deploying test applications (“sun” and “moon”) with ingress, using the staging certificate for the sun and the production certificate for the moon. This all-in-one manifest is shown below:

apiVersion: v1
kind: Namespace
metadata:
  name: verify-cert-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: moon
  namespace: verify-cert-manager
spec:
  selector:
    matchLabels:
      app: moon
  replicas: 3
  template:
    metadata:
      labels:
        app: moon
    spec:
      containers:
        - name: moon
          image: armsultan/solar-system:moon-nonroot
          ports:
            - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sun
  namespace: verify-cert-manager
spec:
  selector:
    matchLabels:
      app: sun
  replicas: 3
  template:
    metadata:
      labels:
        app: sun
    spec:
      containers:
        - name: sun
          image: armsultan/solar-system:sun-nonroot
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: sun-service
  namespace: verify-cert-manager
  labels:
    app: sun
spec:
  type: ClusterIP
  selector:
    app: sun
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: moon-service
  namespace: verify-cert-manager
  labels:
    app: moon
spec:
  type: ClusterIP
  selector:
    app: moon
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sun-ingress
  namespace: verify-cert-manager
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging # or letsencrypt-prod
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - sun.test.demo.cequence.ai
      secretName: sun-t3st-org-tls
  rules:
    - host: sun.test.demo.cequence.ai
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: sun-service
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: moon-ingress
  namespace: verify-cert-manager
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - moon.test.demo.cequence.ai
      secretName: moon-t3st-org-tls
  rules:
    - host: moon.test.demo.cequence.ai
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: moon-service
                port:
                  number: 80

For example,

# download the file
curl https://gist.githubusercontent.com/armsultan/518e249f7f5c18442e7b8008095375cc/raw/72b288cf615953f4b1cefbf1b90d535f1825f168/verify-certmanager-app.yaml -o verify-certmanager-app.yaml

# Edit if needed

# Then, Apply the manifest and create the app
kubectl apply -f verify-certmanager-app.yaml

Find the ingress and external IP addresses. On cloud-managed Kubernetes platforms, it may take a minute for an external load balancer address to get assigned. Make a note of this address for the next step

watch kubectl get ingress -n verify-cert-manager     


# Example output
NAME                        CLASS    HOSTS                ADDRESS                                                                         PORTS     AGE
cm-acme-http-solver-6dckl   <none>   sun.lab.armand.nz    a5345055838744b399dd793c67bb790e-47aafd27f109a290.elb.us-west-2.amazonaws.com   80        67s
cm-acme-http-solver-c9vxj   <none>   moon.lab.armand.nz   a5345055838744b399dd793c67bb790e-47aafd27f109a290.elb.us-west-2.amazonaws.com   80        66s
moon-ingress                nginx    moon.lab.armand.nz   a5345055838744b399dd793c67bb790e-47aafd27f109a290.elb.us-west-2.amazonaws.com   80, 443   68s
sun-ingress                 nginx    sun.lab.armand.nz    a5345055838744b399dd793c67bb790e-47aafd27f109a290.elb.us-west-2.amazonaws.com   80, 443   68s

HTTP01 validation

To complete the HTTP-01 challenge for certificate validation, create a DNS record for the requested certificate’s Fully Qualified Domain Name (FQDN). In my deployment, both applications with FQDNs sun.lab.armand.nz and moon.lab.armand.nz are assigned to an external URL a5345055838744b399dd793c67bb790e-47aafd27f109a290.elb.us-west-2.amazonaws.com. Therefore, I need to create a CNAME record for this URL in my DNS manager. If it were a static IP, an A record would be required instead.

The HTTP-01 challenge will place a token on these FQDN addresses. Let’s Encrypt then attempts to access this token via sun.lab.armand.nz and moon.lab.armand.nz. Success in this retrieval means your certificate will be issued.

Create DNS Entry (A or CNAME Record)

The steps for Creating an A or CNAME record depend on your DNS provider. In my case, using Cloudflare was very straightforward following these instructions. I did the process twice, one for each FDQN (sun.lab.armand.nz and moon.lab.armand.nz)

1. Log in to your Cloudflare account using this link.

Click “Select Website” in the navigation bar and choose your domain which certificate request for the FQDN falls under
1. Click “DNS” in the menu.
In the DNS Records section, click the + add record button and fill in the fields:
- In the Type field, select CNAME.
- In the Name field, enter the External Address (kubectl get ingress)
- In the Target field, enter the FQDN (sun.lab.armand.nz or moon.lab.armand.nz)
- In the TTL field, select Auto.
- Click the Cloudflare cloud and arrow icon to deactivate that Cloudflare CDN.
Save

cloudflare-http01-cname

Validate Certificate

Note: You might need to wait a few minutes after the previous for DNS to propagate

Let’s test it with curl:

The **Sun** application uses a Staging self-signed certificate. Therefore, we must use the -k flag as we use an “invalid” (test only) staging certificate.

The Sun application uses the Staging clusterissuer which until now has not been

export HOSTNAME=sun.lab.armand.nz

curl https://$HOSTNAME  -ks | grep title\>
<title>The Sun</title>

The Moon application uses the prod clusterissuer and it should present a secure certificate fore for production

export HOSTNAME=moon.lab.armand.nz

curl https://$EXTERNALIP -H "Host: $HOSTNAME" -s | grep title\>
<title>The moon</title>

Note: Once our certificate has been obtained, cert-manager will periodically check its validity and attempt to renew it if it gets close to expiry. cert-manager considers certificates close to expiry when the ‘Not After’ field on the certificate is less than the current time plus 30 days.

Clean up

After finishing testing, remove the verify-cert-manager namespace and all its resources

kubectl delete namespace verify-cert-manager

Armand.nz

Automating HTTPS Certificates in Kubernetes using Cert-Manager, Lets Encrypt using and HTTP-01 Challenge

Cert-manager

Lets Encrypt

ACME

Installation

Install cert-manager with helm

Create `ClusterIssuer` for Issuing certificates

Create ClusterIssuer for Staging

ClusterIssuer for Production

Check they have been deployed

Validation

Create and Verify a Let’s Encrypt certificate with a Test Application.

Create the application and get External IP Address

HTTP01 validation

Create DNS Entry (A or CNAME Record)

Validate Certificate

Clean up

Automating HTTPS Certificates in Kubernetes using Cert-Manager, Lets Encrypt using and HTTP-01 Challenge

Cert-manager

Lets Encrypt

ACME

Installation

Install cert-manager with helm

Create ClusterIssuer for Issuing certificates

Create ClusterIssuer for Staging

ClusterIssuer for Production

Check they have been deployed

Validation

Create and Verify a Let’s Encrypt certificate with a Test Application.

Create the application and get External IP Address

HTTP01 validation

Create DNS Entry (A or CNAME Record)

Validate Certificate

Clean up

Create `ClusterIssuer` for Issuing certificates