3. SSL/Cert-manager
To get free and Automatic SSL certificates using Cert manager and Let’s Encrypt
In today's scenario, SSL certificates are the most important part of Deploying an application to the Internet. It is only the most important attributes that determine whether your websites are safe or not.
Image credit: Google
This paddle lock symbol conveys to your customer that the website they are visiting is safe, secured, and verified. So how do you actually achieve HTTPS on your website?
HTTP + SSL = HTTPS.
Getting an SSL certificate is not that easy. Moreover, it is expensive too. In today's world where Kubernetes is Predominantly everywhere. And with tonnes of Ingress resources in Kubernetes, it becomes really hard to get such a huge number of certificates, monitor them, and rotate them every time. This would be a nightmare for the DevOps Engineers. What if I tell you that there a tool that could get you free SSL certificates and rotate them automatically when they expire? Here comes Cert-Manager. Cert-manager was created by Jetstack, and a lot of the development is still sponsored by them. As per the cert-manager’s official guide, the cert-manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self-signed. It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.
The HTTP protocol developed in the early 1990s has become an integral part of our daily life. Today, we cannot live a single day without it. However, it does not even provide a basic level of security when exchanging information between the user and the web server. That is when HTTPS (“S” means “secure” here) comes to the rescue. In HTTPS, the exchanged data is encrypted using SSL/TLS — that family of protocols has proven itself well in protecting privacy and data integrity and is being actively promoted by the industry.
For example, Google has been calling for “HTTPS everywhere” since 2014. When prioritizing search results, it takes into account whether sites use secure, encrypted connections. All this propaganda affects ordinary users as well: modern browsers warn their users about insecure connections and invalid SSL certificates.
A certificate for a personal website might cost tens of dollars. However, buying it is not always justified. Fortunately, since late 2015, there is a free alternative in the form of Let’s Encrypt (LE) certificates. This nonprofit authority was created by Mozilla enthusiasts to make internet-wide hassle-free encryption a reality.
The certificate authority issues domain-validated certificates (the most basic ones available on the market) valid for 90 days, and it is also possible to obtain a so-called wildcard certificate for several subdomains.
The algorithms described in the Automated Certification Management Environment (ACME) protocol (designed specifically for Let’s Encrypt) are used to obtain a certificate. With it, the agent can prove control of the domain either by provisioning an HTTP resource (the so-called “HTTP-01 challenge”) or DNS records (“DNS-01 challenge”) — more information about them you may find below.
Cert-manager is a Kubernetes-native certificate management controller consisting of a set of CustomResourceDefinitions (hence the restriction on the minimum supported version of K8s, v1.12) for configuring CA (certificate authorities) and obtaining certificates. The installation of CRDs in a cluster is straightforward and boils down to applying a single YAML file:
kubectl create ns cert-managerkubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml
Before being able to request certificates, you must create CA resources:
Issuer or ClusterIssuer. They are used for signing CSRs (certificate requests). The difference between these resources is that ClusterIssuer is non-namespaced and can be used in multiple namespaces:Issueris used within a single namespace only;ClusterIssueris a global cluster object.
Let’s start with the simplest case — requesting a self-signed certificate. It is quite common. For example, you can use it in your K8s cluster for testing environments dynamically created for developers needs. It can be also useful in the case of the external load balancer that terminates SSL traffic.
The
Issuer resource would look like this:apiVersion: cert-manager.io/v1alpha2kind: Issuermetadata: name: selfsignedspec: selfSigned: {}
To issue a certificate, you have to define the
Certificate resource that determines the issuance (see the issuerRef section below) and the location of the private key (the secretName field). Then you need to invoke that key in the Ingress (note the tls section in the spec field):---apiVersion: cert-manager.io/v1alpha2kind: Certificatemetadata: name: selfsigned-crtspec: secretName: tls-secret issuerRef: kind: Issuer name: selfsigned commonName: "yet-another.website" dnsNames: - "yet-another.website"---apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: appspec: tls: - hosts: - "yet-another.website" secretName: tls-secret rules: - host: "yet-another.website" http: paths: - path: / backend: serviceName: app servicePort: 8080
The certificate will be issued in a few seconds after these resources are added to the cluster. You can see the confirmation in the output of the command:
kubectl -n app describe certificate selfsigned-crt... Normal GeneratedKey 5s cert-manager Generated a new private key Normal Requested 5s cert-manager Created new CertificateRequest resource "selfsigned-crt-4198958557" Normal Issued 5s cert-manager Certificate issued successfully
Looking at the secret resource itself, you will see:
- the
tls.keyprivate key, - the
ca.crtroot certificate, - and the self-signed
tls.crtcertificate.
You can browse the contents of these files using the
openssl utility:kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuernotBefore=Feb 10 21:01:59 2020 GMTnotAfter=May 10 21:01:59 2020 GMTissuer=O = cert-manager, CN = yet-another.website
It is worth noting that clients consuming certificates issued in such a way (i.e., using the self-signed issuer) will not trust them. The reason is simple: this issuer type does not have a CA (see the note on the project website).
To avoid this, specify the path to the secret file containing
ca.crt in the Certificate resource. For example, you can use the corporate CA. This way, you will be able to sign certificates issued for your Ingress with a key that is already in use by other server services/information systems.As I mentioned before, there are two types of “challenges” available to prove the control of the domain, HTTP-01 and DNS-01.
The first approach (HTTP-01) involves deploying a tiny web server as a separate deployment. It will be serving some information at the
http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> URL per request of the certification server. Therefore, this method implies the accessibility of Ingress from the outer world via port 80 and the publicity of the domain’s DNS record.The second challenge (DNS-01) makes sense if there is an API you can use to change the DNS records of your domain. The Issuer uses these tokens to create TXT records for your domain. Then, ACME server gets these records during confirmation. Let’s Encrypt easily integrates with various DNS providers, including CloudFlare, AWS Route53, Google CloudDNS, and others (as well as with the LE’s own DNS implementation, acme-dns).
Note: Let’s Encrypt imposes fairly strict limits on requests to ACME servers. To avoid unnecessary load on LE’s production environment, we recommend using the letsencrypt-staging certificate for testing (the difference is in the ACME server only).
So, let’s describe the resources:
apiVersion: cert-manager.io/v1alpha2kind: Issuermetadata: name: letsencryptspec: acme: server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt solvers: - http01: ingress: class: nginx---apiVersion: cert-manager.io/v1alpha2kind: Certificatemetadata: name: le-crtspec: secretName: tls-secret issuerRef: kind: Issuer name: letsencrypt commonName: yet-another.website dnsNames: - yet-another.website
Note that we use the staging server in
acme’s server field for our Issuer. You can replace it with the production one later.Let’s apply this configuration and trace the entire process of obtaining the certificate:
1. The creation of a
Certificate leads to the emergence of a new CertificateRequest resource:kubectl -n app describe certificate le-crt...Created new CertificateRequest resource "le-crt-1127528680"
2. In its description there is a notification about an
Order creation:kubectl -n app describe certificaterequests le-crt-1127528680…Created Order resource app/le-crt-1127528680-1805948596
3. The
Order contains the description of parameters of the validation and its current status. The validation is performed by the Challenge resource:kubectl -n app describe order le-crt-1127528680-1805948596…Created Challenge resource "le-crt-1127528680-1805948596-1231544594" for domain "yet-another.website"
4. And finally, there is information about the status of the validation itself in resource details:
kubectl -n app describe challenges le-crt-1127528680-1805948596-1231544594... Reason: Successfully authorized domain ... Normal Started 2m45s cert-manager Challenge scheduled for processing Normal Presented 2m45s cert-manager Presented challenge using http-01 challenge mechanism Normal DomainVerified 2m22s cert-manager Domain "yet-another.website" verified with "http-01" validation
The certificate will be issued in less than a minute if all the prerequisites are met: the domain is accessible from the outer world, the rate limits on the LE side are respected, and so on. If the issuance was successful, you should see the message “Certificate issued successfully” in the output of the
describe certificate le-tls command.Now you can safely change the ACME server address to the production one (
https://acme-v02.api.letsencrypt.org/directory) and re-issue valid certificates signed by Let's Encrypt Authority X3 instead of Fake LE Intermediate X1.But first, you have to delete the
Certificate resource. Otherwise, the issuance procedure will not start because the certificate already exists, and it is valid. Deleting a secret would immediately result in invalidation of the certificate with the following message in the output of the describe certificate command: Normal PrivateKeyLost 44s cert-manager Lost private key for CertificateRequest "le-crt-613810377", deleting old resource
Now it is time to apply the production manifest for the
Issuer with the Certificate described above (it has not changed):apiVersion: cert-manager.io/v1alpha2kind: Issuermetadata: name: letsencryptspec: acme: server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt solvers: - http01: ingress: class: nginx
After receiving the
Certificate issued successfully confirmation, let us check it out:kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuernotBefore=Feb 10 21:11:48 2020 GMTnotAfter=May 10 21:11:48 2020 GMTissuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
To make one step further we will issue a certificate for all subdomains of the site using another method of validation — the DNS one. We will use CloudFlare as our DNS provider to change the domain records we need.
First, let’s create a token to use the CloudFlare API:
1. Profile → API Tokens → Create Token.
2. Set access rights as follows:
- Permissions:
→ Zone — DNS — Edit
→ Zone — Zone — Read
- Zone Resources:
→ Include — All Zones
3. Copy the token generated (for example,
y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M).Create a
Secret resource containing the token and describe it in your Issuer:apiVersion: v1kind: Secretmetadata: name: cloudflare-api-tokentype: OpaquestringData: api-token: y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M---apiVersion: cert-manager.io/v1alpha2kind: Issuermetadata: name: letsencryptspec: acme: server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt solvers: - dns01: cloudflare: email: [email protected] apiTokenSecretRef: name: cloudflare-api-token key: api-token---apiVersion: cert-manager.io/v1alpha2kind: Certificatemetadata: name: le-crtspec: secretName: tls-secret issuerRef: kind: Issuer name: letsencrypt commonName: yet-another.website dnsNames: - "yet-another.website" - "*.yet-another.website"
(Do not forget to use a staging environment for testing!)
It is time to go through the domain ownership confirmation procedure:
kubectl -n app describe challenges.acme.cert-manager.io le-crt-613810377-1285319347-3806582233...Status: Presented: true Processing: true Reason: Waiting for dns-01 challenge propagation: DNS record for "yet-another.website" not yet propagated State: pendingEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Started 54s cert-manager Challenge scheduled for processing Normal Presented 53s cert-manager Presented challenge using dns-01 challenge mechanism
The TXT record will appear at your DNS dashboard:
… and after a while, the status will change to:
Domain "yet-another.website" verified with "dns-01" validation
Let’s make sure that the certificate is valid for all subdomains:
kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -text |grep DNS: DNS:*.yet-another.website, DNS:yet-another.website
The validation over DNS is usually slow since most DNS providers have a so-called “propagation time” — a period showing how long it takes for an updated DNS record to become available on all DNS servers of the provider.
The ACME standard also supports a combination of both types of validation. You can use it to speed up obtaining a certificate for the main domain. In this case, the description of the
Issuer will have the following form:apiVersion: cert-manager.io/v1alpha2kind: Issuermetadata: name: letsencryptspec: acme: server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt solvers: - selector: dnsNames: - "*.yet-another.website" dns01: cloudflare: email: [email protected] apiTokenSecretRef: name: cloudflare-api-token key: api-token - selector: dnsNames: - "yet-another.website" http01: ingress: class: nginx
If you apply this configuration, two
Challenge resources will be created:kubectl -n app describe orders le-crt-613810377-1285319347… Normal Created 3m29s cert-manager Created Challenge resource "le-crt-613810377-1285319347-3996324737" for domain "yet-another.website" Normal Created 3m29s cert-manager Created Challenge resource "le-crt-613810377-1285319347-1443470517" for domain "yet-another.website"
Besides creating certificates directly, you can use cert-manager’s ingress-shim component. It relieves you of the need to create
Certificate resources explicitly. The idea is to obtain a certificate automatically using the Issuer specified in the special annotations of Ingress. Here is an example of the respective Ingress resource:apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: annotations: cert-manager.io/cluster-issuer: letsencryptspec: tls: - hosts: - "yet-another.website" secretName: tls-secret rules: - host: "yet-another.website" http: paths: - path: / backend: serviceName: app servicePort: 8080
In this case, the availability of Issuer is enough to get the work done, meaning that we have to create a lesser number of entities.
We have learned how to obtain auto-renewable, self-signed, and free SSL certificates from Let’s Encrypt for website domains that are managed by Ingresses in the Kubernetes clusters.
The article provides example solutions for the most common problems we face. However, cert-manager features are not limited to those described above. On the project’s website, you can find examples of using it with other services. For example, you can use Vault as a certificate authority or set up external Issuers.
PLEASE NOTE: our blog has MOVED to https://blog.flant.com/! New articles from Flant’s engineers will be posted there only. Check it out and subscribe to keep updated!
Last modified 9mo ago