Canary Deployment with k8s ingress-controller

Posted on | In | Comments:

Canary deployment is useful in many scenarios, like A/B testing.
k8s ingress-controller makes canary deployment easy, I will introduce the steps of it using the example of a microservice we have.

This document will not cover how to set up ELB. It will focus only on how to create a canary deployment along side an existing deployment.

Situation

We have a service A running in k8s cluster, there is an external host name setup to expose the API to public, Requests sent through this API will be route to this service,

Here is the public host name setup:

1
2
3
▶ kubectl get ingress -n ingress
NAME                   HOSTS               ADDRESS   PORTS   AGE
service-a          public.domain.com             80      297d

and it also has k8s service set up:

So far, this is running happily on production,

1
2
3
4
▶ kubectl get pods -n emaildelivery
NAME                                   READY   STATUS    RESTARTS   AGE
service-a-68db48dff7-pckg9         1/1     Running   0          195d
service-a-68db48dff7-rxtm4         1/1     Running   5          159d

Canary Deployment

Deploy new pods with correct annotation

You will need to deploy new pods for sure, without overwriting current pods, you can do this by deploying new pods with a new application name or using a different namespace.

I want to reuse the same namespace, so I changed the application name to service-a-canary.

Then comes to the fun part, Keep all the necessary labels as service-a in the resources. Resources you will need to change include service.yaml, deployment.yaml, ingress.yaml.

Key changes are as follows:

1
2
3
4
5
6
7
8
9
metadata:
  namespace: service-a
  name: service-a-canary
  labels:
    name: service-a
...
selector:
    matchLabels:
      app: service-a

If you release the new pods at this time point, new pods will running fine, but serving no traffic:

1
2
3
4
5
▶ kubectl get pods -n service-a
NAME                               READY   STATUS    RESTARTS   AGE
service-a-68db48dff7-pckg9         1/1     Running   0          195d
service-a-68db48dff7-rxtm4         1/1     Running   0          159d
service-a-canary-6f7784fc7-wg7kz   1/1     Running   0          2d20h

Enable canary with ingress-controller

To make canary deployment serving real traffic, simply adding few lines in ingress.yaml

1
2
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10"

nginx.ingress.kubernetes.io/canary-weight is the percentage of traffic that will be routed to canary deployment.

NOW, you can check your logs to see if traffic routed to canary deployment.