Home / Microservices Architecture Guide / Development / Creating a fast development and canary workflow for Kubernetes

Creating a fast development and canary workflow for Kubernetes

Richard Li   /    August 1, 2017

Creating a fast develop-deploy-canary-workflow can dramatically improve your productivity. In this tutorial, we're going to show you how you do this using some basic operational infrastructure you should already have -- Kubernetes and Docker.

Note: This tutorial has been subsumed by the hands-on tutorials in the Code Faster Guide.

Prerequisites

In order to complete this tutorial, you're going to need the following installed on your local machine:

  • Docker
  • Git
  • Python 2.7 or later (our sample web service is written in Python)

In addition, you'll also need:

  • Access to a Kubernetes cluster (e.g., GKE) with 5 nodes of excess capacity. (Note that the current application is hard-coded to use production-level resources. In the future, expect a separate development-level configuration.)
  • An account with a Docker registry (e.g., Google Container Registry or Docker Hub)

Don't want to set this all up right now? You can also try our interactive, browser-based tutorial which is an abbreviated form of this tutorial -- you should be able to run through it ten minutes.

The Big Picture

Getting source code deployed into Kubernetes so you can test it requires the following steps:

  1. Build a Docker container that contains your source code and any runtime dependencies.
  2. Push the container to a container registry.
  3. Write a Kubernetes manifest. The manifest tells Kubernetes what to run (i.e., it points to the container) and how to run it (e.g., what ports to expose, or the maximum amount of memory to allocate).
  4. Tell Kubernetes to process the manifest, using kubectl, the Kubernetes command line tool.

For canary deployments, there's one additional step:

  • Tell your load balancer to route traffic "asymmetrically", e.g., 90% of the traffic to one service, and 10% to another service

All of these steps are straightforward but complicated. In this tutorial, we'll walk you through each of these steps, and show how everything can be tied together.

A sample application

1. The GitHub repository

We're going to start with a sample application that consists of multiple services. Clone this GitHub repository here:

git clone https://github.com/datawire/todo.git

We've set up this application as a monorepo, with each service in its own subdirectory. The TODO application consists of a few services (tasks, search, auth) as well as a MongoDB database that stores tasks.

2. Create the Dockerfile

Each of these services has a Dockerfile that specifies how it should be built. They're all very similar. Here's a typical Dockerfile:

FROM alpine:3.5
RUN apk add --no-cache python py2-pip py2-gevent
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
WORKDIR /app
EXPOSE 8080
ENTRYPOINT ["python"]
CMD ["app.py"]

This Dockerfile starts with Alpine Linux, installs Python and the necessary dependencies to run the tasks service, and starts app.py, the actual tasks service.

3. Create a Kubernetes manifest

Kubernetes takes care of all the details of running a container: which VM/server to run the container on, making sure the container has the right amount of memory/CPU/etc., and so forth. In order to know how to run your service, you need a Kubernetes manifest. Here's the example manifest for the tasks service:

---
apiVersion: v1
kind: Service
metadata:
  name: {{name}}
spec:
  selector:
    app: {{name}}
  ports:
    - protocol: {{service.protocol or 'TCP'}}
      port: {{service.port or '80'}}
      targetPort: {{service.targetPort or '8080'}}
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata: {name: {{name}}}
spec:
  replicas: 2
  selector:
    matchLabels: {app: {{name}}}
  strategy:
    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
    type: RollingUpdate
  revisionHistoryLimit: 1
  template:
    metadata:
      labels: {app: {{name}}}
      name: {{name}}
    spec:
      containers:
      - image: {{build.images["Dockerfile"]}}
        imagePullPolicy: IfNotPresent
        name: {{service.name}}
        resources:
          limits:
            memory: {{service.memory}}
            cpu: {{service.cpu}}
        terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30

Manifests must reference an actual Docker image. Since the Docker image can change as you build new versions of your service, we're using Jinja2 templates in the manifest file to template key values.

Docker lets you add a tag an image (e.g., "latest"), which could let you statically code in a reference to an image. In practice, however, this approach prevents you from deploying multiple versions of your service live without multiple manifests.

4. Deploy

In our example application, we've gone ahead and defined the Kubernetes manifests and Dockerfiles for each of the services. So, now it's time to deploy the services into Kubernetes, from source.

If you're doing this process by hand, you'll want to:

  • run docker build for each of your services, to build the image
  • run docker push for each of your images, to push the images to your Docker registry
  • run kubectl apply on each of your manifests, to actually deploy the images to Kubernetes

This process is somewhat tedious, so we've written an automation script called Forge that does all this work for you (and it handles dependencies, to boot). Install Forge:

% curl -sL https://raw.githubusercontent.com/datawire/forge/master/install.sh | INSTALL_DIR=${HOME}/forge sh

Then, in the same directory as the TODO application, give Forge some basic information about your username, Docker Registry, and credentials (which get saved locally):

% forge setup

Once Forge is set up, you can deploy the TODO application with a single command:

% forge deploy

5. Testing your application

To verify that the application is properly running, you can send an HTTP request. Get the external IP address of the API Gateway:

% kubectl get services
NAME           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
api            10.11.252.234   104.154.184.70   80:31876/TCP   4d
auth           10.11.253.201   <none>           80/TCP         4d
kubernetes     10.11.240.1     <none>           443/TCP        54d
search         10.11.248.76    <none>           80/TCP         1d
tasks          10.11.251.80    <none>           80/TCP         4d
tasks-canary   10.11.243.82    <none>           80/TCP         4d

Then, send the HTTP request. We've configured the API Gateway to perform basic HTTP authentication by default, with a password of "todo". So let's pass in the authentication credentials as well.

% curl 104.154.184.70/tasks/ -u any:todo

6. The API Gateway

We've just mentioned in passing an API Gateway, so let's dive a little deeper into what's going on with the sample application. The sample application consists of a number of services, each in their own subdirectories: tasks, auth, search, and so forth.

In addition, in the root directory of the TODO application, we've configured one special service -- an API Gateway. In our example, the API Gateway we're using is a specially configured version of Ambassador, an open source API Gateway built on Envoy.

The Envoy instance is configured via the envoy.json file, with a Kubernetes manifest in the k8s directory.

Instead of routing all requests directly to each of the services, requests are first routed to the API Gateway, which then routes the requests to the services. This additional layer lets us:

  1. Add authentication (and other common functionality), as we've seen. The actual authentication process is implemented as the auth microservice. The API Gateway is configured to issue a subrequest to the auth microservice to validate all incoming requests.

  2. We can implement canary testing, where you can deploy new versions of a service side-by-side with an existing service, and route a fraction of your traffic to the new service.

7. Canary deployments

Canary deployments are an essential way to validate new versions of a service without impacting your overall application. Let's see how this works.

First, we'll make a modification to the tasks service. Any change to the tasks service can be canaried (e.g., deployment metadata changes or code changes). Let's simulate a change where we accidentally impact performance by adding a sleep() call. Edit tasks/app.py and update root() as follows:

def root():
    result = mongo.db.tasks.find(projection={"_id": False})
    time.sleep(0.5) ## sleep
    return jsonify({"status": "ok",
                    "tasks": list(result)})

Now, change to the tasks directory:

% cd tasks

And deploy the new code, with a canary:

% CANARY=true forge deploy

Now, send a bunch of HTTP requests to the API Gateway:

% while true; do curl http://104.154.184.70/tasks/ -u any:todo; done

90% of these requests will hit your original, fast service, while 10% of these will hit the slower service. This might be hard to see from the terminal, so let's try a more sophisticated approach.

8. Monitoring

In this sample application, we've also deployed Prometheus for metrics monitoring. We're going to use Prometheus to see the latency spike. Get the public IP address of Prometheus:

% kubectl get services
NAME           CLUSTER-IP      EXTERNAL-IP      PORT(S)           AGE
prometheus     10.11.245.177   35.188.142.214   80:30193/TCP      4m
search         10.11.248.76    <none>           80/TCP            1d
tasks          10.11.251.80    <none>           80/TCP            4d
tasks-canary   10.11.243.82    <none>           80/TCP            4d
todo-db        None            <none>           27017/TCP         4m

Go to the external IP in your browser, which will display the Prometheus dashboard.

Paste the following query into Prometheus:

{__name__=~"envoy_cluster_tasks_upstream_rq_time_timer|envoy_cluster_tasks_canary_upstream_rq_time_timer"}

You'll see how the envoy_cluster_tasks_upstream latency is much lower than the envoy_cluster_tasks_canary_upstream (you may have to hit execute a few times to get enough data on your graph to see the difference.)

Recap

In this tutorial, we've shown how, as a developer, you can build a fast, productive workflow for developing, deploying, and testing microservices on Kubernetes.

We've also shown how this workflow can be used for a canary deployment, and integrate metrics collection into a dashboard.

We've also built this workflow without requiring additional operational primitives, i.e., we're just using Kubernetes and Docker and the Docker Registry.

Note: this document is a draft. If you run into bugs, we'd be happy to help. Send us an email at hello@datawire.io or join our Gitter chat.

Next steps

Try this with your own services. If you're creating a new service, you can:

  1. Clone the GitHub repository: git clone https://github.com/datawire/todo.git

  2. Copy an existing service into a new directory: cp -r tasks/ new_service/

  3. Update the service.yaml to reflect the name of the new service.

  4. Edit the envoy.json to reference the new_service. You can do a search-and-replace on "tasks" and replace it with "new_service".

  5. Run forge deploy in the main directory.

Questions?

We’re happy to help! Look for answers in the rest of the Microservices Architecture Guide, join our Gitter chat, send us an email at hello@datawire.io, or contact our sales team.