Creating a fast development and canary workflow for Kubernetes
Richard Li / July 31, 2017
Creating a fast develop-deploy-canary-workflow can dramatically improve your productivity. In this tutorial, we're going to show you how you do this using some basic operational infrastructure you should already have -- Kubernetes and Docker.
Note: This tutorial has been subsumed by the hands-on tutorials in the Code Faster Guide.
Prerequisites
In order to complete this tutorial, you're going to need the following installed on your local machine:
- Docker
- Git
- Python 2.7 or later (our sample web service is written in Python)
In addition, you'll also need:
- Access to a Kubernetes cluster (e.g., GKE) with 5 nodes of excess capacity. (Note that the current application is hard-coded to use production-level resources. In the future, expect a separate development-level configuration.)
- An account with a Docker registry (e.g., Google Container Registry or Docker Hub)
Don't want to set this all up right now? You can also try our interactive, browser-based tutorial which is an abbreviated form of this tutorial -- you should be able to run through it ten minutes.
The Big Picture
Getting source code deployed into Kubernetes so you can test it requires the following steps:
- Build a Docker container that contains your source code and any runtime dependencies.
- Push the container to a container registry.
- Write a Kubernetes manifest. The manifest tells Kubernetes what to run (i.e., it points to the container) and how to run it (e.g., what ports to expose, or the maximum amount of memory to allocate).
- Tell Kubernetes to process the manifest, using
kubectl
, the Kubernetes command line tool.
For canary deployments, there's one additional step:
- Tell your load balancer to route traffic "asymmetrically", e.g., 90% of the traffic to one service, and 10% to another service
All of these steps are straightforward but complicated. In this tutorial, we'll walk you through each of these steps, and show how everything can be tied together.
A sample application
1. The GitHub repository
We're going to start with a sample application that consists of multiple services. Clone this GitHub repository here:
git clone https://github.com/datawire/todo.git
We've set up this application as a monorepo, with each service in its own subdirectory. The TODO application consists of a few services (tasks, search, auth) as well as a MongoDB database that stores tasks.
2. Create the Dockerfile
Each of these services has a Dockerfile that specifies how it should be built. They're all very similar. Here's a typical Dockerfile:
FROM alpine:3.5
RUN apk add --no-cache python py2-pip py2-gevent
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
WORKDIR /app
EXPOSE 8080
ENTRYPOINT ["python"]
CMD ["app.py"]
This Dockerfile starts with Alpine Linux, installs Python and the necessary dependencies to run the tasks service, and starts app.py
, the actual tasks service.
3. Create a Kubernetes manifest
Kubernetes takes care of all the details of running a container: which VM/server to run the container on, making sure the container has the right amount of memory/CPU/etc., and so forth. In order to know how to run your service, you need a Kubernetes manifest. Here's the example manifest for the tasks service:
---
apiVersion: v1
kind: Service
metadata:
name: {{name}}
spec:
selector:
app: {{name}}
ports:
- protocol: {{service.protocol or 'TCP'}}
port: {{service.port or '80'}}
targetPort: {{service.targetPort or '8080'}}
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata: {name: {{name}}}
spec:
replicas: 2
selector:
matchLabels: {app: {{name}}}
strategy:
rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
type: RollingUpdate
revisionHistoryLimit: 1
template:
metadata:
labels: {app: {{name}}}
name: {{name}}
spec:
containers:
- image: {{build.images["Dockerfile"]}}
imagePullPolicy: IfNotPresent
name: {{service.name}}
resources:
limits:
memory: {{service.memory}}
cpu: {{service.cpu}}
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
Manifests must reference an actual Docker image. Since the Docker image can change as you build new versions of your service, we're using Jinja2 templates in the manifest file to template key values.
Docker lets you add a tag an image (e.g., "latest"), which could let you statically code in a reference to an image. In practice, however, this approach prevents you from deploying multiple versions of your service live without multiple manifests.
4. Deploy
In our example application, we've gone ahead and defined the Kubernetes manifests and Dockerfiles for each of the services. So, now it's time to deploy the services into Kubernetes, from source.
If you're doing this process by hand, you'll want to:
- run
docker build
for each of your services, to build the image - run
docker push
for each of your images, to push the images to your Docker registry - run
kubectl apply
on each of your manifests, to actually deploy the images to Kubernetes
This process is somewhat tedious, so we've written an automation script called Forge that does all this work for you (and it handles dependencies, to boot). Install Forge:
% curl -sL https://raw.githubusercontent.com/datawire/forge/master/install.sh | INSTALL_DIR=${HOME}/forge sh
Then, in the same directory as the TODO application, give Forge some basic information about your username, Docker Registry, and credentials (which get saved locally):
% forge setup
Once Forge is set up, you can deploy the TODO application with a single command:
% forge deploy
5. Testing your application
To verify that the application is properly running, you can send an HTTP request. Get the external IP address of the API Gateway:
% kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api 10.11.252.234 104.154.184.70 80:31876/TCP 4d
auth 10.11.253.201 <none> 80/TCP 4d
kubernetes 10.11.240.1 <none> 443/TCP 54d
search 10.11.248.76 <none> 80/TCP 1d
tasks 10.11.251.80 <none> 80/TCP 4d
tasks-canary 10.11.243.82 <none> 80/TCP 4d
Then, send the HTTP request. We've configured the API Gateway to perform basic HTTP authentication by default, with a password of "todo". So let's pass in the authentication credentials as well.
% curl 104.154.184.70/tasks/ -u any:todo
6. The API Gateway
We've just mentioned in passing an API Gateway, so let's dive a little deeper into what's going on with the sample application. The sample application consists of a number of services, each in their own subdirectories: tasks, auth, search, and so forth.
In addition, in the root directory of the TODO application, we've configured one special service -- an API Gateway. In our example, the API Gateway we're using is a specially configured version of Ambassador, an open source API Gateway built on Envoy.
The Envoy instance is configured via the envoy.json
file, with a Kubernetes manifest in the k8s
directory.
Instead of routing all requests directly to each of the services, requests are first routed to the API Gateway, which then routes the requests to the services. This additional layer lets us:
Add authentication (and other common functionality), as we've seen. The actual authentication process is implemented as the
auth
microservice. The API Gateway is configured to issue a subrequest to theauth
microservice to validate all incoming requests.We can implement canary testing, where you can deploy new versions of a service side-by-side with an existing service, and route a fraction of your traffic to the new service.
7. Canary deployments
Canary deployments are an essential way to validate new versions of a service without impacting your overall application. Let's see how this works.
First, we'll make a modification to the tasks service. Any change to the tasks service can be canaried (e.g., deployment metadata changes or code changes). Let's simulate a change where we accidentally impact performance by adding a sleep()
call. Edit tasks/app.py
and update root()
as follows:
def root():
result = mongo.db.tasks.find(projection={"_id": False})
time.sleep(0.5) ## sleep
return jsonify({"status": "ok",
"tasks": list(result)})
Now, change to the tasks directory:
% cd tasks
And deploy the new code, with a canary:
% CANARY=true forge deploy
Now, send a bunch of HTTP requests to the API Gateway:
% while true; do curl http://104.154.184.70/tasks/ -u any:todo; done
90% of these requests will hit your original, fast service, while 10% of these will hit the slower service. This might be hard to see from the terminal, so let's try a more sophisticated approach.
8. Monitoring
In this sample application, we've also deployed Prometheus for metrics monitoring. We're going to use Prometheus to see the latency spike. Get the public IP address of Prometheus:
% kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus 10.11.245.177 35.188.142.214 80:30193/TCP 4m
search 10.11.248.76 <none> 80/TCP 1d
tasks 10.11.251.80 <none> 80/TCP 4d
tasks-canary 10.11.243.82 <none> 80/TCP 4d
todo-db None <none> 27017/TCP 4m
Go to the external IP in your browser, which will display the Prometheus dashboard.
Paste the following query into Prometheus:
{__name__=~"envoy_cluster_tasks_upstream_rq_time_timer|envoy_cluster_tasks_canary_upstream_rq_time_timer"}
You'll see how the envoy_cluster_tasks_upstream
latency is much lower than the envoy_cluster_tasks_canary_upstream
(you may have to hit execute a few times to get enough data on your graph to see the difference.)
Recap
In this tutorial, we've shown how, as a developer, you can build a fast, productive workflow for developing, deploying, and testing microservices on Kubernetes.
We've also shown how this workflow can be used for a canary deployment, and integrate metrics collection into a dashboard.
We've also built this workflow without requiring additional operational primitives, i.e., we're just using Kubernetes and Docker and the Docker Registry.
Note: this document is a draft. If you run into bugs, we'd be happy to help. Send us an email at hello@datawire.io or join our Gitter chat.
Next steps
Try this with your own services. If you're creating a new service, you can:
Clone the GitHub repository:
git clone https://github.com/datawire/todo.git
Copy an existing service into a new directory:
cp -r tasks/ new_service/
Update the
service.yaml
to reflect the name of the new service.Edit the
envoy.json
to reference the new_service. You can do a search-and-replace on "tasks" and replace it with "new_service".Run
forge deploy
in the main directory.
Questions?
We’re happy to help! Look for answers in the rest of the Microservices Architecture Guide, start using our open source Ambassador API Gateway, send us an email at hello@datawire.io, or contact our sales team.