• For devs

Devgun: Creating Development Environments With Kubernetes

Derrick Wippler
5 min read

When we first approached the problem of creating local development environments, we reached for common tools like vagrant. But as with most vagrant-built environments, build times are long. This means vagrant images have long lives and tend to drift overtime.

Our services and infrastructure change constantly, so the vagrant build was almost always broken. We wanted to avoid the traditional “new developer right of passage,” which easily entails days of work to get vagrant to successfully build a new development VM.

Ideally, new developers can be productive within hours of hire, not days or weeks.

All of this was compounded by the fact that we have a lot of micro services that need to run in the development environment and integrate seamlessly.

The goals of the project were simple:

  • Setup and destroy a dev environment quickly and in a reproducible manner.

  • Single step development environment setup and teardown.

  • Start and stop services, including their dependencies, at will.

Evaluating orchestration platforms

Since our micro services were already running in containers, we knew we needed to choose an orchestration system.

Several devs on our team were interested in Mesos Marathon, so we tested that out first. Unfortunately, the memory and hardware requirements required to run marathon in a small local development environment were far too restrictive for our purposes. At the time we tested, minimesos was also unable to correctly network containers in Marathon to the local network – a major roadblock for our project. These issues made it clear that Mesos would be impractical for our needs.

Next, we evaluated Kubernetes and Minikube as we were attracted to the vibrant community around these projects. We found the memory footprint to be minimal, and networking worked perfectly. Most of the features that interested us worked out of the box, and getting our third-party applications running inside Minikube proved easier than expected.

So with Minikube as our chosen platform, we were able to move forward with the project quickly and spend most of our time developing what would become Devgun.

Building a CLI

To accomplish our project goals, we decided to build an all-in-one command-line interface written in Golang. We wanted a CLI that could be distributed as a single binary to our developers, so we created one called Devgun.

Devgun makes no assumptions about the development environment. It automatically downloads all the tools it requires, including Minikube, kubectl, and Mailgun-specific tools, into the /.devgun/bin directory. All you need to do is add your path after the initial setup has been completed. Installing everything under a single directory makes uninstall as simple as rm -rf ~/devgun.

We compiled all the kubernetes manifest files into the static binary using the go-bindata project, thus enabling a single file download and run to install a development environment.

Running Devgun looks like this:

1$ devgun start
2Pulling kubectl [done]
3Pulling minikube [done]
4Writing Minikube configuration [done after 0.702698 seconds]
5Booting Minikube environment [done after 77.297039 seconds]
6Waiting for Kubernetes.. [done]
7INFO[0148] Setting up minikube routes -- enter your sudo password if prompted
8Modifying routes [done after 0.012867 seconds]
9Adding kube-dns resolver entry [done after 0.002070 seconds]
10Applying kube-dns-external configuration [done after 0.212881 seconds]
11Enabling Heapster [done after 37.506335 seconds]
13Minikube is now configured and running....
15# Set up path for our bin directory with dev tools
16export PATH="$PATH:/Users/thrawn/.devgun/bin"
18Next you should choose a mailgun service to start using <code data-enlighter-language="generic" class="EnlighterJSRAW">devgun service start</code>
20* Use <code data-enlighter-language="generic" class="EnlighterJSRAW">devgun service list</code> to see a list of services to start
21* Use <code data-enlighter-language="generic" class="EnlighterJSRAW">devgun service start</code> to start a service and its dependencies
22* Use <code data-enlighter-language="generic" class="EnlighterJSRAW">kubectl get pods</code> to list running pods
23* Use <code data-enlighter-language="generic" class="EnlighterJSRAW">kubectl describe pod <pod-name></code> to inspect a pod and see the pod log
24* Use <code data-enlighter-language="generic" class="EnlighterJSRAW">kubectl log <pod-name></code> to see the container logs

In addition to creating a Kubernetes environment, Devgun also adds some network routes to the local box. For instance, on OS X, the following routes are added:

route -n add <minikube ip> route -n add <minikube ip>

This allows our developers to run and test applications on their local boxes with full connectivity to containers in the k8 cluster as if they were running in the cluster themselves. We also enable DNS lookup by utilizing the resolver functionality built into OS X. We create /etc/resolver/local with the content:

search cluster.local svc.cluster.local default.svc.cluster.local nameserver

Thus enabling DNS resolution of kubernetes services.

1$ devgun service start etcd
2Starting service: etcd [done after 7.084730 seconds]
4$ ping etcd
5PING etcd.default.svc.cluster.local ( 56 data bytes
664 bytes from icmp_seq=0 ttl=63 time=0.220 ms
8--- etcd.default.svc.cluster.local ping statistics ---
91 packets transmitted, 1 packets received, 0.0% packet loss
10round-trip min/avg/max/stddev = 0.220/0.220/0.220/0.000 ms

Managing custom and third-party services with a spec

Because Devgun has to manage a mix of custom and third-party services, we developed a custom yaml spec file we define for each service. The spec file looks something like this:

2 name: scout
3 resources:
4 <<: *k8s_deploy_resources
5 depends_on:
6 - name: vulcand
7 - name: cassandra
8 - name: kafka-pixy
9 - name: kafka
10 - name: etcd
11 - name: metrics
12 hooks:
13 pre:
14 start:
15 - exec: >
16 cqlsh -e "CREATE KEYSPACE IF NOT EXISTS scout WITH REPLICATION={'class':'SimpleStrategy','replication_factor':1}" $POD_IP
17 where: service:cassandra:cassandra

It defines a custom service called ‘scout’ which reads events from Kafka and records analytics about the events in Cassandra. The spec tells devgun which custom and third-party services it depends on and which hooks to run on different containers before starting the scout service.

The resources key references the kubernetes manifest files like config map, deployments, and service descriptions defined elsewhere in the spec file that devgun uses to start the containers in K8.

Where do we go from here… future work

When we designed Devgun, we started from the bottom up without consideration of the bigger development and deployment stories. Now that we’ve had time to use Devgun, we find ourselves wondering if we got the abstractions correct.

We started asking questions: What is the experience we want our developers to have when they sit down to create and deploy a new service? Where do the responsibilities of the developer end and operations begin? How much of operations can we automate and how much requires human interaction? How do we facilitate the passing of information between developer and operations?

We’ve started to define stories and personas that will shape how Devgun evolves going forward. In my next post, I’ll share some of our work and thoughts on how the developer experience could evolve as we continue to work with Kubernetes and Devgun. Be sure to subscribe to our blog so you don’t miss out.


Learn about our Deliverability Services

Looking to send a high volume of emails? Our email experts can supercharge your email performance. See how we've helped companies like Lyft, Shopify, Github increase their email delivery rates to an average of 97%.

Learn More

Last updated on May 17, 2021

  • Related posts
  • Recent posts
  • Top posts
View all

Always be in the know and grab free email resources!

No spam, ever. Only musings and writings from the Mailgun team.

By sending this form, I agree that Mailgun may contact me and process my data in accordance with its Privacy Policy.

sign up
It's easy to get started. And it's free.
See what you can accomplish with the world's best email delivery platform.
Sign up for Free