Linkerd on GKE

We have been hearing a lot about service mesh recently, coinciding with the fact that at Transit we are parting ways with our APM vendor in favour of open source monitoring tools, the time seemed right to jump on board with the “world’s most over-hyped technology".

While we were already capturing golden signals at the ingress level (check out this dashboard by @n8han that we contributed upstream), the realization that we would be losing service to service metrics prompted us to implement a mesh.

Being a GKE shop, the managed Istio add-on was very appealing, but in the end it was a mix of this blog post, the amazing slack community, the performance benchmarking results, these differences, the overall ethos of less is more and last but definitely not least, the relative simplicity of operating Linkerd that helped us decide.

Shoutouts to Istio though! It was a tough choice, but enough blog spam now, let’s get to it!

Kick the tires

We were able to take Linkerd for a quick spin with the linkerd cli tool. After a quick run through of their getting started guide to install the mesh, we decided to add some of our own workloads to it to get a better sense of what Linkerd offers out of the box.

We did so by:

1. Annotatating the namespace whose workloads we wanted to add to the mesh:

kubectl annotate ns namespace-with-http-workloads linkerd.io/inject=enabled

2. Restarting all workloads in that namespace:

for type in deploy sts ds; do
  for workload in $(kg $type | awk ‘{print $1}’); do
    kubectl rollout restart $type “$workload”
  done
done

3. Running linkerd dashboard and proceeding to drop our jaws! :)
What I mean by that is that we were quickly sold on it after seeing the gold mine of metrics Linkerd provides.

Revert linkerd for now

But now, we wanted to revert this, because the next step for us was to deploy Linkerd in a declarative, automated and repeatable manner (i.e. via a pipeline), as we strive to do with anything that gets installed in a k8s cluster. (This approach to deployment serves as a major part of our disaster recovery strategy.)
We uninstalled Linkerd by:

1. Removing the injection annotation:

kubectl annotate ns namespace-with-http-workloads linkerd.io/inject-

2. Restarting all the workloads again.

3. Uninstalling Linkerd via thelinkerd cli.

Automated, declarative, repeatable deployment of Linkerd

For those who Terraform apps

If you use Terraform to deploy apps to your cluster, checkout the Automating Linkerd installation on Kubernetes in Terraform tutorial by Chris McKeown.

For the rest of us

We however, for better or for worse have decided to draw the (subject to moving) line for Terraform at cloud infra resource (e.g. GKE cluster) management, and decided to manage apps separately via a combo of Cloud Build and helmfile (which is essentially a helm for your helm) by means of the community-maintained helmfile cloud builder.

Certificates and Helm

The first thing the instructions on installing Linkerd via helm mention is the prerequisite:

The identity component of Linkerd requires setting up a trust anchor certificate, and an issuer certificate with its key

Linkerd docs go into this in much more (and needed when you are actually doing this) detail, but tl;dr is we created:

1. a signing key pair certificate and shoved it in a k8s secret
2. a cert-manager CA issuer that references the secret from 1.
3. a cert-manager Certificate that references the Issuer from 2. and creates a k8s secret with the resulting certificate (tls.ca, tlskey and tlscrt).

While that’s the gist of it, I’d like to re-iterate that the official docs walk you through this in detail.

Then, we just used helmfile in our cloud build gke config pipeline and bingo bango we had linkerd installed the way we like to install things.

Add long-running http/grpc workloads to the mesh…

At this point we are sold on Linkerd and have figured out our deployment strategy. It’s go time!

There are 2 main approaches to adding services to your mesh:

1. Catch-all annotation at the namespace level, and selectively remove workloads that you do not want in the service mesh by overriding them. IMHO this makes sense when most (or all) of a namespace’s workloads are destined for the mesh.

2. No namespace level annotation, but instead just inject workloads explicitly. Some teams are more comfortable with this, especially since this might be a new tech that everyone is just familiarizing themselves with.

At the end of the day we ran this by the teams and decided on a namespace basis how to approach it, but ended up getting most of our internal unencrypted http workloads into the mesh.

I’d recommend to embrace the quick wins that bring lots of value in the mesh, namely:

• Http/grpc servers
• Http/grpc clients that call those servers unencrypted*

* If these clients initiate calls immediately upon starting up (before any networking related health check for example) you will have to implement a retry strategy, or add linkerd-await to the app’s Dockerfile.

However we did notice a couple of issues in staging and even some that only poked their heads in production.

Don’t mesh it all

Implementing a service mesh is a significant change and introduces risk, but that risk can be greatly reduced by trying it out in a test environment to see what breaks. However some problems may only show their heads in production.

Here’s a list of some of my recommended workload/protocol/port exclusions:

• Https clients (by skipping outbound port 443 in linkerd config). We noticed some new intermittent and unexplainable errors with regards to external https queries. That and the fact that not much value is gained today as linkerd-proxy cannot inspect https requests anyways.
• Non http protocols. After seeing a few of these break as expected such as meshed redis sentinel clusters, cassandra clusters (internode communication) and mysql on a non-standard port, we basically play it safe and skip all non-http protocol ports
• (Cron)Jobs, Argo workflow pods, and any short lived pods. Sidecars just don’t terminate easily for these sort of workloads. Until kubernetes recognizes sidecars by means of something like this proposal, we’ll just leave them out of the mesh to keep it simple. People who really need them meshed can make it work.
• kube-system workloads… not meshing those anytime soon.
• Open source/3rd party monitoring services. With the exception of Linkerd itself which ships meshed naturally, I am on the fence about putting other monitoring services in the mesh. They are expected to work, and they are not the most interesting to gather metrics on. I’d wait a while before (if ever) adding these.
• …There may be more, but that’s all that comes to mind!

Regarding all these excluded workloads/ignored ports, we will re assess periodically as Linkerd continues to evolve(e.g. TCP mTLS is in the works at the time of writing this) and as we continue to familiarize ourselves with the tech.

Conclusion

We are very happy with the goldmine of metrics we now have on a per service level thanks to Linkerd! But our service mesh journey has just begun. For example at this time, we have not yet added our ingress-controller to the mesh as we continue to familiarize ourselves with Linkerd, nor have we tried traffic splitting (canary/blue-green) nor have we set up service profiles to get per-route metrics*, etc.

*edit: We have begun leveraging (auto-created) service profiles to implement infra-level retries where needed — quite happy so far! 👍

Thanks for reading!