Jenkins and Spinnaker in GCP: Married for 7 lives
When it comes to CI and CD tools, the list starts with Jenkins and Spinnaker respectively, if not they always get placed within the top 5, and in our experience every 4th customer in the industry prefer to use open-source tool for CI and CD to get most of it. Since Jenkins and Spinnaker both are open-source and offer a vast variety of customization and capabilities even after having several managed CICD tools in the market, enterprises still love them. The project I am currently working on uses Jenkins monstrously for the infrastructure and application deployment automation over the different clouds.
In this article, we will be demonstrating how to leverage Jenkins(CI) and Spinnaker(CD) to automate your application production deployment flawlessly in GCP.
This blog is a collaboration of Rahul Kumar Singh and Rohan Singh, Senior Cloud Infrastructure Engineer at SADA comes with wonderful experience in Google Cloud, DevOps, Infrastructure, and Automation. Also, he is a Google Cloud Champion Innovator for Modern Architecture and actively contributes to the Google Cloud community. We have plans to write more blogs in the future. Right now we are also working together on a company project as well as writing technical blogs in collaboration.
Read continuation article of this blog explaining the hidden features of Spinnaker in-depth here.
Prerequisites
- Google Kubernetes Engine (GKE) Cluster
- GCS bucket, run to create a bucket
- A Service Account with Storage Admin, Logging Admin, and Monitoring Admin (for Spinnaker)
Jenkins
Jenkins is one of the most famous open-source CI/CD tools out there. You can integrate Jenkins with the versioning-control tool to trigger the Jenkins pipeline upon new commits, which could be a branch-based or folder-based trigger, and do according to the pipeline script. Jenkins pipeline can be written in Declarative Syntax Language or Groovy.
Rohan has written a Jenkins installation blog on Google Compute Engine and Google Kubernetes Engine thus installation is not covered in this blog.
CI/CD with Jenkins on Google Compute Engine (GCP)
Setting up Jenkins on Google Kubernetes Engine with Helm
Spinnaker
Spinnaker is a famous, free, muti-cloud, and open-source CD tool in the industry. It provides application management and deployment with an flexible pipeline management system for rapid and easy deployment to production. Spinnaker offers you tons of deployment capabilities like canary deployment, blue-green deployment, manual approvals to production servers, tool integrations, rollback capabilities, etc.
Did you know? Spinnaker is originally developed by Netflix and extended by Google.
Spinnaker Installation
Spinnaker installation is fairly straightforward and easy. Run
kubectl -n cicd apply -f https://gist.github.com/Rahulks1807/928f2a79fa609684c4b0817938a37074The gist will deploy the number of pods with services required to run the spinnaker successfully. You will see one external load balancer.
We have deployed Jenkins and Spinnaker in the cicd namespace. Best practice suggests installing the CI/CD tool separate namespace.
You can see the pods and services in GKE Console Workloads and Services & Ingress respectively. Hit the public IP on the 9000 port to see Spinnaker
Workflow (small explanation)
With Jenkins and Spinnaker, we are deploying our application on GKE where Jenkins will perform all the CI tasks — cloning a source code repo, then building, tagging and pushing a docker image to a container registry(gcr), once the image being pushed to GCR, spinnaker takes over. Basically, the spinnaker monitors our GCR and gets triggered every time there is a new version of an image. It will deploy an app to GKE using that image. We will manage deployments (Blue Green and Canary) from Spinnaker.
Currently, spinnaker does not have any direct integration for Google Artifact Registry to docker images hence we have gone with GCR, and we are trying to find some workaround to get this done.
We have not covered Spinnaker in detail in this blog, Rahul will cover Spinnaker along with its components in detail in his next blog dedicated to Spinnaker only.
Blue Green Deployment with Spinnaker
In blue-green deployment, we run two versions of an application simultaneously, the old version is known as blue whereas green is the new or upgraded one. We roll out traffic to the green one gradually with blue as a standby in case of rollbacks.
To configure Blue Green, from the spinnaker homepage, create an application from the Actions dropdown. Fill in all the details and hit Create.
You can see your app in the Applications section, open it to see other details and configure blue-green. From PIPELINES➞Create, enter Pipeline Name.
We have two flows for Blue Green Deployment:
- Auto Switch (suitable for non-production environments)
- Non-auto switch (suitable and recommended for production environment as it needs approval; test and manually approves it)
- We always recommend to the client the latter one to avoid any uncertainties. We have the same in this blog.
Now configure your pipeline, it has multiple stages. As per need multiple stages can be added.
In the first stage, you configure the trigger which looks for Docker Image in GCR. If your pipelines expect some artifact you can configure it in Expected Artifacts, notification configuration can also be done at the same stage.
Moving to the second stage, in this stage set our Manifest file along with execution Options. The first snapshot shows the manifest configuration, if you see in our manifest we have the value of the image key has ${trigger.tag} which actually fetches the image and tag info from the first step.
The second snapshot shows the rollout strategy and execution options. Once you enable Rollout Strategy Options, choose your application namespace, services that would be a part of blue-green, don’t enable Send client requests to new pods and keep the strategy None. The reason for selecting None is to create a new replica with no traffic movement, it will be the green deployment post-approval.
Execution options are here to select the outcome of the stage, how often the would-be run and how it would react upon failing.
The third stage is Manual Judgement where we configure our Manual approval. The stage is automatically configured to its best thus no configurations are needed.
The last and final step is Patch where the patching strategy gets configured. Fill in the values for Resource To Patch Options and enter Patch Content. The patch content would only contain the values needed to be patched. Keep rest as it is.
We are done with configuring the Blue Green pipeline, now run the Jenkins pipeline or push code to your GitHub or whichever source control tool you are using, spinnaker would perform Blue Green now and you would have to Manually Approve it.
Canary Deployment with Spinnaker
Canary Release deployment is a beneficial deployment strategy if the release is an early stage of an application and needed to be released for a small user cohort to test features, and get feedback parallel to the production, once the release is accepted and successful, it’s being released entirely. Canary Deployment is widely done by big enterprises to test new features or capabilities of an app.
Start by creating an application the same way we did in Blue-Green Deployment. From the application page open your app and from Config, start configuring your app.
Enable Canary in the Features section, rest sections can be left as it is.
Once done, go to Delivery, you will see Tabs (Pipelines, Canary Configs and Canary Reports). From Canary Configs, add configuration. Add required group in Metrics section. These metrics would be used for Canary Analysis, in our sample case we have added CPU and Memory.
Scoring is used for weightage i.e. how much percentage are necessary to analyzed and total of both will determine the success and failure of canary analysis. In our sample case, we have put 50 50 for both.
Now pipeline needs to be configured. From Pipeline tab, create a pipeline. Below screenshot shows our pipeline workflow b/w stages and will be explaining what goes in canary as rest step are quite straightforward to fill.
As per the screenshot, the first two stages are Find Initial Version and Deploy Canary Config.
The Find Initial Version stage looks for any existing deployment of the app that you are deploying with canary whereas the Deploy Canary Config actually deploys the canary config i.e. new version of the same app. In our sample case we are looking for a configmap for the specific app, if it exists it deploy initial version else canary. Why this is so? Because every new version would have configuration change needs to be deployed with the app i.e app’s newer version.
The below screenshot shows the configurations of Find Initial Version and Deploy Canary Config respectively.
The 2nd set of stages have Deploy Initial and Deploy Canary, the configuration is similar to what we did in blue-green deploy stage. This is deploying your initial and new version of the app.
The next stage — Canary Analysis is an important one, this is where magic happens. In this stage, the lifetime option does the data collection and analysis every 5 minutes, the interval is the frequency at which a canary score is generated, step is determining what steps are getting performed every minute, loopback type is how do you want to test an application and rest you can leave as it as or blank.
Baseline + Canary Pair is here to select the version of the app you want to split traffic in between.
Scoring Thresholds is the parameter in percentage of success and failure of the canary analysis.
Metrics Account in Advanced Settings is what actually collects the metrics for the canary analysis of the application. This configuration is the part of Spinnaker deployment and we are collecting metrics from Stackdriver, you may configure Prometheus, New Relic etc as your metrics collection agent.
Delete Initial will delete the deployments on the basis of the canary analysis. The below screenshot show the configurations of it.
The Manifest section letting the canary know what to delete along with the grace period.
Conditional on Expression will delete the initial version if and only if canary result are successful.
Last stage is only for letting the user know Canary deployment is successful.
The below screenshot illustrate how the canary deployment pipeline looks like. In our case we have deliberately failed the execution that’s why it’s red, it will be all green though for full success pipeline.
When the pipeline is successful you would see the canary stage with all greens and have the report of canary analysis in the “Canary Report” tab in application home page.
Now we are done and that’s how you configure two major and popular open-source tools for CICD at production level.
You can read more about the feature of spinnaker that were not part of this blog here.
I (Rahul) will publish one proper and dedicated blogs on Spinnaker in depth soon. Till then follow me, read my other published blog here:
Read Rohan’s blog
Connect with us on LinkedIn: Rahul Kumar Singh and Rohan Singh
