Airbnb only released code using Deployboard, an in-house tool we created over the last eight years, until a year ago. Deployboard allows over 1,000 developers to contribute to our monolithic Ruby on Rails software, which we call Monorail, by coordinating continuous integration (CI), Git merges, and deployments to pre-production environments.
Deployboard, unfortunately, does not offer flexible deployment pipelines. To offer this automation, we began migrating from Deployboard to Spinnaker, an open-source continuous delivery pipeline management technology developed by Netflix and Google.
Here’s how the project has progressed thus far, along with some lessons learned.
Breaking up the monolith
Airbnb has been migrating from a monolith to a service-oriented design for the past two years (SOA). As a result, the number of services and the number of teams that manage and deliver their own services has expanded. More integration test plugins and automation were written inside and on top of Deployboard by service owners.
Even with some increased automation, the number of team runbooks began to increase. Over the course of hours or days, several teams completed 20 or more manual steps. A large number of issues were caused by operational errors and oversights during releases.
Figure 1. The number of projects in Deployboard over time has increased rapidly as we move to a service-oriented architecture.
Our Deploy Infrastructure team realized that by employing tooling to aid the rollout process and an automated deploy pipeline, many accidents might be avoided. We prototyped this logic in Deployboard but recognized that instead of constructing our own, we should use a platform developed from the ground up to organize deploy pipelines. Using an established tool would allow us to gain more powerful features right away (such as canary analysis) while also reducing the amount of bespoke code our team would have to maintain in the future.
The move away from Deployboard
Our team expected to have to port certain Deployboard functionality to any new deploy solution. Deployboard displays detailed CI results and build artifacts, includes particular support for in-house Kubernetes custom resources, and integrates with numerous in-house integration test frameworks that we’ve developed over time. When switching to a new tool, we didn’t want to lose these features.
We chose Spinnaker to replace Deployboard in part because we could quickly plug in new logic to fill in functionality holes without forking the core code. As a collection of bespoke extensions, we’re integrating important Deployboard functionalities into Spinnaker.
Extending Spinnaker’s user interface
We customized Spinnaker to look similar to what Airbnb engineers were used to in Deployboard to make the onboarding experience with Spinnaker less jarring (see the figures below).
When developers use Deployboard and look at their project, they choose the version to deploy from a list of options. In Spinnaker, we mirrored this original view. This familiarity allows developers to get started with Spinnaker right away.
Figure 2. This example of the snapshots view in Deployboard shows code versions on the master branch in chronological order.
Figure 3. Here’s an example of the same snapshots view in Spinnaker. The similarity to Deployboard lets developers transition easily to Spinnaker.
This and other changes may be made in a standalone Airbnb module that connects to Deck, Spinnaker’s UI service.
How we extended Spinnaker’s functionality
We’ve been enhancing Spinnaker’s functionality with Kubernetes tasks, custom webhooks, and Spring Boot components that have been introduced to back-end Spinnaker services, in addition to the user interface improvements.
It’s simple to set up Kubernetes jobs and combine them with a Spinnaker pipeline. Custom jobs are used by new Spinnaker teams to automate non-standard aspects of their deployment process. These teams have complete control over the job’s compute resources, retry policies, and permissions.
Some teams have already established services to execute integration tests before Spinnaker. We constructed a webhook stage to call services that use Airbnb’s service interface specification language to onboard these test runners into Spinnaker (IDL). We were able to more easily plug new and existing services into Spinnaker by making this native to both Spinnaker and Airbnb.
Figure 4. Orca, Spinnaker’s orchestration service, can perform operations in other Airbnb services using a custom webhook plugin.
We had to supplement existing features in Spinnaker in addition to adding new ones. Airbnb-specific Kubernetes resources, for example, require custom logic to detect when they have stabilized. By putting bespoke Spring Boot components in a build alongside the core Spinnaker library, we were able to alter this logic in Spinnaker without forking it. We were able to easily alter functionality without needing to fork Spinnaker code by adding our custom components in a build alongside the core Spinnaker library.
Figure 5. Sketch of a Kubernetes custom resource handler for Spinnaker. Components such as this let us add features to Spinnaker without changing any core code.
Final results
In January 2019, we began integrating Spinnaker with Airbnb. We steadily onboarded early customers, starting with internal services with minimal traffic, after implementing our initial UI extension and expanding the built-in Kubernetes functionality. With Deployboard, we were able to close important gaps thanks to consumer input. We began onboarding a varied range of production services after Spinnaker fulfilled our fundamental use cases, ensuring that we had all of the capabilities required for Airbnb-wide adoption.
By October, we had deployed over 30 services using Spinnaker, including those that were relatively large and had over 40 active developers. We track progress by measuring regressions that are discovered before full production deployment, in addition to adoption and customer feedback. When our early adopters use Spinnaker, they have fewer regressions promoted to production and spend anecdotally less time managing their deployment process. With these case studies, we’ll be able to more easily encourage Airbnb to embrace Spinnaker.
The next phase
We’re concentrating on ensuring that Spinnaker prevents regressions, improves productivity for our early adopters, and addresses some functionality gaps with Deployboard through the end of the year. In the interim, we’re running scalability tests to ensure that Spinnaker can manage all of Airbnb’s deployments. By 2020, we want to have onboarded over 1,000 services and be able to contribute more of our unique logic to Spinnaker, which is open-source.
For more info: https://www.qaaas.co.uk/testing-services/
Also Read: https://www.guru99.com/software-testing.html