CI/CD and IaC
What are the opportunities and challenges?
The integration, delivery and deployment of code are state of the art today, but they involve various non-technical challenges that need to be overcome.
In this blog post, we introduce Continuous Integration, Continuous Delivery and Continuous Deployment (CI/CD) and show what added value can be gained by using them. The article also explains what role CI/CD plays in the Infrastructure as Code (IaC) approach and which deployment variants exist on-prem and in the cloud.
Introduction - Concept CI/CD
In the time before CI/CD, the integration of changes to a digital product rarely went smoothly. The holistic consideration and testing of all dependencies between new and existing code proved to be difficult, and often time-consuming reworking of the code was necessary before it could finally be integrated into the productive system.
The processes, techniques and tools of the CI/CD concept create a premise that enables the continuous and immediate integration of new or changed code into the existing solution. Using CI/CD, applications can be continuously monitored in an automated manner throughout their entire life cycle (Software Development Life Cycle, or SDLC for short), from the integration phase, through the test phase, to the delivery and deployment of the application. The practices used in CI/CD are collectively referred to as the "CI/CD pipeline." The development and operations teams working according to the DevOps approach are thus supported.
CI/CD in the use of IaC
The use of a CI/CD pipeline is also of central importance in the Infrastructure as Code (IaC) approach, among others. In the context of IaC, in addition to the automation logic, various properties of hardware- and software-based infrastructure components from the cloud or on-prem are stored in a repository. Such properties stored in files do not have to be 1:1 copies of current configurations. Parameterized files, for example in YAML notation, are often preferred. The big advantage is that the information in the configuration files in the repository, despite a high degree of automation, can still be read (human-readable) and understood by a person. Whether human-readable or not, in any case the CI/CD pipeline is the prerequisite for an automated, transparent and continuous validation of configuration files and the subsequent integration of changes in the infrastructure.
Continuous Integration (CI)
CI consists of the automated process that ensures the continuous integration of customizations to an application. Continuous Integration supports developers in regularly making and publishing changes to the code. Automated testing processes such as code analysis or unit tests (code tests) ensure that edited branches in the code repository can be merged so that the application continues to function. The branch of a repository is part of the development process and can be viewed as a branch referencing the existing, validated code. If developers need to make changes to the existing code, a new branch is first created in which the change or enhancement is made. A merge is then performed to merge the final code changes with the existing code in the repository.
In summary, the successful CI contains the automated
- Creating the customized application (build)
- as well as their testing (test)
- and merging the changed code into a code repository such as GitHub (Merge).
Unless otherwise configured, a merge is only successful when all defined processes (jobs) within the "CI/CD pipeline" have been successfully run through.
Continuous Delivery (CD)
The abbreviation "CD" has two meanings. On the one hand Continuous Delivery, on the other Continuous Deployment. These similar concepts are often used synonymously. Within the framework of both concepts, the automated processes that follow CI are designed in the pipeline. In some cases, the terms Continuous Delivery and Continuous Deployment are used to specify the characteristics of automation. What this means is explained in this blog in the section Continuous Deployment. A CI integrated into the pipeline is the prerequisite for an efficient CD process. Continuous Delivery enables automated deployment of applications to one or more environments after successful code validation by the CI. Developers typically use environments for the build, test and deploy phases. Possible processes (jobs) from the different phases are many.
For example, this can be
- creating a build of the application,
- managing and schema validation of environment variables that define an entire infrastructure or individual component of it,
- testing and resetting an environment or specific components of it if the tests fail
- and many other activities.
Successfully passing through all Continuous Delivery phases allows the operations team to quickly and easily deploy a validated application or infrastructure configuration to production.
Continuous Deployment (CD)
Continuous Deployment is an extension of Continuous Delivery. Continuous Deployment enables the automation of releases of customizations of an app or even infrastructure configuration files. Continuous Deployment thus allows a seamless and fast integration of code adaptations into a productive environment, sometimes in just a few minutes. This is made possible by sophisticated and extensive automatic tests within the various phases of the CI/CD pipeline. However, fully automated deployment does not always make sense. If the automatic testing elements of the pipeline cannot cover compliance with all guidelines and thus ensure governance due to certain circumstances, deployment must be done manually. At least until the necessary premise has been established. This can be enabled by means of improvements such as standardization, adapting existing processes (if possible), resolving external dependencies or taking them into account by means of access via an interface, etc.
CD/CD and IaC deployment variants
In the cloud as well as on-prem, there are various use cases that are suitable for CI/CD. For example, the DevOps team can distribute serverless or fast, automated code updates using container technology. Depending on the target architecture and the scope of the adaptation, different distribution approaches are available. The following list contains the three best-known, which can be used in the cloud and on-prem.
|CI/CD - Distribution approaches|
|Blue-Green deployment provides for development on parallel dedicated infrastructures. The production environment (blue) contains the latest, working version of applications or adjustments to the infrastructure. In the staging layer (green), the customized applications or the infrastructure modified by customized configuration files (IaC) are extensively tested for their functions and performance. This deployment approach enables efficient implementation of changes, but can be cost-intensive, depending on the complexity of the production environment.|
|Rolling deployment is used to distribute adjustments incrementally. This reduces the risk of failures and enables simple rollbacks to the previously functioning state in the event of problems. Consequently, as a prerequisite for rolling deployment, the services must be compatible with the old and new versions. Depending on the case, these are application versions or, in the case of IaC, infrastructure configuration files.|
|Side-by-side deployment is similar to Blue-Green deployment. In contrast to Blue-Green, the changes are not distributed across two environments, but are made available directly to a selected user group in production. Once the user group confirms the functionality and performance previously tested in the CI/CD pipeline, the updates can be deployed to all other users. This allows developers to run different versions in parallel, just as with rolling deployment, and additionally to gather real user feedback without high risk of downtime.|
A recommendation of the deployment variant is situational and depends on the product to be deployed and the infrastructure.
Below we describe an exemplary excerpt of solutions for CI/CD and IaC projects:
|Terraform||Project by HashiCorp, which is very flexible to use and compatible with well-known cloud providers such as AWS, Azure, GCP and OpenStack.|
|Ansible||Project from Red Hat, which is an orchestration and configuration tool that enables the automation of repetitive and complex processes using playbooks.|
|AWS CloudFormation||IaC service that enables managing, scaling, and automating AWS resources using templates within the AWS environment.|
|Azure Resource Manager||IaC tool of the Azure environment, which enables, among other things, the deployment and management of Azure resources using ARM templates.|
|Google Cloud Deployment Manager||Infrastructure deployment service from Google that enables the creation, provisioning and configuration of GCP resources using templates and code.|
|Chief||Well-known IaC tool that can be used in AWS, Azure and GCP together with Terraform due to its flexible deployment options and provisioning of its own API.|
|Puppet||Similar tool to Chef, which is also commonly used for monitoring defined and provisioned IaC properties and automatically correcting deviations from the target state.|
|Vagrant||Another solution from HashiCorp, which enables rapid creation of development environments and is aimed at smaller environments with a small number of VMs.|
In the hybrid cloud environment, the in-house CI/CD and IaC solutions of cloud platforms such as AWS, GCP and Azure are usually not sufficient. For example, Terraform and Ansible can be a suitable solution for IaC due to their high flexibility and compatibility, especially in multicloud environments.
Find out what else there is to consider regarding multicloud scenarios in our blog post about multicloud.
Implementation - opportunity and challenges
Various opportunities can be profited from the implementation of CI/CD. The most important opportunities are summarized in the following list as examples:
|Shorter time to production and fast feedback - Automated testing and validation in the CI/CD pipeline eliminates tedious, manual and therefore time-consuming steps.|
|More robust releases & earlier detection of errors (bugs) - Extensive testing of code and functions. Simple errors are avoided.|
|High visibility (transparency) - Using the CI/CD pipeline, individual test results can be checked in detail. If defects or errors are discovered in new code, this is shown transparently.|
|Cost reduction - Reduction of costs based on reduction of simple errors. In the long term, the use of CI/CD is less error-prone due to automation and thus sustainably cheaper.|
|Increased customer satisfaction - The consistent and reliable development process results in more reliable releases, updates, and bug fixes. This increases customer satisfaction.|
However, various challenges must also be mastered for the implementation and operation of CI/CD. One of the biggest and most important is standardization in the company's own infrastructure. It determines the degree of automation to a large extent. In general, it can be assumed that a homogeneous infrastructure enables a high and cost-efficient degree of automation. Heterogeneous environments should be standardized as far as possible if high automation is the goal. It should be noted that introducing or merely increasing standardization can be a major undertaking in itself. Other challenges that need to be mastered are:
|Adaptations in the corporate culture - CI/CD is used and lived in the context of agile corporate cultures and approaches, especially DevOps. Consequently, teams must be comfortable and familiar with the iterative, agile way of working.|
|Expertise - Correct implementation of CI/CD requires a lot of expertise and experience. Not only in the technical area, but also in the organizational area.|
|Reactive Resource Management - To ensure performance across all of CI/CD's automated processes even under increased demand, resource management should be monitored and responsive.|
|Initial development costs - The initial expenses for a development environment, the build-up of know-how, conceptual design, standardization and process adaptation can be high, but are justified by the added value gained from CI/CD.|
|Microservice Environment - To ensure high scalability, a microservice architecture is ideally built. Those responsible for the architecture must be aware of the accompanying increase in complexity and dependencies and the requirement for their administration.|
The setup and design of CI/CD depends on the setup in one's own development and infrastructure environment. In our estimation, there is no ready-made concept for the CI/CD pipeline. Which tests and validations are implemented in Continuous Integration, which phases are implemented in Continuous Delivery and how the distribution is carried out in the context of CD must be individually determined and conceptualized despite various best practice approaches. At least at the beginning, the necessary know-how in the various specialist areas such as software development, quality assurance and, especially in the IaC approach, the existing expertise in the various infrastructure areas is often neglected.
The IaC approach offers a visionary solution that has already been tested in practice and enables infrastructure and its services to be designed adaptively, easily maintainable and securely both on-prem and in the cloud. CI/CD is a core element that enables the correct and transparent mapping of configurations from a central repository to infrastructure components. As explained in this blog post, the use of CI/CD is worthwhile. The cost-effective and short implementation times of changes into production, more robust releases and the resulting increased customer satisfaction are just a few of the many opportunities CI/CD offers. In order to take advantage of these opportunities, the implementation as well as the operation of CI/CD must be successfully managed. Challenges such as a high level of standardization in the IT landscape, possible adaptation to an agile organizational form, and the procurement of expertise must be mastered.
We are happy to support you in analyzing where you stand in terms of requirements for the use of CI/CD (degree of standardization, form of organization, etc.) and in evaluating and designing possible CI/CD solutions in the cloud.
Our years of experience in the essential disciplines for your individual Cloud Journey. With our support, you can master the hot topics such as network, organization, availability, automation, CI/CD, IaC, governance/compliance, security & cost management.