DevOps, Agile and Microservices
DevOps provides the principles, practices, and tools, coupled with organizational and cultural changes, to allow application-focused teams to deliver functionality to the end user via repeatable, predictable processes.
Agile methodologies such as Scrum and SAFe break down this DevOps pipeline into smaller, manageable chunks allowing teams to work with their end users to collaboratively iterate towards the right solution.
When organizations incorporate a Microservices architecture as the target platform with DevOps and Agile methods, they are able to further break down the application along services and align the Agile process and DevOps pipelines with these individual services. Ideally, each service on the Microservices platform has a dedicated Agile team and DevOps pipeline.
The expected benefits are an acceleration of delivery closely aligned to the needs of the service and end user, decoupling delivery of individual services and increased agility in responding to changing customer needs and priorities.
First, the Culture Shift
More companies today, large and small, are turning to DevOps as indicated by a recent DevOps adoption survey by Puppet Labs of over 4,000 IT professionals across a wide variety of companies in over 90 countries. Over 63% of the respondents have implemented DevOps since the 2011, representing a 26 percent increase in adoption.
These organizations are starting to see real value from DevOps in the form of accelerated delivery, increased stability, faster issue resolution, and increased quality. Organizations are able to deliver more functionality of greater value to the user in a shorter period of time than with previous methodologies and processes. But before they could realize the full benefits of DevOps, these organizations came to understand that organizational and cultural changes were required.
Several years ago, John Willis, currently an Evangelist at Docker Inc., coined the acronym CAMS, which stands for Culture, Automation, Measurement and Sharing, as key principles and components of DevOps. While each of these is essential and important, it is the required changes in cultural and organizational changes that appear to be undervalued and overlooked when organizations embark on the DevOps journey.
Conway’s Law states, “Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” This well-known and time proven theorem states that systems are designed and delivered in ways that echo their organizational structure. Given this, it is imperative that organizations are structured accordingly to facilitate the DevOps process.
Traditionally, organizations have been formed and aligned their resources by technical domains and expertise such as development, testing, and operations. In order for DevOps to be successful, organizations must be realigned to optimize for application delivery. The organizational barriers that map to the Waterfall approach need to be removed and replaced with an application-centric delivery model - a delivery model where teams are responsible for and able to take an application from inception to production.
Building the right team to take advantage of this new organization and culture is critical to the success of DevOps. It is imperative that you construct your DevOps teams with the right mixture of skills from across the various disciplines and domains that your application will touch. The team should truly be cross-functional, drawing from organizations and teams that in the past have been separated only because of their underlying organizational structure. It is this team that will be responsible for taking the application and functionality from inception to production. This team will own and be responsible for this application and ensure that it is delivering real value to the end user. Additionally, it is this team that will form the relationship with the user community, build trust, and gain a true understanding and appreciation of the user’s domain. It is this type of team and organizational structure that can support the DevOps mantra - “you build it, you run it”.
While organizations can expect to see significant improvement in the process of delivering value to the end user with DevOps, DevOps itself is lacking in terms of the level of granularity in the delivery process. While DevOps offers the potential for greater stability, higher quality, increased responsiveness on product issues, and closer customer relationships, it still suffers from pushing the same large application out to the user. What it needs is a way to divide up that larger application into smaller, manageable chunks that can be more easily delivered to the end user in shorter intervals. It needs a way to respond with more agility to unforeseen changes in the needs of the user.
Enter Agile methodologies. Agile methodologies such as Scrum, SAFe, and KanBan, offer the ability to break down the delivery cycle into smaller, more manageable chunks of work and incorporate user feedback early and often.
The fundamental advantage is that development can proceed forward without having to deliver all the requirements up front for the application or for a specific function. This allows useful functionality to be delivered to the end user faster, and while incomplete, allows the user to provide key feedback to the development process.
This feedback is vital in that it allows the development team to make small, incremental adjustments in the direction of the product allowing the team to vector to the best solution. This situation is similar to driving a car. If you were taking a trip across the country, while you certainly would have the general direction and route mapped out, there are thousands of small adjustments you make along the way. Can you imagine turning the wheel every 30 miles, or going through city traffic ignoring rush hour traffic simply because that is what was decided hundreds of miles ago? In fact, we make thousands of adjustments along the way to navigate the winding road and ensure we are on the best route considering the conditions.
Just as with steering a car to its final destination, Agile allows the software development process to deliver functionality early and often, continually incorporating feedback from the user in order to make the necessary adjustments to ensure that we deliver real value to the end user. As changes happen in the user domain, as they always do, an Agile approach allows us to react accordingly.
Focusing with Microservices
DevOps provides us with an application delivery focused team and an organizational structure to support it. Agile methodologies bring an incremental, functionality first approach to break that delivery process into manageable pieces, enabling us to vector on a much finer granularity towards delivering real value to our end user.
But while Agile and DevOps combined allow a feature focused delivery team the ability to divide the application delivery into more manageable pieces, the delivery of the application must still deal with the dependencies and coupling between the different functions across the entire application. If I want to deliver a piece of functionality to a user for review and feedback, it still requires delivering the entire application and all its various components that have been delivered to date. What if I could take what Agile did for DevOps - breaking the process up one step further?
Enter Microservices. Microservices is an architectural style that composes functionality in small, self-contained components. Each component can stand on its own and, more importantly, it can be deployed independently. So what does that do for the team using a DevOps process and Agile methodologies? What it does is provide an even finer grained approach to the delivery model. With Microservices, functionality can be delivered without the inherent coupling with other functions and features. Each function can evolve at its own pace without the artificial dependency on the progression of other functions. This flexibility is not only valuable during the initial development process but also as the application evolves over time. Features can be added and/ or modified, and independently deployed into production providing even more flexibility and agility in response to changing user requirements.
The initial DevOps and Agile process can then be split up further along service lines with independent teams and supporting DevOps pipelines. Feedback from the user is incorporated as before and is focused on the service. Services can be independently deployed and migrated along the path to production. They can stand on their own (or with a small number of dependencies) and can serve in a sense as micro-releases within the context of the overall application. Users, product owners, and architects have much more flexibility of what and when functionality is delivered to the end user.
DevOps, Agile methodologies, and Microservices are each by themselves powerful technologies and techniques for improving the quality of software and bringing greater value to the end user. However, bringing these technologies together in the software development effort allows for leverage of each to bring even greater overall benefit to the end product. DevOps breaks down the barriers between teams, aligning the organization and delivery model and streamlining the path to operations. Agile methodologies can break down the delivery of the application into manageable pieces. Microservices further breaks the overall application into service channels providing even more flexibility and agility to the development and deployment process. In the end, this powerful combination provides organizations greater agility to deliver more functionality of greater value to the end user.