Support management teams to make decisions to create DevOps teams
ABN AMRO
Financial services
19000 employees worldwide
from mid-2019 to the end of 2020
Learn how ABN AMRO set up their DevOps transformation program, enabling departments to adopt DevOps principles and operating model while having processes that fit their contexts. There was a blend of expert guidance from inside and outside the organization. ABN AMRO invested in several pillars supporting the development and management teams, such as learning and development, platformization, and team autonomy. You can read more about it below and use the key takeaways and insights for your journey.
ABN AMRO is a Dutch bank offering a wide range of financial services to retail, small, medium, and corporate clients. The bulk of their business is in the Netherlands, and they also offer financial services in other countries worldwide. At the time of their DevOps transformation, they had 19000 employees worldwide and 6500 in the IT department. The bank has the ambition to be one of the top banks with its product portfolio and also with its technical landscape. As such, they created a DevOps program to create autonomous DevOps teams, minimize the handovers in a value stream, and enable the organization to realize its ambitions. As far as they knew, it was one of Europe’s most significant organizational transformations at the time.
The DevOps transformation program followed the Agile transformation (2014-2017). The Agile transformation created 550 teams (called blocks) within 45 departments (called grids) in the IT space. A department is a matrix organization structure with a mix of business and IT at management and operational levels. It allowed them to have defined boundaries for their responsibilities. At the beginning of 2018, a post-Agile movement emerged, with around 100 teams scattered across different departments, experimenting with their workloads on the public cloud. With those learnings, the IT senior management team couldn’t ignore the benefits and started to question what needed to happen in the organization to scale the approach, and what were the consequences of evolving the Agile model?
With those questions in mind (organization, enablement, financial implications, processes & controls, approach), ABN AMRO took the first steps and built a business case, not only for the productivity and financial benefits of
adopting DevOps but also incorporate other initiatives such as the broader adoption of public cloud. The Executive Board and Executive Committee of ABN AMRO were convinced, and the DevOps program became part of the bank-wide strategy. The DevOps transformation program had four main streams:
- Organizational design – redesign processes with the change to a DevOps way of working, taking into account the accountability and autonomy of the teams while balancing the regulations that a bank is subject to. It strongly emphasized the continuous learning and development of the staff, with a significant investment in training and career progression from an HR perspective.
- DevOps enablement – ensure that tools and platforms can offer the different capabilities a DevOps team requires, spanning continuous integration and delivery, observability, infrastructure consumptions, and cost to security and compliance.
- Cloud migration – offering a fast-track approach for teams to move their workloads to the cloud, with gold paths for typical workloads. To be successful, it builds on the previous two points.
- DevOps execution – create an approach to support the departments and their teams on the DevOps transformation since the change (development) and run (operations) silos responsibilities will be merged into DevOps teams.
Execution by learning
I have been involved in the DevOps execution stream almost since its inception. The program set up a DevOps enabling team, with a transformation coach and a person from the different capabilities a DevOps team needs to have autonomy (think about the capabilities as architecture and design, development, testing, security, and operations, among others). The DevOps enabling team had around nine people, and as a transformation coach, I led the team.
There was the acknowledgment that each department was unique. Although ABN AMRO wanted a DevOps way of working, they knew that each department’s different business and technical responsibilities led to different solutions. The focus was on DevOps principles and the enablement for their adoption. With this in mind, my involvement as a transformation coach was twofold: (1) support one department with the adoption of DevOps and (2) bring the learnings to the program level, allowing it to adapt and scale at a later stage.
The pioneering department was responsible for the mobile apps for retail banking. At the time, they had four different mobile apps, with around 17 teams owning different parts of those apps. Initially, they had a mix of native mobile app developers, user experience designers, backend developers, scrum masters, and product owners. On top of that, they would have new colleagues from operations, forming the new DevOps teams based on their needs. Upon my arrival, I discovered a mismatch with the expectations of the department management team. Where they expected a blueprint with fixed steps, I expected to support them in a journey to adopt the DevOps principles with their context in mind. Managing the expectation mismatch was my first learning, starting by defining the common ground and goals. It also signaled to the program management team that improving the initial communication with the departments about the expectations was crucial.
After the initial expectations mismatch, we (the department management team and I) quickly set up a new action plan: based on our shared belief that decisions should be made where information is, we set a mission command structure, where goals are set, and teams define how they want to achieve those. Bare mind that teams are also involved in the goal-setting process. Therefore, the management team has feedback about the gaps that need to be filled.
From my work with this department, two examples stood out: how to integrate the operations processes into their reality and how a team used their new scope to achieve cost savings for the services that they own.
Integrating operations processes
As part of my responsibilities as a transformation coach, I need to support the department management team in incorporating operations into their scope. This means that processes such as on-call, compliance, or vulnerability management, among others, will need to be integrated. Since there weren’t steps to incorporate those processes, I developed a workshop to explore what on-call means to them. Traditionally, the run department offered two models of application support: (1) 24/7 or (2) during regular business hours. It allowed them to explore a different model that could fit their responsibilities and keep costs under control. In their case, they end up with parts of the banking mobile app being 24/7, since those features are crucial for the business operations; other features have extended business hours (08:00 to 20:00 during weekdays and Saturday mornings) since those features connected customers to customer service agents; and lastly, different features were deemed to have support during regular business hours.
It is one example of a management team adapting new responsibilities to their context, always with business outcomes in mind. Plus, from the technical side, they manage mobile apps, which depend on the backend services to operate, although they don’t own those services. It strengthens the communication between all of those teams across the departments.
Service efficiency and cloud cost savings
The new DevOps teams embodied the principles in no time. One of the teams responsible for the backend-for-frontend services for the retail banking mobile app used their new responsibilities to improve the efficiency of their application and, at the same time, save on their cloud costs. As part of their migration to the cloud, the team moved their data store from the data center to the cloud-managed service for relational databases. They used the observability tools to understand the usage patterns and moved their database from the managed service available 24/7 to a cloud-native serverless database service. Given the predictable and discrete nature of the workload, they achieved savings of around 90% on their cloud costs and performance improvements.
These team behaviors are possible since the environment around the teams enables them. In this case, the processes with a DevOps orientation and safety nets provided by the different platforms allow teams to choose the technology that fits their context, fits the budget, and offers good business outcomes.
Scale and innovate
ABN AMRO’s strategic ambitions include the DevOps transformation program, which must support all IT departments. After a couple of months with the first department, I started to bring what I had learned to the program, allowing the program managers to adapt and prepare to scale the support to a bigger group of departments. The strategy was to balance the business needs with the technical real estate. Therefore, the program created different waves, allowing cohorts of departments to start simultaneously. For that, it was essential to diffuse the learning at different levels and across different boundaries.
Diffusion of learnings enables the adoption of proven practices
And those proven practices allow the DevOps program to scale. There were several examples of diffusion of learning, some of which I did not influence and others in which I was an active participant.
Gamification as a vehicle for teams to share
One of the early examples of diffusion of knowledge was the leverage of gamification within the retail banking mobile app teams. At the time, seven teams were responsible for the different services and parts of the mobile app and were keen to improve their practices and ways of working. However, rather than having a significant planning and coordination exercise, the scrum masters organized a game where the different teams could achieve different milestones, and they were awarded a Lego brick. Those Lego bricks made a small astronaut figure, but what was more important was that it promoted communication between the teams, sharing what worked and, most importantly, what didn’t work.
Later, they expanded it to other mobile apps voluntarily. It was a great way to create a feedback loop inside the department, sharing the achievements and the mechanics of the improvements made. It also served to inspire other teams outside of the department.
Internal conferences to spark curiosity
As the DevOps program gained traction, other teams in different departments started to be curious about what was happening. Some were afraid that it was just more work, others were seeking an opportunity to own the full software development value stream for their services and applications.
The bank had several internal conferences and lunch talks organized by the different capabilities and business units. One of the internal conferences focused on the developer’s community and invited one of the mobile app teams to showcase how they leverage the observability platform to expose how payment transactions transvest the different services from the customer’s smartphone to the mainframe, crossing the cloud. They aimed to show how a DevOps team could leverage an internal platform, in this case, observability, and use the available tools to increase the transparency for anyone in IT about the performance of a critical business function.
Another instance is when I was invited to speak at the Retail Banking Day, where all retail banking staff (business, business operations, product, and IT) show and tell what is happening. The managers and senior managers were interested in the shape of the program and how it could support their goals. It gave insights into the questions and challenges of other parts of the retail bank, allowing the program to support the following journeys better.
Participatory workshops between management teams
One critical feedback loop was between the first department under the DevOps transformation journey and the DevOps program management team. In the initial shape of the program, it didn’t envision this feedback loop; however, as I enabled the department management team to apply the DevOps principles, I noticed that some overarching concerns needed to be addressed. To address this gap, I created a recurring workshop between the stakeholders and designed different workshop experiences that could address the burning topics. These participatory workshops allowed them to align on the issues that needed attention and for which the department management team didn’t have the authority to decide. I designed and facilitated those moments and created the space for informed decision-making.
Use of field stories to persist the knowledge
As more teams were on their DevOps journey, they generated much new knowledge. However, given the organization’s size, having only an oral culture doesn’t allow the knowledge to be easily transferred across different people. Also, remember that it was before the COVID-19 pandemic with an in-office culture.
I like to read and write (albeit not the best writer), and I started to collect small field stories that could showcase how people solve challenges. In the beginning, I began to write those myself, from my day-to-day experiences within the DevOps transformation, and share them with people. Some of those folks started to contribute, and at the program level, we created a library where people could read and get inspiration. Those field stories were not recipes; we don’t tell people to add A to B to get C but rather describe the challenge, the different options at the time, and the heuristics behind the decision. Hopefully, it inspired people along the way.
Shadowing to onboard new transformation coaches
As the program started to scale, they began onboarding more transformation coaches to perform the same role as I did. At the time, the program management team challenged me to diffuse my knowledge and experience with my new colleagues.
I believe that we learn best at the job, and rather than thinking about an extensive onboard program and trying to lecture people, I adopted a shadowing approach to onboard the new colleagues. The program had enough documentation to guide the departments and could be used as a foundation for the new joiners. What was missing was the feeling of being with the departments, the questions asked, and the challenges that they faced. During some weeks, a new colleague joined me in the workshops and meetings with different stakeholders. It allowed them to see the typical day-to-day of a transformation coach. After those workshops and meetings, we had a debrief session where the new colleague asked their questions and provided feedback. It proved crucial to hold these debriefs since I could understand where I needed to invest the time to onboard the new colleague and, at the same time, get insights that improved my approach. It was a win-win situation for everyone involved.
Innovation and the COVID-19 pandemic
A great aspect of the program was the capacity to innovate. The DevOps transformation program had clear goals and ambitions and, simultaneously, room to experiment with mobilizing and supporting change at this scale. One of the innovative aspects was the use of the Team Topologies patterns. The Team Topologies book was published in September 2019, and their ideas and clear language for team types and team interactions were what I was missing to increase my effectiveness.
With the new knowledge, I could prototype new workshops that blended Team Topologies with the program topics and helped management and their teams with their decisions. The techniques’ visual nature helps people focus on the challenges to be solved instead of a typical roundtable discussion. Some departments were able to redesign their boundaries with the emergent workshop insights.
Those different workshops were adopted at a program level. As we scale for other parts of ABN AMRO, we iterate and improve the materials, making them available in the library of tools that transformation coaches could use at any point. Reflecting back, this approach was a great asset for the program; rather than having a fixed set of steps, there was a vision, goals to be achieved, and a path that needed to be discovered between all the stakeholders. There was a fantastic toolbox to be used, and the number of tools grew as the program moved forward.
With the program’s increasing reach, I was challenged by the DevOps program management team to support them in creating cohorts of departments to be part of the different waves. There were staff and resource limitations, and even if they were not in place, it is not a good idea to change an IT department of this size all at once. As such, I designed a workshop with the goal in mind, and it could also take into consideration several dimensions, such as the dependencies between departments or even their strategic options for the foreseeable future. Taking the management teams of all retail banking departments (circa 11 departments), they could leverage cross-department initiatives that were not obvious in the first place but also discover hidden dependencies that usually are only discovered later in time. Considering the various dimensions, the departments self-selected for the different cohorts. It was crucial to create the necessary ownership for the DevOps change that each department needed to perform, but at the same time, it also protected their own context and business initiatives that they needed to realize. The quality of the decisions was far superior compared to a typical top-down decision from a program committee. And it proved to be the right one, considering what happened afterward.
The COVID-19 pandemic was here. Luckily, the self-selection process for the cohorts was done, which allowed ABN AMRO to move from an office reality to a remote reality. After the first weeks that everyone needed to adapt to a new reality, the new cohorts went smoothly. All the effort into explaining the why, onboarding new transformation coaches, and having self-selection processes provided the right environment for the departments to take ownership of the transformation.
João did an exceptional job in this guiding help, shaping the approach not just for his direct scope, but also lifting it up for all remaining Tribes and DevOps teams (550+) at ABN AMRO. From operational effectiveness of the teams, on CI/CD, security, ITIL, to the strategic approach we are currently executing. A great teacher and guide, that pushes you and the rest of the organization to do better and venture beyond your own boundaries.
I hope to work with him again in the foreseeable future!