Shifting Change Control Left: How We Turned Six-Week Release Cycles into Daily Deployments

When I joined Spectrum Health (now Corewell Health)—one of the largest health systems in the world—I was tasked with leading a new product team that inherited a legacy application. This legacy system, while functional, was plagued by a dysfunctional release cycle. On a good day, it took six weeks to release updates, and that wasn’t cutting it for anyone:

Executives were frustrated by the inability to quickly adapt.
Marketing teams were stuck in limbo, waiting on development.
Product teams felt weighed down by unnecessary processes and roadblocks.

The challenge was clear: we needed to shift from sporadic, cumbersome releases to a system capable of delivering continuous updates to production multiple times daily. And we needed to do it while maintaining 99% uptime and ensuring the highest standards of quality and compliance.

This article outlines how we achieved that transformation, focusing on automating and modernizing our change control process. This journey was a testament to the power of collaboration, automation, and a shared commitment to continuous improvement. It also took time, and there was a lot more to it than I can cover here today. I may revisit this topic in future posts, but for now, let’s dive into the heart of the matter.

The Starting Point

We were working with an antiquated GitOps delivery system, implemented and guarded by a support team disconnected from product teams. Their lack of collaboration limited our ability to evolve the system.

To make matters more complex, we had to navigate a strict and highly regulated change control process. As a healthcare provider, these risk mitigations were critical—our systems directly impacted patient care. But while the process made sense for life-critical systems, it was clear that not all products carried the same level of risk.

Despite the technological debt, it was the inefficiencies in the change control process that posed the greatest threat to our goals. I realized we couldn’t simply overhaul the system without buy-in. Instead, we would embrace the objectives of the Change Advisory Board (CAB)—risk mitigation and compliance—and integrate them into our development workflows frictionessly.

The Strategy: Shifting Change Control Left

Shifting change control left means embedding release-readiness tasks into the development process, automating as much as possible, and ensuring compliance is baked into every step. Here’s how we approached it:

Adopt a Cloud-Native Strategy
Cloud-native technologies provided the flexibility and scalability we needed to modernize both our application and our delivery processes.
Build with Infrastructure as Code (IaC)
By leveraging IaC, we could create fully reproducible environments. Every branch had its own sandbox for testing, ensuring consistency and isolation.
Implement a Continuous Delivery Pipeline
Our new pipeline automated testing, deployment, and quality checks, allowing us to deliver changes quickly and reliably. The pipelines also self-mutated our stack as we made changes, ensuring even greater consistency across environments. Self-mutation w/ IAC was the secret sauce to our success, which I'll dive into in a future post.
Automate Change Control
This was the game-changer. By embedding change control tasks directly into the development process, we reduced lead times and eliminated process times and manual bottlenecks.
Rewrite the DNA of the Team
We fostered a culture of collaboration, curiosity, and ownership; empowering every team member to contribute to the change control process. Everyone was going to be a part of this, and we were going to do it the best; it was a decided and concious and unanimous effort, one that was very special to be a part of. This shift in mindset was crucial to our success.

Automating Change Control

To automate change control, we started by conducting a collaborative value stream mapping session. We documented our entire process, measured process and lead times, and identified pain points. The results were staggering:

The existing process was bloated and riddled with inefficiencies.
Many steps were redundant or could be automated.

From there, we designed an ideal value stream and developed an iterative roadmap to reach it. Here's how we tackled the key challenges:

Risk Mitigation Through Automation

Many of our release tasks were risk-related, particularly around documentation and testing. We automated these tasks wherever possible:

Testing and Quality Checks:
- Automated pipelines tested every change against defined thresholds.
- CAB approved that passing these checks satisfied compliance requirements.
Environment Validation:
- Lower environment testing was integrated into our workflow.
- Sign-offs were automated through our task management board.
ADA Compliance and Security Scans:
- Lighthouse checks for accessibility and vulnerability scans were added to our pipelines.
- Builds failed if compliance thresholds weren’t met.

Rearchitecting Development Practices

To support automation and ensure quality, we made several changes to our development process:

Trunk-Based Development:
Rebasing and working directly on trunk ensured every change was tested with the latest code.
Full-Stack Deployment Previews:
Every branch had its own environment, enabling isolated, end-to-end testing.
Disaster Recovery Protocols:
We rehearsed and refined protocols, ensuring downtime was never caused by our releases.
Release on Demand: We shifted from scheduled releases to releasing on demand, and given our size, we chose to do it straight from trunk. This allowed us to deliver value as soon as it was ready, rather than waiting for a predefined release window. This also meant we had to build in extra protections, like feature flags; another topic too large for today that I might cover more in depth in the future.

Results

The transformation was nothing short of remarkable:

We shifted from 6-week release cycles to delivering multiple updates daily.
We maintained 99% uptime, with no downtime attributable to our changes.
Our team became the go-to example within the organization, mentoring others to adopt similar practices.

Key Takeaways

Automating change control isn’t just about tools; it’s about mindset. By embedding compliance and risk mitigation into our workflows, we demonstrated that rigorous change control processes could coexist with continuous delivery.

If your organization struggles with inefficient release cycles, consider these steps:

Map your current process and identify pain points.
Collaboratively design an ideal value stream.
Automate tasks wherever possible, ensuring compliance is built into development.
Adopt modern development practices like trunk-based development and IaC.

By embracing change control as a development enabler, rather than a blocker, you can achieve faster delivery, higher quality, and greater organizational agility.

Looking Ahead: Scaling Success Across the Enterprise with TestifySec's Platform

Achieving continuous delivery and automating change control were transformative steps for our team, but scaling that success across an enterprise presented a whole new challenge. Teaching other teams to adopt our practices wasn’t as simple as sharing results or playbooks. Every team and product brought unique considerations, and recreating the same magic proved difficult.

What if we’d had the tools to make this process easier?

In an upcoming blog post, I’ll share how the journey of scaling continuous compliance inspired me and my cohorts at TestifySec to build a platform designed to solve this exact problem. By leveraging tools like in-toto witness and archivista, this platform empowers organizations to automate compliance, monitor adoption, and scale high-performing practices across teams.

Stay tuned to learn how these tools could transform enterprise-wide adoption and make scaling success not just achievable, but seamless.

This was just one facet of the modernization journey I led at Corewell Health. Future posts may dive into how we scaled our CI/CD pipelines, redefined team culture, and implemented disaster recovery protocols. Stay tuned!

12-03-2024