Background image

Release on Demand: A Key to Risk Reduction in Continuous Delivery

Achieving continuous delivery is a game changer for any organization, but doing so with low risk requires more than just deploying frequently—it demands a systematic approach to release on demand.

Years ago, my health product team at Corewell Health embarked on a journey to adopt these practices. At the time, these concepts were new to us, and while we aspired to achieve them, our legacy product presented a significant roadblock: we had never decoupled releases from deployments.

The solution? Feature toggles—a deceptively simple yet transformative pattern that enabled us to decouple deployments from releases, reduce risk, and empower our teams to innovate without hesitation.

If you’re familiar with A/B testing or experimentation, the concept of feature toggles might not be foreign to you. Likewise, if you’ve worked with canary releases, you’ve likely already seen how release on demand, powered by feature toggles, can transform delivery pipelines.

In this post, I’ll share our journey to release on demand, the role of feature toggles in achieving it, and the transformative impact it had on our team and product.

Separating Deployments from Releases

The idea of separating deployments from releases initially baffled us. How could deployment—putting code into production—be disconnected from releasing features to users? It sounded counterintuitive.

But we had ambitious goals: deliver changes to our product continuously, several times a day, with minimal friction and no team intervention. To achieve 99% uptime, remediation within an hour, and low risk, we needed a new way forward.

Feature Toggles: The Game Changer

Enter feature toggles, a pattern that allows teams to enable or disable software features independently of deployments. You’ve likely experienced this as a user: remember when Facebook rolled out a new design to select users while others waited their turn? That’s feature toggling in action.

At its core, feature toggles are implemented as a dictionary of key-value pairs, where each key represents a feature and its value is a boolean: true (on) or false (off). Here’s a simple example in JSON:

{
  "new_dashboard": false,
  "dark_mode": true,
  "global_search": true
}

While conceptually simple, feature toggles unlock significant capabilities. They allow you to:

  • Manage features at runtime, build time, or both.
  • Toggle features on or off remotely, even for specific users or groups.
  • Perform canary releases, where features are gradually rolled out to minimize risk.

Feature toggles, coupled with monitoring, provide a powerful mechanism to control your application’s state for your entire user base—without requiring code changes.

Risk Mitigation for Continuous Delivery

Feature toggles are instrumental in achieving release on demand within a continuous delivery framework. Here’s how:

  1. Test Toggles in All States: Ensure code changes are 100% production-ready before merging into your product.

    • Test features in the "off" state to verify they don’t impact the application at all.
    • Test features in the "on" state to confirm the application remains stable—even if the feature itself is a work-in-progress.

  2. Enable Immediate Reversibility: If an issue arises, simply toggle the feature off, eliminating the need for emergency rollbacks.

This approach drastically reduces the risk of continuous delivery, shortens remediation times, and enables instant releases.

Outcomes: A Transformative Pattern

When we adopted release on demand, our entire world transformed:

  • No more release meetings.
  • No release schedules.
  • Developers could merge and deploy changes multiple times a day without hesitation—even work-in-progress features.
  • Ownership shifted: Product and marketing teams took control of releasing features, turning them on when ready.

This process empowered everyone. Product teams loved the autonomy. Content teams appreciated the safety net to disable problematic features instantly. And for developers, on-call incidents became virtually nonexistent.

Fix-Forward: Planning for the Unexpected

Even with the best systems, things can go wrong. That’s why we embraced fix-forward—a rehearsed strategy for addressing production issues in a continuous delivery environment.

Key elements of a fix-forward strategy include:

  • Expecting failure: Plan for what could go wrong and document clear steps for resolution.
  • Rehearsing failure: Regularly test your recovery plans to ensure they work when needed.

For example, we once faced an AWS Kendra outage that disrupted our global search feature. Instead of scrambling to deploy a patch, we simply toggled off search until AWS resolved the issue. Our uptime goal of 99% was maintained, and stress levels remained low.

In an upcoming post, I’ll share our fix-forward playbook to help you create or refine your own.

What Feature Toggle Platforms Are Available?

If you are looking for some open source solutlions, you can check out:

  • OpenFeature: An open-source feature flagging platform aiming to standardize feature toggling.
  • Unleash: An open-source feature toggle platform that provides a simple, scalable way to manage feature toggles.

Several paid feature toggle platforms are available to help you manage toggles at scale. Here are a few popular options:

  • LaunchDarkly: A feature management platform that enables teams to control feature lifecycles, from concept to launch.
  • Split.io: A feature flagging and experimentation platform that allows teams to manage feature releases and monitor performance.
  • Optimizely: An experimentation platform that enables teams to test and optimize features for better user experiences.

Bringing Release on Demand to TestifySec

The transformative power of Release on Demand and Continuous Delivery isn't just a lesson from my past—it's a principle I'm actively applying today at TestifySec.

At TestifySec, we’re building a platform that enables continuous compliance and automation for secure software delivery. Our goal is to empower teams to release frequently and confidently while maintaining high standards of quality and security.

Why This Matters

When software delivery is fast, reliable, and low-risk, teams can focus on innovation instead of firefighting. At TestifySec, we’re not just creating a tool—we’re building a foundation for organizations to scale their practices and achieve the kind of transformative outcomes that Release on Demand made possible for my team in the past.

If you’re curious about how TestifySec can help your organization achieve this level of agility and confidence, check us out or feel free to reach out.

Addressing the Elephant in the Room: Release on Demand and SAFe

Sometimes, Release on Demand gets associated with a four-letter word: SAFe.

Yo Dawg, I heard you like Planning, so we put planning inside your planning!

an over-simplification of SAFe

The Scaled Agile Framework (SAFe) has faced its fair share of criticism in recent years. I’ll be the first to admit—it doesn’t work for every organization or product. In many cases, teams misapply its principles, confuse its concepts, or worse, turn it into a rigid, bureaucratic nightmare. This has led to SAFe becoming a taboo topic in some circles.

But let’s clear something up: Release on Demand is not exclusive to SAFe.

While SAFe teaches and emphasizes this practice, Release on Demand has been a cornerstone of successful software delivery for decades—long before SAFe entered the scene. It’s been used by some of the biggest names in tech, like Facebook, Twitter, Microsoft, and Google.

Release on Demand is a universal practice, independent of frameworks or methodologies. It’s a tool in your toolbox, one that empowers teams to separate releases from deployments, enabling flexibility, risk mitigation, and faster iteration cycles. The key is understanding how to adapt it to your organization’s unique context, rather than being bound to any one framework’s interpretation of it.

More to Come

Release on demand opens the door to a wealth of advanced practices, from canary releases to runtime feature toggles with defaults, and automated disaster recovery. These topics deserve deep dives of their own, so stay tuned for future posts!

In the meantime, if you haven’t yet explored feature toggles and release on demand, I encourage you to start. They’re more than just technical practices—they’re enablers of innovation, agility, and peace of mind.

12-10-2024