@reiver

Dark Launching Software Features

Having used dark launching for a little while now, I've found that it helps improve, not only the efficiency of a software development team, but also helps increases the rate of flow which new features are "put in hands" of users.

(This post serves to document my current impression & experience with it.)

This technique that I'm calling "dark launching" is also known as "branching in code", "feature flags", "feature bits", "feature toggles", "flag flippers", and "latent code". I like "dark launching", so I'll mostly use that in this post :-)

Dark launching help make it so there is no need to let releases of completed features be held up until one or more incomplete features are done. Dark launch each incomplete feature (so that it is inert and disabled on production), and launch whenever you want so that complete features "get into the hands" of your users ASAP. (Note, typically software engineers and other developers will add special code to enable dark launching as part of their engineering or development process.)

(There are more use cases for dark launching later in this post.)

Also, this means that there's no need to resort to the pain of branching strategies (in code revision systems); or at least, you won't have to resort to it as often. You can (tend to) avoid the problems with merging. Avoid the problems with critical fixes on release branches, and merging those back to the main development branch. And avoid the difficulties with maintaining multiple divergent versions of your code (in the various branches).

And it can put you on the path towards and help enable you in doing releases multiple times per day. It can also put you on the path towards and help enable you to always ship from trunk (in the code revision system). (Yes, there's reasons you would want to release from trunk, but the reasons are beyond the scope of this post.)

(Note that to make this work well, things such as unit tests, software testing frameworks (such as selenium), smoke tests, and continuous integration should be in place first.)

What is Dark Launching

Dark launching is a technique of "wrapping" the code of new software features in a way that let you turn them on or off. Perhaps turning them on and off for all your users, or perhaps turning them on and off for only a subset of your users (that meet some criteria).

(There are explicit examples of this in PHP, JavaScript, CSS, etc code later in this post.)

The dark launching subsystem also tends to expose some kind of administrative interface so that features can be managed.

For example, so that someone on the marketing team can turn on a feature just before they send out a press release. Or, for example, so that ops can turn on a feature for 10% of the users, and monitor the servers to make sure no problems arise, before ramping it up to 20%, 40%,... and finally 100%. Or even, for example, so that QA can turn on a feature whose code is already in production but hadn't a chance to test yet.

It's also a good idea for this dark launching subsystem to log when by whom the status of a feature is changed, such as turning them on or off, changing percentages, etc.

Why is Dark Launching Useful

Obviously, dark launching is useful in cases where you want to turn a software feature on or off. But what are the details of this want?

One example would be if you may want to do a private beta release, where only a select few users or (users in) a select few organizations can see, try out, and (hopefully) report problems with a new feature. (A good way of doing testing on production.)

Another example is if you are developing a new feature, and want to commit your code in logical chunks, but the feature as a whole is not ready and should not show up on production yet. (This is a very important use case for feature dark launching.)

Yet another example would be if you want to release a new feature to all your users at a specific date; perhaps to time it with a press release from the marketing department.

Another example would be if you are not yet confident in how stable or how scalable a new feature is, and you want to test it on only a small percentage of your user base (perhaps 5% or whatever you want to set it to) and not all of them. That way if there are problems with it, you don't piss off your entire user base, and problems can be worked out before the feature is released to all your users.

Feature Dark Launching Types

As you can probably infer from my examples, there's a number of ways you may want to dark launch a feature -- a number of dark launching types.

The types I am familiar with are: boolean, percentage, list, identity, and nil.

(There may be other dark launching types you come up with. But these are the types I find useful.)

(Note that percentage and list are each not really a single type, but a type of types. It will make more sense later on in this post.)

Boolean

Sometimes you just want a feature to be on or off.

For example, this is useful if you want to release the code to all your users at a specific date; perhaps to time it with a press release from the marketing department.

As another example, this is also useful for being able to commit code to the production (i.e., live!) code base of the software system while the feature is incomplete and still in development. (There's reasons why you may want to be doing this, but they won't be discussed here.)

Or, for another example, the code for the feature is already on production but made inert because it is wrapped in dark launching flags. QA finally has time to test the feature, gives the "thumbs up". And then turns on the feature on in production.

The boolean type will let you do these things.

Percentage

Sometimes you may want to release a new feature to just a certain percentage of your users, and not all of them. Thus, for some percentage of your user base, the feature will be on and for the rest of your user base, the feature will be off.

For example, perhaps you are not yet confident in how stable or how able to scale a new feature is, and you want to test it on only a small percentage of your user base (perhaps 11%) and not all of them, so that you don't piss off your entire user base if there is a problem. (In this example, the users in the percentage of your user base, should stay the same. Meaning, if Tom, Dick and Harry are in that 11%, then they should stay in that 11%, and not be switched out with other people. You'd probably implement this based on user ID or a tracking code or something like that.)

Or, for example, perhaps the ops team wants to stress test a new service, under production load patterns. So, they decide they'll start off such that 12% of the users will, from the users point-of-view, secretly make a API call to the service, when users visit a certain page. So, this will happen every time a user, who is randomly chosen in the 12%, loads a certain page. If the first stress test passes, the ops team can increase that load test to 25%. Then 50%. Etc. (For this kind of percentage type, choosing users with a random number generator makes the most sense.)

The percentage type will let you do these things.

As you can probably infer from the examples, the percentage type can come in different "flavors". And really, there's more than one percentage type. (Ex: the "percentage user", "percentage cookie", "percentage random", "percentage group", etc.)

(Note, although you could use a percentage dark launching feature flag for things such as a/b testing, it's better not to. If you are using dark launching properly, you'll be removing feature flags within a couple weeks to a month of creating them (if not sooner). For a/b testing, you should create/use something specially created for that purpose, and not overload your dark launching subsystem.)

List

Sometimes you have a list of something that tells you who may have access to the feature.

For example, perhaps you want to list user IDs of users who may access the feature. This list could be your private beta testers.

Or for example, perhaps you want to list some kind of group ID, and all the users in those groups may access the feature. Perhaps the group ID is an organization ID, and you are only letting users in a handful of organizations see the feature.

The list type will let you do this.

As you can probably infer from the examples, the list type can come in different "flavors". (The type of group IDs that you have will depend on what type of groups are modelled in your software system. For example, organization, sports teams, families, or whatever groups you system has modelled.)

(Note, although you could use a list dark launching feature flag for things such as entitlements, it's better not to. If you are using dark launching properly, you'll be removing feature flags within a couple weeks to a month of creating them. For entitlements, you should create/use something specially created for that purpose, and not overload your dark launching subsystem.)

Identity

The identity type is very simple. It is always on! and cannot be turned off.

(For the math-inclined, think of it like multiplying the number 1 to anything. It doesn't change anything.)

Note that once a dark launching feature flag has gotten to the point where you set the type to identity, you should remove the feature flag from the code and the database soon.

Nil

The nil type is also very simple. It is always off! and cannot be turned on.

(For the math-inclined, think of it like multiplying the number 0 to anything. It turns it to "nothing".)

Note that once a feature flag has gotten to the point where you set the type to nil, you should remove flag from the code and database AND remove the code it wraps soon.

In The Code

It may be best to see what this looks like when used in the code.

I've kept these examples very very simple. (Although often your usage may be this simple. Which can be a good thing.)

Let's take a look....

PHP

Here's how usage in PHP might look like:


// Some code

if (  Feature::isEnabled('MY_NEW_FEATURE')  ) {

    // Feature specific code here.

}

// More code

JavaScript

In JavaScript there would be something similar:


// Some code

if (  Feature.isEnabled("MY_NEW_FEATURE")  ) {

    // Feature specific code here.

}

// More code

Smarty

In Smarty, you could have a custom Smarty filter help you with this, as in:


<!-- Some code -->

{% if 'MY_NEW_FEATURE'|feature_is_enabled %}

    <!-- Feature specific code here. -->

{% if %}

<!-- More code -->

Or, you could even create your own custom Smarty tag block to handle this:


<!-- Some code -->

{% feature code='MY_NEW_FEATURE' %}

    <!-- Feature specific code here. -->

{% /feature %}

<!-- More code -->

CSS

In CSS you could do something such as:


<!-- Some code -->

<div class="feature feature-MY_NEW_FEATURE">

    <!-- Feature specific code here. -->

</div>

<!-- More code -->

Where the "feature" CSS class would have the definition:


div.feature {
    display:none;
}

And you could have a (perhaps PHP based) dynamically generated CSS file that defines the "feature-MY_NEW_FEATURE" CSS class and other feature flag CSS classes. For example:


<?php

header('Content-type:text/css');

?>

/* other feature flag codes */

<?php if(  Feature::isEnabled('MY_NEW_FEATURE')  ): ?>
    div.feature-MY_NEW_FEATURE {
        display:block;
    }
<?php endif; ?>

/* more feature flag codes */

You'd want these CSS definitions to appear or be included (with the HTML <link> element) in this order, to get the cascading to work properly!

More In The Code

Other patterns you'll see in the code will use (various styles of) else-clauses. For a PHP example:


// Some code

if (  Feature::isEnabled('MY_NEW_FEATURE')  ) {

    // Dark Launched Feature specific code here.

} else {

    // Non-Dark Launched version.

}

// More code

And also for a Smarty example:


<!-- Some code -->

{% feature code='MY_NEW_FEATURE' %}

    <!-- Dark Launched Feature specific code here. -->

{% feature_else %}

    <!-- Non-Dark Launched version. -->

{% /feature %}

<!-- More code -->

You may also need to explicitly look up if a specific ID in a list type has access to the dark launched feature or not. (Perhaps because there is no implicit context due to a logged in user, because you are running from a cron job or daemon job.) For example:


// Some code

if (  Feature::isEnabledInList('MY_NEW_FEATURE', $id)  ) {

    // Dark Launched Feature specific code here.

} else {

    // Non-Dark Launched version.

}

// More code

Database Design

There's of course, various ways you can design the database for dark launching features. And various places you can store said database.

You could put it right in the code as an associative array or a set of constants. You could use a relational database, such as MySQL. You could use a non-relation database, such as MongoDB, Riak or Redis. Etc.

Either way, conceptually one way you could design your feature database, which can be used for dark launching is like the following. (Note, I've tried to keep this very simple. You could modify this design to include foreign keys and referential integrity.)


Field Name Type Notes
id ID type.

In MySQL this would probably be an INT, such as 5812.

In MongoDB this would probably be an ObjectId, such as ObjectId( "47cc67093475061e3d95369d" ).
The primary key of the database.
code string or enumeration type. A unique key on the database.

This is used in the (PHP, JavaScript, CSS, etc) code, rather than the primary key, to put code into a feature for dark launching.

For example: "MY_NEW_FEATURE", "DEMOGRAPHIC_REPORT" or "TAG_CLOUD".

You may want to employ some kind of name spacing mechanism, for your dark launching feature flag codes, especially if you have multiple teams creating these and coordination becomes a bit challenging.

You should ensure that this field is indexed.
type string or enumeration type. The type determines the restrictions and permissions for who does and doesn't have access to the feature being dark launched.

Ex: boolean, (the various "flavors" of) percentage, (the various types of) list, identify, nil, etc.
value string or multi type. What is used to determine if/when the feature is on or off.

The meaning of this field is dependent of what the type field is!

If this is a boolean type, then it might have a value of "1" or "0".

If this is one of the "flavors" of the percentage type, the value would be any number from "0" to "100", which represents a percent. So, for example, a value of "14", then that means 14%.

If this is one of the "flavors" of the list type, the value would be a list of some sort of IDs. So, for example, a value of "10,581,589,1442,10018", then that means we have a list of 5 IDs: 10, 581, 589, 1442, 10018; and (if this isn't a list of users already then) somehow we use this to come up with a list of users. (Note, with this simple design you don't have foreign keys or referential integrity, for these IDs. You could alter the database design to include that if you wanted.)

For the identity and nil types the value doesn't matter!

You could also add fields that let QA specify that the feature has been QA. (And if it hasn't been, then no matter what the "value" is, it won't be turned on.)

A "description" field could also make the dark launch feature flags easier to manage.

If you wanted, you could also probably just use the "code" as the primary key of your database, and get rid of the "id" field altogether.

(Also, in case it needs to be said, you will probably want to engage in some kind of caching scheme. Perhaps using memcache and other methods. But that's really a larger system architecture question to address.)

Clean Up

It's a good idea to clean up your dark launching feature flag code once there's no longer a need for it. Remove it's usage in the code. Remove it's entry from the feature database.

Really, this is a very important part of their usage. And you should probably schedule this to be done. Perhaps put something in your calendar to remind you in 2 weeks or a month. Or, if your issue tracking system supports it, create a ticket to do this to be worked on in the near future.

--