Feature Toggles

We are using feature toggles with the Flipper gem to control availability of the various components.

Toggle Types

We are making use of two different types of toggles in our application

Release toggles

Release toggles hide in-progress features from our users. They are supposed to go away once we declare a feature to be "stable". Usually, they are accompanied by a feature release like a blog post, documentation updates, other marketing or maybe even a full OBS version release!

Operations toggles

Operations toggles are for finished features that might not be interesting for everybody. For instance distribution specific workflows, features that would use many resources or things like that. Which feature needs to be toggle-able is a case by case decision. The toggles however are managed by the config/options.yml files and in our code like

unless CONFIG['my_feature'] == :off
...
end

Best Practices

• Keep features behind a toggle as small as possible. Small features are easier to handle and faster to go to rollout.

Workflow

For us, features usually go through different phases (new->beta->rollout->released). There are different things you should do for each of those phases.

We use Flipper which supports more sophisticated things and is easily extendable. Please check its documentation to get more information about to use it.

Your feature is new and currently unusable

Instead of development branches, we make use of feature toggles to hide in-progress code paths. You can put the code behind the switch, e.g.

Flipper.enabled?(:feature_name) do
 ...
end

Features are by default off in all environments. You can enable them on your instance by running this in your console.

Flipper.enable(:my_feature)

You want to test your feature additionally to the defaults

If you have a specific test for your new feature or you want to run an existing spec also with the feature enabled, simply enable it in a before block.

describe '#add_globalrole' do
  before do
    Flipper[:my_feature].enable
  end
  
  it 'does something amazing with my feature enabled' do
    ...
  end
end

A good example is to look into this spec

You want all OBS developers to see/test your feature

Extend the Flipper initializer with something like:

if Rails.env.development?
  Flipper[:my_feature].enable unless Flipper[:my_feature].enabled?
end

You want all OBS developers to see/test your feature on production

You want to try out some features on production before enabling them for customers. For this, you need to enable it for the staff group on the production console of your instance. This can also be done in the Flipper UI by clicking on the feature and enabling it for the staff group. You need to have the Staff role to access the Flipper UI, otherwise you get a 404.

Flipper.enable(:my_feature, :staff)

You want customers to test your feature

Bigger features should be first offered in the beta program, so as soon as the users join the beta program, they will be able to test your feature and give you feedback before it goes live.

For this, you need to add your feature to ENABLED_FEATURE_TOGGLES in the Flipper initializer. This will register a group for your feature whenever the changes are deployed. You then need to enable your feature for this group, this is done with:

# This is already part of our deployment workflow, so only do this in your development environment
bundle exec rake flipper:enable_features_for_group

The feature is now part of the beta program and users can manage their beta features.

You want to enable your feature, but not for everyone at once.

You might also want to avoid to release your big feature to all your users at once because of the support load or other concerns. For this, we have another group called rollout. So once your feature is ready to roll out, you can move it from its feature group to rollout. Feature groups have the same name as their feature. This can also be done in the Flipper UI by clicking on the feature and replacing its feature group by the rollout group. You need to have the Staff role to access the Flipper UI, otherwise you get a 404.

Flipper.disable(:my_feature, :my_feature) # Feature groups have the same name as their feature
Flipper.enable(:my_feature, :rollout)

and then change the in_rollout attribute for specific users. Like all the beta users.

User.where(in_beta: true).in_batches.update_all(in_rollout: true)

Steps for Rolling Out

There are some rake tasks that can help us with the rollout.

1. We need to announce the rollout before it starts. Create a status message with the severity Announcement which contains a link to a blog post explaining the new feature (which will be visible for everyone from now on).

2. Move all the existing users (including _nobody_) out of the rollout program before enabling the new feature for the rollout group.

rails rollout:all_off
rails rollout:anonymous_off

3. Enable the new feature for the rollout group. This step can also be done in the Flipper UI by clicking on the feature and enabling it for the rollout group. You need to have the Staff role to access the Flipper UI, otherwise you get a 404.

Flipper.enable(:my_feature, :rollout)

NOTE: New users created from that moment on, will be in the rollout program by default. This is ok, they don't need to see the old version of the feature which is going to disappear soon.

4. Start moving batches/groups of users to the rollout program. Depending on the kind of feature, you can decide to wait some period of time between one group and the next one, in case you expect an increase of support load.

# Examples: 
rails rollout:from_beta
rails rollout:recently_logged_users

5. The anonymous user, _nobody_, is the last user you should move to the rollout program.

rails rollout:anonymous_on

6. Once the rollout is finished, remove the feature toggle from the code and merge the feature's code into the codebase. This should happen at a later stage to have the feature rolled out for at least one month, just in case this needs to be reverted. Better safe than sorry!

7. Disable the feature completely by deleting it. Follow these steps.

NOTE: It is a good practice to move back all the users out of the rollout at the end of the process. However, it isn't compulsory since this is the first step in the next rollout.

Delete a Feature Flag

Once a feature is rolled out and its code is integrated into the codebase, it's time to remove the feature flag from Flipper. Follow these steps:

• Remove it from ENABLED_FEATURE_TOGGLES in the Flipper initializer.
• Remove it from Flipper. Through the Flipper UI (you need to have the Staff role) or running Flipper.remove(:my_feature).
• Restart the server.

List feature toggles

In the Rails console, run: Flipper::Adapters::ActiveRecord::Feature.all.

This lists all feature toggles which can be used in the code.

List feature gates

In the Rails console, run: Flipper::Adapters::ActiveRecord::Gate.all.

This tells us which features have been enabled for which users/groups and when they have been enabled (look at the created_at field).

Feature Toggles Planning

Feature Name	Enabled At	Rolled Out In OBS At	Rolled Out In IBS At	Deleted At	Description
bootstrap	Dec. 2018	Jul. 2019	untracked	Sept. 2019	Migrate the whole UI to Bootstrap.
responsive_ux	27/01/2020	14/01/2021	untracked	17/03/2021	Redesign of the layout, including typography, new navigation structure and redesign of Build Results and Sponsor sections, among others.
notifications_redesign	10/08/2020	17/12/2021	21/12/2021	16/05/2022	Introducing notifications page.
user_profile_redesign	21/10/2020	29/03/2021	10/11/2021	14/12/2021	User profile page redesign.
trigger_workflow	31/05/2021				Better SCM and CI integration with the trigger workflow.
new_watchlist	22/02/2022				New implementation of watchlist including packages and requests apart from projects.

Architecture for Management of Beta Features

Whenever a user disables a beta feature, we create a record for this user and beta feature in the table disabled_beta_features. Whenever a user enables back a beta feature, we delete the record which was previously created for this user and beta feature in the table disabled_beta_features. This way, since we assume most users will have most, if not all beta features enabled, the table footprint stays small.

Each feature toggle has its own group. This is how users can enable or disable a specific feature. This logic is defined in the Flipper initializer when we register those groups.

Here's a class diagram on how the data is stored:

Code for the diagram (GitHub Wiki doesn't render Mermaid diagrams yet, so use Mermaid Live Editor):

  classDiagram
  disabled_beta_features "1" <-- "0..n" flipper_features: key --> name
  disabled_beta_features "1" <-- "0..n" users: id --> user_id

  class disabled_beta_features {
    name
    user_id
  }

  class users {
    id
    ...
  }

  class flipper_features {
    <<existing internal table from flipper-active_record>>
    key
    ...
  }

Loading

RPM package releases

We don't do any specific configuration on the RPM package other than the defaults. So no beta or roll-out features are going to be configured in the RPM package.