Feature Toggles

• Toggle Types
  • Release/Feature toggles
  • Operations toggles
  • RPM package releases
• Best Practices
• Feature Toggles Phases
  • 'New' phase. The feature is only enabled for developers or Staff
  • 'Beta' phase. You want customers to test your feature
  • 'Rollout' phase. The feature is enabled out of beta
  • 'Release' phase. Clean up after ourselves
• Tricks for Feature Toggles
  • List feature toggles
  • List feature gates
• Architecture for Management of Beta Features
• Feature Toggles Planning

We are using feature toggles with the Flipper gem to control availability of the various components.

Toggle Types

We are making use of two different types of toggles in our application

Release/Feature toggles

Release/feature toggles hide in-progress features from our users. They are supposed to go away once we declare a feature to be "stable". Usually, they are accompanied by a feature release like a blog post, documentation updates, other marketing or maybe even a full OBS version release!

Operations toggles

Operations toggles are for finished features that might not be interesting for everybody. For instance distribution specific workflows, features that would use many resources or things like that. Which feature needs to be toggle-able is a case by case decision. The toggles however are managed by the config/options.yml files and in our code like

unless CONFIG['my_feature'] == :off
...
end

RPM package releases

We don't do any specific configuration on the RPM package other than the defaults. So no beta or roll-out features are going to be configured in the RPM package.

Best Practices

• Keep features behind a toggle as small as possible. Small features are easier to handle and faster to go to rollout.

Feature Toggles Phases

For us, features usually go through different phases (new -> beta -> rollout -> released). There are different things you should do for each of those phases.

We use Flipper which supports more sophisticated things and is easily extendable. Please check its documentation to get more information about to use it.

'New' phase. The feature is only enabled for developers or Staff

Your feature is new and currently unusable

Instead of development branches, we make use of feature toggles to hide in-progress code paths. You can put the code behind the switch, e.g.

Flipper.enabled?(:feature_name) do
 ...
end

Features are by default off in all environments. You can enable them on your instance by running this in your console.

Flipper.enable(:my_feature)

You want to test your feature additionally to the defaults

If you have a specific test for your new feature or you want to run an existing spec also with the feature enabled, simply enable it in a before block.

describe '#add_globalrole' do
  before do
    Flipper[:my_feature].enable
  end
  
  it 'does something amazing with my feature enabled' do
    ...
  end
end

A good example is to look into this spec

You want all OBS developers to see/test your feature

Extend the Flipper initializer with something like:

if Rails.env.development?
  Flipper[:my_feature].enable unless Flipper[:my_feature].enabled?
end

You want all OBS developers to see/test your feature on production

You want to try out some features on production before enabling them for customers. For this, you need to enable it for the staff group on the production console of your instance. This can also be done in the Flipper UI by clicking on the feature and enabling it for the staff group. You need to have the Staff role to access the Flipper UI, otherwise you get a 404.

Flipper.enable(:my_feature, :staff)

'Beta' phase. You want customers to test your feature

Bigger features should be first offered in the beta program, so as soon as the users join the beta program, they will be able to test your feature and give you feedback before it goes live.

For this, you need to add your feature to ENABLED_FEATURE_TOGGLES in the Flipper initializer. This will register a group for your feature whenever the changes are deployed. You then need to enable your feature for this group, this is done with:

# This is already part of our deployment workflow, so only do this in your development environment
bundle exec rake flipper:enable_features_for_group

_{It's done here in our deployment workflow.}

The feature is now part of the beta program and users can manage their beta features. To revert this, thus removing the feature from the beta program, undo the changes you did to the Flipper initializer.

'Rollout' phase. The feature is enabled out of beta

Once the feature is stable, we push it out of the beta program (roll it out). We can either enable the feature for all users at once or do it in batches.

Follow these steps:

• Write down the blog post announcing the rollout. Do not merge it yet.
• Write down the texts for the different communication channels (status message, twitter, email, Slack). Do not publish them yet.
• [Optional] Create a PR with a data migration or rake task(s) if there is something to migrate or run specifically. Don't deploy until everything is prepared.
• Go to the production instance and move users to the rollout group. Learn how to do so for all users or in batches.
• The following steps should be executed one right after the other with no delay:
  • Deploy the PR when the packages are built.
  • [Optional] Run the rake task(s) if any (the data migration is run in the deployment process).
  • Enable the feature for the rollout group in Flipper. As we describe here.
  • Merge the blog post PR.
  • Announce the rollout on the rest of the communication channels. Create the status message with severity Announcement and scope Logged in users.
• Note down the rollout date here.

Move all the users to rollout group

There are some rake tasks that can help us with the rollout. To enable the feature for the rollout group we first have to move all users to that group:

rails rollout:all_on
rails rollout:anonymous_on

NOTE: New users created from that moment on, will be in the rollout program by default. This is ok, they don't need to see the old version of the feature which is going to disappear soon.

Move users to rollout group in batches

We might prefer to move batches of users to the rollout group, not all at once, because of the support load or other concerns.

1. Move all users out of the rollout by running in rails console:

rails rollout:all_off
rails rollout:anonymous_off

2. Start moving batches/groups of users to the rollout group. Depending on the kind of feature, you can decide to wait some period of time between one group and the next one, in case you expect an increase of support load.

# Examples: 
rails rollout:from_beta
rails rollout:recently_logged_users

3. The anonymous user, _nobody_, is the last user you should move to the rollout program.

rails rollout:anonymous_on

NOTE: New users created from that moment on, will be in the rollout program by default. This is ok, they don't need to see the old version of the feature which is going to disappear soon.

Add the rollout group to the feature in Flipper

To enable the new feature only for the rollout group you can move it from its feature group to rollout. Feature groups have the same name as their feature.

This can be done in the Flipper UI by clicking on the feature and replacing its feature group by the rollout group. You need to have the Staff role to access the Flipper UI, otherwise you get a 404.

This can also be done from rails console in production:

Flipper.disable(:my_feature, :my_feature) # Feature groups have the same name as their feature
Flipper.enable(:my_feature, :rollout)

'Release' phase. Clean up after ourselves

Once the rollout is finished and after a prudencial period of time, we can remove the feature toggle from the code and merge the feature's code into the codebase.

After that, we can safely disable the feature completely by deleting it. Follow the steps below.

Delete a Feature Toggle

Once a feature is rolled out and its code is integrated into the codebase, it's time to remove the feature toggle from Flipper. Follow these steps:

• Remove it from ENABLED_FEATURE_TOGGLES in the Flipper initializer.
• Remove it from Flipper. Through the Flipper UI (you need to have the Staff role) or running Flipper.remove(:my_feature).
• Restart the server.

Additional steps:

• Set the date of the Deleted At field for the deleted feature toggle in the Feature Toggles Planning table.
• Inform the BuildOps team about this for the internal SUSE instance.

Tricks for Feature Toggles

List feature toggles

In the Rails console, run: Flipper::Adapters::ActiveRecord::Feature.all.

This lists all feature toggles which can be used in the code.

List feature gates

In the Rails console, run: Flipper::Adapters::ActiveRecord::Gate.all.

This tells us which features have been enabled for which users/groups and when they have been enabled (look at the created_at field).

Architecture for Management of Beta Features

Whenever a user disables a beta feature, we create a record for this user and beta feature in the table disabled_beta_features. Whenever a user enables back a beta feature, we delete the record which was previously created for this user and beta feature in the table disabled_beta_features. This way, since we assume most users will have most, if not all beta features enabled, the table footprint stays small.

Each feature toggle has its own group. This is how users can enable or disable a specific feature. This logic is defined in the Flipper initializer when we register those groups.

Here's a class diagram on how the data is stored:

  classDiagram
  disabled_beta_features "1" <-- "0..n" flipper_features: key --> name
  disabled_beta_features "1" <-- "0..n" users: id --> user_id

  class disabled_beta_features {
    name
    user_id
  }

  class users {
    id
    ...
  }

  class flipper_features {
    <<existing internal table from flipper-active_record>>
    key
    ...
  }

Loading

Feature Toggles Planning

Feature Name	Enabled At	Rolled Out In OBS At	Rolled Out In IBS At	Deleted At	Description
bootstrap	Dec. 2018	Jul. 2019	untracked	Sept. 2019	Migrate the whole UI to Bootstrap.
responsive_ux	27/01/2020	14/01/2021	untracked	17/03/2021	Redesign of the layout, including typography, new navigation structure and redesign of Build Results and Sponsor sections, among others.
notifications_redesign	10/08/2020	17/12/2021	21/12/2021	16/05/2022	Introducing notifications page.
user_profile_redesign	21/10/2020	29/03/2021	10/11/2021	14/12/2021	User profile page redesign.
trigger_workflow	31/05/2021	21/09/2022	22/09/2022	14/02/2023	Better SCM and CI integration with the trigger workflow.
new_watchlist	22/02/2022	26/09/2022	05/10/2022		New implementation of watchlist including packages and requests apart from projects.
request_show_redesign	20/07/2022				Redesign of the request pages to improve the collaboration workflow.
content_moderation	15/09/2023				Moderation of various user generated content.
color_themes	19/01/2024				Integrating the color theme feature with the dark mode.
request_index	17/07/2024				Redesign of listing requests