Update configuration management documentation

readme/features-workflow.md → readme/configuration-management.md

@@ -1,36 +1,46 @@
-# Feature-based configuration management
-## Workflow and best practices for developers and TAs
+# Configuration management with BLT
+## Overview
 
-There are many ways to manage and deploy configuration in Drupal 8, one of which is the Features module. Features allows you to bundle related configuration files (such as a content type and its fields) into individual feature modules. Drupal treats features just like normal modules, but Features and its dependencies add some special sauce that allow features to not only provide default configuration (like normal modules), but to also update (track and import) changes to this configuration.
+BLT supports several methods of configuration management (CM) in Drupal 8. All of these rely to varying degrees on Drupal core's configuration entities, which can be "imported" into a database or "exported" to disk as yml files.
 
-A note on capitalization and terminology: "Features" is the module on drupal.org, while "features" are the individual collections of configuration on your own project. Also, Features relies heavily on the core configuration management system as well as the contributed Configuration Update module. It's easier to refer to this system collectively as Features, but keep in mind that many bugs and issues may actually relate to the underlying modules.
+BLT _strongly recommends_ a CM workflow based on the [Configuration Split](https://www.drupal.org/project/config_split) module, as described below. For most projects, this strikes the best balance of flexibility, reliability, and ease of maintenance and development. This document will also describe a Features-based workflow (analogous to most CM workflows in Drupal 7) that can better accomodate certain multisite architectures, but generally has a much higher development and maintenance overhead.
 
-## Overview of a Features-based workflow
-A good Features-based workflow makes it easy for developers to logically bundle configuration into portable version-controlled features that are easy to update. It also makes it easy to deploy these changes and verify that the active configuration on any given site matches what is stored in VCS.
+## General principles
+
+This section describes aspects of BLT's development and deployment process that are common to all CM workflows.
+
+### Basics of configuration management
+
+The primary goal of configuration management is to ensure that all configuration changes can be reviewed, tested, and predictably deployed to production environments. Some simple changes, such as changing a site's name or slogan, might have limited, atomic, and predictable effects, and therefore not require strict change management. Other types of changes, such as modifying field storage schemas, _always_ need to go through a review process. Additionally, different projects might have different degrees of risk tolerance. For instance, some might prefer that configuration be strictly read-only in production, prohibiting even the simple site name change above.
+
+A good CM workflow should be flexible enought to accomodate either of these use cases, and make it easy for developers to make and capture configuration changes, review and test these changes, and reliably deploy these changes to a remote environment.
 
 Generally speaking, a configuration change follows this lifecycle:
 
 1. A developer makes the change in her or his local environment.
-2. The developer uses the Features UI to export the configuration as a new or updated feature module.
-3. The developer commits the new or updated module to VCS and opens a pull request.
-4. Automated testing ensures that the feature can be installed from scratch on a new site as well as imported without conflicts on an existing site.
-5. After the feature is deployed, deployment hooks automatically import the new or updated configuration.
+2. The developer uses CM commands to export the configuration change to disk.
+3. The developer commits the new or updated configuration to VCS and opens a pull request.
+4. Automated testing ensures that the configuration can be installed from scratch on a new site as well as imported without conflicts on an existing site.
+5. After the change is deployed, deployment hooks automatically import the new or updated configuration.
 
-BLT-based projects already support this workflow, including automatic imports of features during updates (see the `setup:update` BLT task). The task `local:update` can be run by developers to replicate these deployment commands locally.
+The way that configuration is captured and deployed between environments in Drupal 8 is typically via YAML files. These YAML files, typically stored in a root `config` directory, or distributed with individual modules in `config/install` directories, represent individual configruation objects that can be synchronized with the active configuration in an environment's database via a variety of methods. See [documentation on core configuration management](https://www.drupal.org/docs/8/configuration-management). 
 
-Be aware that reverting all features and config on every deploy creates a risk of discarding server-side changes. This risk should be controlled by carefully managing permissions, and must be balanced against the greater risk of allowing for divergent configuration between your DB and VCS.
+This document address the challenge of capturing ("exporting") and deploying ("importing") configuration in a consistent way in order to support the workflow described above.
 
-## Best practices
+### How BLT handles configuration updates
 
-### Using bundles
-Features lets you define custom "bundles" that essentially let you train Features to support your project's individual workflow. At the most basic level, they are a way to namespace your features, so you'd want to choose a bundle name based on your project name (an "Acme" bundle would prefix all of your feature machine names with "acme_").
+BLT-based projects already support this workflow, including automatic imports of configuration updates. BLT defines a generic `setup:update` task that applies any pending database and configuration updates. This same task can be re-used locally, in a CI environment, or remotely (via the `local:update`, `ci:update`, and `deploy:update` wrappers, respectively) to ensure that configuration changes and database updates behave identically in all environments.
 
-Bundles can also do a lot more to make your life easier. For instance, Features automatically suggests features based around content types and taxonomies. If you'd also like to automatically create features for, say, custom block types, you can configure that preference in your custom bundle. You can also choose to always exclude certain types of configuration (such as permissions--see below), or always group certain types of configuration (such as field storage) into a "core" bundle, which is helpful for breaking circular dependencies.
+When you run one of these update commands, they perform the following updates (see `setup:config-import`):
 
-Note that as of version 8.3.3, Features can manage user roles and permissions, but not in an independent fashion. Permissions can only be exported for an entire role at once, unlike in D7 where you could export roles and their associated permissions separately. For this reason, Features excludes roles and permissions by default. If you wish to export them, change the "alters" setting on your Features bundle. ([reference](https://www.drupal.org/node/2383439))
+- Database updates: the equivalent of running `drush updb` or hitting `update.php`, this applies any pending database updates.
+- Config import: runs the core configuration-import command to import any configuration stored in the root `config` directory. This is either a full or partial import, depending on how BLT is configured.
+- Features import: runs features-import-all, which imports any configuration stored in a feature module's `config/install` directory.
+
+There are also pre- and post-config import hooks that you can use to run custom commands.
 
 ### Config vs content
-Drupal’s core config system (and by extension, the Features ecosystem) cannot be used to manage entities that Drupal considers to be content, such as nodes, taxonomy terms, and files. This can create conflicts when a configuration entity depends on a content entity, such as:
+Drupal’s config system cannot be used to manage entities that Drupal considers to be content, such as nodes, taxonomy terms, and files. This can create conflicts when a configuration entity depends on a content entity, such as:
 
 * You have a block type that includes a file upload field, and you want to place the block in a theme and export the block as a feature.
 * You have a view that is filtered by a static taxonomy term, and you want to export that view as a feature.
@@ -42,6 +52,45 @@ The solution is to make sure that the referenced content exists before the featu
 * Use the default_content module to export the referenced content as JSON files, and store these files with your feature or in a dependency.
 * Use Migrate and a custom module to create default content from any number of custom sources, such as JSON files stored with your feature.
 
+### Updating core and contributed modules
+
+Caution must be taken when updating core and contributed modules. If those updates make changes to a module’s configuration or schema, you must make sure to also update your exported configurations. Otherwise, the next time you run updates it will import a stale configuration schema and cause unexpected behavior.
+
+The best way to handle this is to always follow these steps when updating contributed and core modules:
+
+1. Start from a clean local setup or refresh. If you are using Features, ensure that there are no overridden configuration. The `cm.features.no-overrides` flag in [project.yml](https://github.com/acquia/blt/blob/8.x/template/blt/project.yml#L62) can assist with this by halting builds with overridden features.
+2. Use `composer update` to download the new module version(s).
+3. Run `drush updb` to apply any pending updates locally.
+4. If any updates were applied, check if they modified any stored configuration. If using Features, simply check for overridden features. If using core CM, re-export all configuration and check for any changes on disk.
+5. Re-export and commit any changes you found in the previous step, along with the updated `composer.json` and `composer.lock`.
+
+We need to find a better way of preventing this than manually monitoring module updates. Find more information in [these](https://www.drupal.org/node/2745685) [issues](https://github.com/acquia/blt/issues/842).
+
+## Configuration Split workflow
+
+BLT recommends using the Config Split module to manage configuration on most projects. For a description of how to use Config Split, see this excellent [blog post by Jeff Geerling](https://www.jeffgeerling.com/blog/2017/adding-configuration-split-drupal-site-using-blt-and-acquia-cloud).
+
+Note that this workflow currently has two major limitations. The first is that individual configurations can't be entirely excluded from configuration management. For instance, if you want administrators to be able to set the site name (as above) or create new contact forms / webforms in production, this would be difficult given the current state of the module. However, this should theoretically be possible.
+
+TODO: Update once we figure out the graylist, and with additional information about how to integrate with BLT.
+
+The second limitation is that it's difficult to define configuration that varies between sites in a multisite installation. Multisite installations that require highly customized bundles of configuration per-site might be better suited by a Features-based workflow.
+
+## Features-based workflow
+
+Features allows you to bundle related configuration files (such as a content type and its fields) into individual feature modules. Drupal treats features just like normal modules, but Features and its dependencies add some special sauce that allow features to not only provide default configuration (like normal modules), but to also update (track and import) changes to this configuration.
+
+Because of this more modular architecture, Features can be a better solution for certain multisite applications where functionality needs to be customized on a per-site basis. For instance, if you have a number of content types exported as separate features, but a given site only needs a subset of those content types, you could disable the unused features to make for a cleaner content editing experience. This also has the advantage of logically grouping functionality and custom code alongside its corresponding configuration.
+
+However, the downside to this more granular approach is that Features cannot make some of the same assumptions as the core configuration system, and relies much more heavily on the developer to manage the architecture and handle configuration changes that it can't. This makes the overall system much more error-prone and more of a burden to maintain.
+
+### Using bundles
+Features lets you define custom "bundles" that essentially let you train Features to support your project's individual workflow. At the most basic level, they are a way to namespace your features, so you'd want to choose a bundle name based on your project name (an "Acme" bundle would prefix all of your feature machine names with "acme_").
+
+Bundles can also do a lot more to make your life easier. For instance, Features automatically suggests features based around content types and taxonomies. If you'd also like to automatically create features for, say, custom block types, you can configure that preference in your custom bundle. You can also choose to always exclude certain types of configuration (such as permissions--see below), or always group certain types of configuration (such as field storage) into a "core" bundle, which is helpful for breaking circular dependencies.
+
+Note that as of version 8.3.3, Features can manage user roles and permissions, but not in an independent fashion. Permissions can only be exported for an entire role at once, unlike in D7 where you could export roles and their associated permissions separately. For this reason, Features excludes roles and permissions by default. If you wish to export them, change the "alters" setting on your Features bundle. ([reference](https://www.drupal.org/node/2383439))
+
 ### Testing features
 It’s important to ensure via automated testing that features can be installed on a new site as well as enabled on existing sites.
 
@@ -114,20 +163,6 @@ This depends on a helper function like this, which I suggest adding to your cust
       return $storage[$module]->read($id);
     }
 
-### Updating core and contributed modules
-
-Caution must be taken when updating core and contributed modules. If those updates make changes to a module’s configuration or schema, you must make sure to also update your exported features definitions. Otherwise, the next time you run features-import it will import a stale configuration schema and cause unexpected behavior.
-
-The best way to handle this is to always follow these steps when updating contributed and core modules:
-
-1. Start from a clean local setup or refresh. If you are using Features, ensure that there are no overridden features. The `cm.features.no-overrides` flag in [project.yml](https://github.com/acquia/blt/blob/8.x/template/blt/project.yml#L62) can assist with this by halting builds with overridden features.
-2. Use `composer update` to download the new module version(s).
-3. Run `drush updb` to apply any pending updates locally.
-4. If any updates were applied, check if they modified any stored configuration. If using Features, simply check for overridden features. If using core CM, re-export all configuration and check for any changes on disk.
-5. Re-export and commit any changes you found in the previous step, along with the updated `composer.json` and `composer.lock`.
-
-We need to find a better way of preventing this than manually monitoring module updates. Find more information in [these](https://www.drupal.org/node/2745685) [issues](https://github.com/acquia/blt/issues/842).
-
 ### Overriding configuration
 
 Drupal normally prevents modules from overriding configuration that already exists in the system, producing an exception like this:
@@ -143,4 +178,6 @@ If you need to override the default configuration provided by another project (o
 
 ### Other gotchas
 
+Be aware that reverting all features and config on every deploy creates a risk of discarding server-side changes. This risk should be controlled by carefully managing permissions, and must be balanced against the greater risk of allowing for divergent configuration between your DB and VCS.
+
 Features is a ground-up rewrite in Drupal 8 and is maturing quickly, but may still have some traps. Developers should keep a close eye on exported features, and architects need to carefully review features in PRs for the gotchas and best practices listed above.