Tarantool config storage #4056

andreyaksenov · 2024-02-21T14:18:10Z

Updated topics:

Updated intro: Configuration
Updated the Centralized configuration section in Configuration overview.
Reworked Centralized configuration storages to make it storage-agnostic. Added diagrams that demonstrate the difference between local and centralized configs.

New and updated samples used in topics and reference pages:

Created a new centralized_config folder for samples related to centralized configurations.
Moved the existing etcd sample (etcd) to this folder and renamed it to config_etcd.
Removed the etcd_full sample. The updated config_etcd sample now includes additional settings (auth and timeout).
Added the etcd_config_storage.sh script for quick setting up a protected etcd storage.
Created the tarantool_config_storage sample that shows how to set up a Tarantool-based storage.
Created the config_storage sample that shows how to connect to the Tarantool-based storage.
Added the source.yaml file to a tt environment directory. Can be used to test tt cluster publish.

What's new in 3.0:

Updated literalinclude settings in a sample inside Centralized configuration (EE).

Reference:

Added a new config.storage.* reference section.
Added a new roles section without descriptions. Will be documented in Roles #3702.
Documented the config module API - it includes the config.storage API.
Added sample to conn:watch().

askalt

Thank you! lgtm

p7nov

Good job; some notes from my side regarding lack of explanations for some conceptual and specific things.

p7nov · 2024-03-06T08:13:22Z

doc/concepts/configuration.rst


-Tarantool enables you to store configuration data in one reliable place using `etcd <https://etcd.io/>`_.
+Tarantool enables you to store configuration data in one reliable place, for example, a Tarantool or etcd-based configuration storage.


These are not for example - they are exact two supported ways of centralized config. Let's explain this clear from the start.

p7nov · 2024-03-06T08:15:08Z

doc/concepts/configuration.rst


-    ..  literalinclude:: /code_snippets/snippets/config/instances.enabled/etcd/config.yaml
+3.  Configure a connection to the storage by providing a local YAML configuration with an endpoint address and key prefix in the ``config`` section:


Nit: there can also be other connection parameters, maybe say it in a more general wording?

This sample shows this minimum configuration and works more like a demo version of a centralized storage.

p7nov · 2024-03-06T08:16:31Z

doc/concepts/configuration/configuration_etcd.rst

@@ -1,94 +1,228 @@
 ..  _configuration_etcd:


Should we change this anchor to reflect the new content?

p7nov · 2024-03-06T08:22:06Z

doc/concepts/configuration/configuration_etcd.rst


+Tarantool enables you to store configuration data in one place using a Tarantool or etcd-based storage.
+With a :ref:`local YAML configuration <configuration_file>`, you need to make sure that all cluster instances use identical configuration files:


I don't really like the explanation about the YAML file and the diagram with files on this page. Its a practical information for those who use local file config, while this page is not for them.
I think the proper place for the first diagram is the Configuration in a file section.
The centralized diagram can be used both on the general Configuration (Centralized section) page and here.

p7nov · 2024-03-06T08:26:02Z

doc/concepts/configuration/tarantool_config_centralized.png

Why is the top block Configuration servers is left-aligned? I think the pic will look better with this block centered.

To keep the space for the potential diagram with TCM/tt/other tools (for example, tarantool_config_centralized_tcm.png, which is not used yet).

p7nov · 2024-03-06T08:47:05Z

doc/concepts/configuration/configuration_etcd.rst

+
+.. _centralized_configuration_storage_tarantool_interact:
+
+Interacting with the storage


Not sure why this section is here. I expected the section about setting up etcd storage right here.
This content seems relevant after at least one config is published to the storage.

dropped the Interacting with the storage and Watching configuration updates sections

moved the net.box-related content to a reference page

created a new Publishing configuration using the ‘config’ module section that acts as an entry point for the config.storage API

p7nov · 2024-03-06T08:47:33Z

doc/concepts/configuration/configuration_etcd.rst

-Publishing a cluster's configuration to etcd
--------------------------------------------
+Publishing a cluster's configuration
+------------------------------------


An intro sentence about ways to publish?

I hope, section names are self-explanatory and the intro won't add any additional info.

p7nov · 2024-03-06T08:49:08Z

doc/concepts/configuration/configuration_etcd.rst


 .. code-block:: console

-    $ tt cluster publish "http://localhost:2379/example" instances.enabled/app/cluster.yaml
+    $ tt cluster publish "http://localhost:2379/myapp" source.yaml


I think that credentials are required in most cases, so let's maybe show them to save readers some time on reading the tt cluster reference?

p7nov · 2024-03-06T08:50:29Z

doc/concepts/configuration/configuration_etcd.rst



-.. _etcd_publishing_configuration_etcdctl:
+.. _centralized_configuration_storage_publish_config_etcdctl:

 Publishing configuration using etcdctl
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


This maybe obvious but I'd start this section from an intro that it's only for etcd.

Because in the previous section we said that there is no difference for tt between etcd and Tarantool storage :)
Somebody can be confused.

p7nov · 2024-03-06T08:55:44Z

doc/concepts/configuration/configuration_etcd.rst

+Configuring connection to a storage
+-----------------------------------
+
+To use a centralized cluster's configuration, you need to provide connection settings in a local configuration file.


This is where I would put a thought from your intro: despite we are talking about the centralized config, it's the reader's job to keep these files consistent and available on all cluster hosts.

This reads as centralized cluster's. Maybe rephrase to
To use a configuration from a centralized storage for your cluster... ?

p7nov

Approved; just a couple more questions.

p7nov · 2024-03-07T05:14:21Z

doc/concepts/configuration/configuration_etcd.rst

+Tarantool-based storage
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To make a replica set act as a configuration storage, use the built-in ``config.storage`` :ref:`role <configuration_reference_roles_options>`.


Can this replicaset be used for storing data at the same time? If not, this should be stated explicitly.

I think, it can - it's just a regular replica set.

p7nov · 2024-03-07T05:17:53Z

doc/concepts/configuration/configuration_etcd.rst

+Configuring connection to a storage
+-----------------------------------
+
+To use a centralized cluster's configuration, you need to provide connection settings in a local configuration file.


This reads as centralized cluster's. Maybe rephrase to
To use a configuration from a centralized storage for your cluster... ?

andreyaksenov force-pushed the tarantool-config-storage branch 26 times, most recently from 8dc9157 to 12f10f9 Compare February 26, 2024 07:28

andreyaksenov linked an issue Feb 26, 2024 that may be closed by this pull request

[Config] Document the config module API #3662

Closed

andreyaksenov force-pushed the tarantool-config-storage branch 3 times, most recently from f8e49d2 to 3219f51 Compare February 26, 2024 12:14

andreyaksenov force-pushed the tarantool-config-storage branch 7 times, most recently from b16f435 to 462dedd Compare February 29, 2024 12:50

andreyaksenov marked this pull request as ready for review February 29, 2024 12:53

andreyaksenov force-pushed the tarantool-config-storage branch 7 times, most recently from 053bebe to 5e7e9e3 Compare February 29, 2024 14:29

andreyaksenov requested a review from askalt February 29, 2024 14:32

andreyaksenov force-pushed the tarantool-config-storage branch from 5e7e9e3 to 4262a0b Compare March 1, 2024 07:29

Centralized storage

4c38036

andreyaksenov force-pushed the tarantool-config-storage branch from 4262a0b to 4c38036 Compare March 1, 2024 07:49

askalt approved these changes Mar 5, 2024

View reviewed changes

andreyaksenov requested a review from p7nov March 6, 2024 07:07

p7nov requested changes Mar 6, 2024

View reviewed changes

andreyaksenov force-pushed the tarantool-config-storage branch from de3c76c to fd572a0 Compare March 6, 2024 12:07

andreyaksenov requested a review from p7nov March 6, 2024 12:16

p7nov approved these changes Mar 7, 2024

View reviewed changes

Centralized storage: update per review

3ffa4fd

andreyaksenov force-pushed the tarantool-config-storage branch from fd572a0 to 3ffa4fd Compare March 7, 2024 07:23

andreyaksenov merged commit f56c25d into latest Mar 7, 2024
1 check passed

andreyaksenov deleted the tarantool-config-storage branch March 7, 2024 07:32

p7nov pushed a commit that referenced this pull request Apr 4, 2024

Tarantool config storage (#4056)

7911187

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tarantool config storage #4056

Tarantool config storage #4056

andreyaksenov commented Feb 21, 2024 •

edited

Loading

askalt left a comment

p7nov left a comment

p7nov Mar 6, 2024

p7nov Mar 6, 2024

andreyaksenov Mar 6, 2024

p7nov Mar 6, 2024

p7nov Mar 6, 2024

p7nov Mar 6, 2024

andreyaksenov Mar 6, 2024 •

edited

Loading

p7nov Mar 6, 2024

andreyaksenov Mar 6, 2024 •

edited

Loading

p7nov Mar 6, 2024

andreyaksenov Mar 6, 2024

p7nov Mar 6, 2024

p7nov Mar 6, 2024

p7nov Mar 6, 2024

p7nov Mar 6, 2024

p7nov Mar 7, 2024

p7nov left a comment

p7nov Mar 7, 2024

andreyaksenov Mar 7, 2024

p7nov Mar 7, 2024


		Tarantool enables you to store configuration data in one reliable place using `etcd <https://etcd.io/>`_.
		Tarantool enables you to store configuration data in one reliable place, for example, a Tarantool or etcd-based configuration storage.


		.. literalinclude:: /code_snippets/snippets/config/instances.enabled/etcd/config.yaml
		3. Configure a connection to the storage by providing a local YAML configuration with an endpoint address and key prefix in the ``config`` section:


		Tarantool enables you to store configuration data in one place using a Tarantool or etcd-based storage.
		With a :ref:`local YAML configuration <configuration_file>`, you need to make sure that all cluster instances use identical configuration files:


		.. _centralized_configuration_storage_tarantool_interact:

		Interacting with the storage

Tarantool config storage #4056

Tarantool config storage #4056

Conversation

andreyaksenov commented Feb 21, 2024 • edited Loading

askalt left a comment

Choose a reason for hiding this comment

p7nov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyaksenov Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyaksenov Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p7nov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyaksenov commented Feb 21, 2024 •

edited

Loading

andreyaksenov Mar 6, 2024 •

edited

Loading

andreyaksenov Mar 6, 2024 •

edited

Loading