records.yaml - Add support to load multiple YAML docs from the same file and let traffic_ctl to modify a records.yaml file (#9404)

* records.yaml - Add support to load multiple YAML docs from the same
stream/file. This will work as the legacy records.config where a later
variable would override a previous value.

* traffic_ctl: Allow config set/get from a yaml configuration file.

* DOC: Add supported documentation

Author: brbzull0

doc/admin-guide/files/records.yaml.en.rst

@@ -31,9 +31,7 @@ records.yaml
 
 The :file:`records.yaml` file (by default, located in
 ``/usr/local/etc/trafficserver/``) is a YAML base configuration file used by
-the |TS| software. Many of the fields in :file:`records.yaml` are set
-automatically when you set configuration options with :option:`traffic_ctl config set`. After you
-modify :file:`records.yaml`, run the command :option:`traffic_ctl config reload`
+the |TS| software. After you modify :file:`records.yaml`, run the command :option:`traffic_ctl config reload`
 to apply the changes.
 
 .. note::
@@ -47,7 +45,8 @@ to apply the changes.
 YAML structure
 ==============
 
-All fields are located inside the ``ts`` root node.
+All fields are located inside the ``ts`` root node. ATS supports reading multiple documents from
+the same YAML stream, subsequent documents overrides earlier fields.
 
 
 .. code-block:: yaml

doc/appendices/command-line/traffic_ctl.en.rst

@@ -170,16 +170,17 @@ Display the current value of a configuration record.
 .. program:: traffic_ctl config get
 .. option:: --records
 
-   If this flag is provided, :option:`traffic_ctl config get` will emit results in
-   :file:`records.yaml` format.
+   If this flag is provided, :option:`traffic_ctl config get` will emit results in internal ats variable format.
+
+   The option :ref:`--cold <traffic_ctl_config_cold>` is available to get the values from a file.
 
 .. program:: traffic_ctl config
 .. option:: match [--records] REGEX [REGEX...]
 
    :ref:`admin_lookup_records`
 
    Display the current values of all configuration variables whose names match the given regular
-   expression. The ``--records`` flag has the same behavior as :option:`traffic_ctl config get
+   expression. The ``--records`` flag has the same behavior as `traffic_ctl config get
    --records`.
 
 .. program:: traffic_ctl config
@@ -203,6 +204,77 @@ Display the current value of a configuration record.
    documentation for a list of the configuration variables you can specify. Note that this is not a
    synchronous operation.
 
+   Supports the following options.
+
+
+.. _traffic_ctl_config_cold:
+
+   .. option:: --cold, -c [filename]
+
+
+   This option indicates to `traffic_ctl` that the action should be performed on a configuration file instead of using the ATS RPC
+   facility to store the new value. `traffic_ctl` will save the value in the passed `filename`, if no `filename` passed, then the sysconfig
+   :file:`records.yaml` will be attempted to be used.
+
+   ATS supports parsing multiple documents from the same YAML stream, so if you attempt to set a variable on a document with
+   none, one or multiple documents then a new document will be appended. In case you want to modify an existing field then `-u` option
+   should be passed, so the latest(top to bottom) field will be modified, if there is no variable already set in any of the documents,
+   then the new variable will be set in the latest document of the stream.
+
+   Specifying the file name is not needed as `traffic_ctl` will try to use the build(or the runroot if used) information to figure
+   out the path to the `records.yaml`.
+
+   If the file exists and is empty a new document will be created. If a file does not exist, an attempt to create a new file will be done.
+
+   This option(only for the config file changes) lets you use the prefix `proxy.config.` or `ts.` for variable names, either would work.
+   If different prefix name is prepend, then traffic_ctl will work away using the provided variable name, this may not be what is intended
+   so, make sure you use the right prefixes.
+
+
+   Appending a new field in a records.yaml file.
+
+   .. code-block:: bash
+
+      $ traffic_ctl config set proxy.config.diags.debug.enabled 1 -c records.yaml
+      $ cat records.yaml
+      ts:
+      ...
+      # Document modified by traffic_ctl Mon Feb 13 23:07:15 2023
+      #
+      ---
+      ts:
+         diags:
+            debug:
+               enabled: 1
+
+   .. note::
+
+      The following options are only considered if ``--cold, -c`` is used, ignored otherwise.
+
+   .. option:: --update -u
+
+      Update latest field present. If there is no variable already set in any of the documents, then the new variable will be set in
+      the latest document.
+
+   .. option:: --type, -t int | float | str
+
+      Inject a tag information on the modified/new field, this is useful when you set a non registered record inside ATS.
+
+      .. code-block:: bash
+
+         $ traffic_ctl config set ts.some.plugin.config.max 100 -t int -c records.yaml
+         $ cat records.yaml
+         ...
+         # Document modified by traffic_ctl Mon Feb 13 23:07:15 2023
+         #
+         ---
+         ts:
+            some:
+               plugin:
+                  config:
+                    max: !<tag:yaml.org,2002:int> 100
+
+
 .. program:: traffic_ctl config
 .. option:: status
 

src/records/RecConfigParse.cc

@@ -268,9 +268,14 @@ swoc::Errata
 RecYAMLConfigFileParse(const char *path, RecYAMLNodeHandler handler)
 {
   try {
-    auto root = YAML::LoadFile(path);
-    if (auto ret = ParseRecordsFromYAML(root, handler); !ret.empty()) {
-      return ret;
+    auto nodes = YAML::LoadAllFromFile(path);
+    // We will parse each doc from the stream, subsequently config node will overwrite
+    // the value for previously loaded records. This would work similar to the old
+    // records.config which a later CONFIG variable would overwrite a previous one.
+    for (auto const &root : nodes) {
+      if (auto ret = ParseRecordsFromYAML(root, handler); !ret.empty()) {
+        return ret;
+      }
     }
   } catch (std::exception const &ex) {
     return swoc::Errata(ERRATA_ERROR, "Error parsing {}. {}", path, ex.what());

src/traffic_ctl/CtrlCommands.cc

@@ -24,11 +24,16 @@
 #include "CtrlCommands.h"
 #include "jsonrpc/CtrlRPCRequests.h"
 #include "jsonrpc/ctrl_yaml_codecs.h"
+#include "utils/yaml_utils.h"
+#include "swoc/TextView.h"
+#include "swoc/BufferWriter.h"
+#include "swoc/bwf_base.h"
+
 namespace
 {
 /// We use yamlcpp as codec implementation.
 using Codec = yamlcpp_json_emitter;
-
+} // namespace
 const std::unordered_map<std::string_view, BasePrinter::Options::OutputFormat> _Fmt_str_to_enum = {
   {"pretty", BasePrinter::Options::OutputFormat::PRETTY},
   {"legacy", BasePrinter::Options::OutputFormat::LEGACY},
@@ -59,7 +64,6 @@ parse_print_opts(ts::Arguments *args)
 {
   return {parse_format(args)};
 }
-} // namespace
 
 //------------------------------------------------------------------------------------------------------------------------------------
 CtrlCommand::CtrlCommand(ts::Arguments *args) : _arguments(args) {}
@@ -75,7 +79,7 @@ CtrlCommand::execute()
 }
 
 std::string
-CtrlCommand::invoke_rpc(std::string const &request)
+RPCAccessor::invoke_rpc(std::string const &request)
 {
   if (_printer->print_rpc_message()) {
     std::string text;
@@ -96,15 +100,15 @@ CtrlCommand::invoke_rpc(std::string const &request)
 }
 
 shared::rpc::JSONRPCResponse
-CtrlCommand::invoke_rpc(shared::rpc::ClientRequest const &request)
+RPCAccessor::invoke_rpc(shared::rpc::ClientRequest const &request)
 {
   std::string encodedRequest = Codec::encode(request);
   std::string resp           = invoke_rpc(encodedRequest);
   return Codec::decode(resp);
 }
 
 void
-CtrlCommand::invoke_rpc(shared::rpc::ClientRequest const &request, std::string &resp)
+RPCAccessor::invoke_rpc(shared::rpc::ClientRequest const &request, std::string &resp)
 {
   std::string encodedRequest = Codec::encode(request);
   resp                       = invoke_rpc(encodedRequest);
@@ -207,6 +211,7 @@ ConfigCommand::config_set()
 
   _printer->write_output(response);
 }
+
 void
 ConfigCommand::config_reload()
 {

src/traffic_ctl/CtrlCommands.h

@@ -26,6 +26,30 @@ limitations under the License.
 #include "shared/rpc/RPCClient.h"
 #include "CtrlPrinters.h"
 
+/// @brief Class that provides access to the RPC side of things.
+struct RPCAccessor {
+protected:
+  /// @brief Invoke the remote server. This is the very basic function which does not play or interact with any codec. Request
+  ///        and message should be already en|de coded.
+  /// @param request A string representation of the json/yaml request.
+  /// @return a string with the json/yaml response.
+  /// @note This function does print the raw string if requested by the "--format". No printer involved, standard output.
+  std::string invoke_rpc(std::string const &request);
+
+  /// @brief Function that calls the rpc server. This function takes a json objects and uses the defined coded to convert them to a
+  ///        string. This function will call invoke_rpc(string) overload.
+  /// @param A Client request.
+  /// @return A server response.
+  shared::rpc::JSONRPCResponse invoke_rpc(shared::rpc::ClientRequest const &request);
+
+  /// @brief Function that calls the rpc server. The response will not be decoded, it will be a raw string.
+  void invoke_rpc(shared::rpc::ClientRequest const &request, std::string &bw);
+
+  std::unique_ptr<BasePrinter> _printer; //!< Specific output formatter. This should be created by the derived class.
+private:
+  shared::rpc::RPCClient _rpcClient; //!< RPC socket client implementation.
+};
+
 // ----------------------------------------------------------------------------------------------------------------------------------
 ///
 /// @brief Base Control Command class.
@@ -49,24 +73,6 @@ class CtrlCommand
   virtual void execute();
 
 protected:
-  /// @brief Invoke the remote server. This is the very basic function which does not play or interact with any codec. Request
-  ///        and message should be already en|de coded.
-  /// @param request A string representation of the json/yaml request.
-  /// @return a string with the json/yaml response.
-  /// @note This function does print the raw string if requested by the "--format". No printer involved, standard output.
-  std::string invoke_rpc(std::string const &request);
-
-  /// @brief Function that calls the rpc server. This function takes a json objects and uses the defined coded to convert them to a
-  ///        string. This function will call invoke_rpc(string) overload.
-  /// @param A Client request.
-  /// @return A server response.
-  shared::rpc::JSONRPCResponse invoke_rpc(shared::rpc::ClientRequest const &request);
-
-  /// @brief Function that calls the rpc server. The response will not be decoded, it will be a raw string.
-  void invoke_rpc(shared::rpc::ClientRequest const &request, std::string &bw);
-
-  std::unique_ptr<BasePrinter> _printer; //!< Specific output formatter. This should be created by the derived class.
-
   /// @brief The whole design is that the command will execute the @c _invoked_func once invoked. This function ptr should be
   ///        set by the appropriated derived class base on the passed parameters. The derived class have the option to override
   ///        the execute() function and do something else. Check @c RecordCommand as an example.
@@ -81,15 +87,14 @@ class CtrlCommand
 
 private:
   ts::Arguments *_arguments = nullptr; //!< parsed traffic_ctl arguments.
-  shared::rpc::RPCClient _rpcClient;   //!< RPC socket client implementation.
 };
 
 // -----------------------------------------------------------------------------------------------------------------------------------
 ///
 /// @brief Record Command Implementation
 ///        Used as base class for any command that needs to access to a TS record.
 ///        If deriving from this class, make sure you implement @c execute_subcommand() and call the _invoked_func yourself.
-class RecordCommand : public CtrlCommand
+class RecordCommand : public CtrlCommand, public RPCAccessor
 {
 public:
   using CtrlCommand::CtrlCommand;
@@ -114,6 +119,8 @@ class ConfigCommand : public RecordCommand
   static inline const std::string DIFF_STR{"diff"};
   static inline const std::string DEFAULTS_STR{"defaults"};
   static inline const std::string SET_STR{"set"};
+  static inline const std::string COLD_STR{"cold"};
+  static inline const std::string APPEND_STR{"append"};
   static inline const std::string STATUS_STR{"status"};
   static inline const std::string RELOAD_STR{"reload"};
   static inline const std::string REGISTRY_STR{"registry"};
@@ -125,6 +132,7 @@ class ConfigCommand : public RecordCommand
   void config_diff();
   void config_status();
   void config_set();
+  void file_config_set();
   void config_reload();
   void config_show_file_registry();
 
@@ -147,7 +155,7 @@ class MetricCommand : public RecordCommand
   MetricCommand(ts::Arguments *args);
 };
 // -----------------------------------------------------------------------------------------------------------------------------------
-class HostCommand : public CtrlCommand
+class HostCommand : public CtrlCommand, public RPCAccessor
 {
 public:
   HostCommand(ts::Arguments *args);
@@ -163,7 +171,7 @@ class HostCommand : public CtrlCommand
   void status_up();
 };
 // -----------------------------------------------------------------------------------------------------------------------------------
-class PluginCommand : public CtrlCommand
+class PluginCommand : public CtrlCommand, public RPCAccessor
 {
 public:
   PluginCommand(ts::Arguments *args);
@@ -173,7 +181,7 @@ class PluginCommand : public CtrlCommand
   void plugin_msg();
 };
 // -----------------------------------------------------------------------------------------------------------------------------------
-class DirectRPCCommand : public CtrlCommand
+class DirectRPCCommand : public CtrlCommand, public RPCAccessor
 {
 public:
   DirectRPCCommand(ts::Arguments *args);
@@ -194,7 +202,7 @@ class DirectRPCCommand : public CtrlCommand
   bool validate_input(std::string const &in) const;
 };
 // -----------------------------------------------------------------------------------------------------------------------------------
-class ServerCommand : public CtrlCommand
+class ServerCommand : public CtrlCommand, public RPCAccessor
 {
 public:
   ServerCommand(ts::Arguments *args);
@@ -208,7 +216,7 @@ class ServerCommand : public CtrlCommand
 };
 //
 // -----------------------------------------------------------------------------------------------------------------------------------
-struct StorageCommand : public CtrlCommand {
+struct StorageCommand : public CtrlCommand, public RPCAccessor {
   StorageCommand(ts::Arguments *args);
 
 private:
@@ -219,3 +227,6 @@ struct StorageCommand : public CtrlCommand {
   void set_storage_offline();
 };
 // -----------------------------------------------------------------------------------------------------------------------------------
+
+BasePrinter::Options::OutputFormat parse_format(ts::Arguments *args);
+BasePrinter::Options parse_print_opts(ts::Arguments *args);