You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In our WYWSIWYG solution, we want to manage the native configs the similar way as managing KRM resources. This requires a mechanism to consume native configs and to represent these native configs as KRM objects.
How Augeas can be used in the Kpt solution
Augeas can "parse native config and transform them into a tree". It provides the API that allows users to read and write these native configurations. #3210 suggests considering Augeas.
From the POC experimental code, Augeas can be integrated in Kpt as shown below. Everything in the green block does not exist in current KPT yet. It is only trying to give a high level idea and only show the native config "read" scenario: Convert native file to a structured KRM objects. The "write" scenario is basically in a reverse order.
Besides the above POC demo, I also explore the following approaches to integrate Augeas with Kpt in different ways, and encountered non-trivial problems in all of them.
Different Augeas released tools
Augeas as a Go library : Import the Augeas as a Go library in the native-config-adaptor function directly.
Augeas as CLI: Build the Augeas CLI "augtool" into the native-config-adaptor function image.
Augeas supports different lenses to parse INI file. I explored two: generic IniFile lens and Puppet lens.
Generic INI Lens: The parsed output uses a "target[N]" as placeholder to distinguish filepath and section.
- The Augeas lens name is IniFile.lns, the config is here.
- In the MariaDB my.cnf example, the generic INI lens parsed output is here.
- Here's my native-config-adaptor code to convert the generic INI parsed result to a KRM object.
A Puppet INI Lens: The parsed results has no placeholder, it distinguishes section via empty value.
- The Augeas lens name is Puppet.lns, the config is here.
- In the MariaDB my.cnf example, the puppet lens parsed output is here.
- Here's my native-config-adaptor code to convert the puppet INI parsed result to a KRM object.
During the comparison, the generic INI File lens does not allow me to specify a custom file path. It can only load files from the /etc/* field, excluding sub directories. The Puppet lens supports loading native config from a custom file path.
Problems and Limitations
Here's a quick summary of the problems.
(Augeas) Inactive Maintenance
This concern has been raised by @bgrant0607 in the early exploration stage. Here's some more findings.
If we choose Augeas, it's necessary for us to learn Augeas lens config and even contribute to Augeas if key features are missing. This can be a challenge considering the different knowledge set.
(Augeas) Insufficient docker support
The Augeas Docker is released 3 years ago. It cannot build against the latest Augeas release (v1.13.0), filed #769. If we do want to use Augeas in kpt, that means we need to maintain our own Augeas version that matches the release until the docker build is fixed.
This insufficient Docker support could be a signal that potential issues in the container runtime environment may not have been fully explored or exposed.
(Kpt) Limitations on Augeas formats
Augeas does not have a generic syntax for the parsed output. This is different than we originally thought that Augeas handles the most Native files to tree structure job, and kpt converts the tree to a KRM object. But rather, Kpt needs to provide individual conversion to each file formats.
More than that, Augeas lists the supported native file formats in http://augeas.net/stock_lenses.html, many are not auto loaded, each has its own lens configurations. The same INI file, with different parsing option (generic INI lens vs puppet INI lens), can have different syntax and behaviors, even on the same file. This means Kpt needs to have a rich knowledge about its supported Augeas lenses and their behavior differences. This is a huge overhead.
(Kpt users) challenges to adopt
After we provide the basic native config converters to KRM objects, we can expect users to adopt and enrich the kpt supported native file formats. Developing a Augeas custom lens and have it work in Kpt can be a much challenging task than developing a Golang parser.
Conclusions
Augeas is not a good choice for Kpt in the current stage from the perspectives of development and maintenance cost, knowledge gaps, and user inclusiveness and adoptions.
We plan to choose a set of most common native config file format, and develop our own KRM and native file parsers.
We can re-iterate the Augeas approach if the above problems or conditions change.
This issue is under #3351
Background
In our WYWSIWYG solution, we want to manage the native configs the similar way as managing KRM resources. This requires a mechanism to consume native configs and to represent these native configs as KRM objects.
How Augeas can be used in the Kpt solution
Augeas can "parse native config and transform them into a tree". It provides the API that allows users to read and write these native configurations. #3210 suggests considering Augeas.
From the POC experimental code, Augeas can be integrated in Kpt as shown below. Everything in the green block does not exist in current KPT yet. It is only trying to give a high level idea and only show the native config "read" scenario: Convert native file to a structured KRM objects. The "write" scenario is basically in a reverse order.
![Screen Shot 2022-08-15 at 12 43 22](https://user-images.githubusercontent.com/5606098/184706003-e71a3ed1-51c2-484c-b5e7-27e4e178405f.png)
POC Experimental Source Code
Explorations
Besides the above POC demo, I also explore the following approaches to integrate Augeas with Kpt in different ways, and encountered non-trivial problems in all of them.
Different Augeas released tools
Different "INI" parsing format (Augeas "lenses")
Augeas supports different lenses to parse INI file. I explored two: generic IniFile lens and Puppet lens.
Generic INI Lens: The parsed output uses a "target[N]" as placeholder to distinguish filepath and section.
- The Augeas lens name is
IniFile.lns
, the config is here.- In the MariaDB
my.cnf
example, the generic INI lens parsed output is here.- Here's my native-config-adaptor code to convert the generic INI parsed result to a KRM object.
A Puppet INI Lens: The parsed results has no placeholder, it distinguishes
section
via empty value.- The Augeas lens name is
Puppet.lns
, the config is here.- In the MariaDB
my.cnf
example, the puppet lens parsed output is here.- Here's my native-config-adaptor code to convert the puppet INI parsed result to a KRM object.
During the comparison, the generic INI File lens does not allow me to specify a custom file path. It can only load files from the /etc/* field, excluding sub directories. The Puppet lens supports loading native config from a custom file path.
Problems and Limitations
Here's a quick summary of the problems.
(Augeas) Inactive Maintenance
This concern has been raised by @bgrant0607 in the early exploration stage. Here's some more findings.
The Augeas releases new version in a yearly basis (https://github.com/hercules-team/augeas/releases) and it seems to be a less active project that cannot fulfill Kpt's fast growing needs in the future.
I spot checked the open issues and unreleased fixes, here's a short list of problems still open in latest release:
If we choose Augeas, it's necessary for us to learn Augeas lens config and even contribute to Augeas if key features are missing. This can be a challenge considering the different knowledge set.
(Augeas) Insufficient docker support
The Augeas Docker is released 3 years ago. It cannot build against the latest Augeas release (v1.13.0), filed #769. If we do want to use Augeas in kpt, that means we need to maintain our own Augeas version that matches the release until the docker build is fixed.
This insufficient Docker support could be a signal that potential issues in the container runtime environment may not have been fully explored or exposed.
(Kpt) Limitations on Augeas formats
Augeas does not have a generic syntax for the parsed output. This is different than we originally thought that Augeas handles the most Native files to tree structure job, and kpt converts the tree to a KRM object. But rather, Kpt needs to provide individual conversion to each file formats.
More than that, Augeas lists the supported native file formats in http://augeas.net/stock_lenses.html, many are not auto loaded, each has its own lens configurations. The same INI file, with different parsing option (generic INI lens vs puppet INI lens), can have different syntax and behaviors, even on the same file. This means Kpt needs to have a rich knowledge about its supported Augeas lenses and their behavior differences. This is a huge overhead.
(Kpt users) challenges to adopt
After we provide the basic native config converters to KRM objects, we can expect users to adopt and enrich the kpt supported native file formats. Developing a Augeas custom lens and have it work in Kpt can be a much challenging task than developing a Golang parser.
Conclusions
Augeas is not a good choice for Kpt in the current stage from the perspectives of development and maintenance cost, knowledge gaps, and user inclusiveness and adoptions.
We plan to choose a set of most common native config file format, and develop our own KRM and native file parsers.
We can re-iterate the Augeas approach if the above problems or conditions change.
@bgrant0607 @justinsb @johnbelamaric @selfmanagingresource
The text was updated successfully, but these errors were encountered: