Reference to a hash

[DouweM]

I think there needs to be a way to reference a hash in the same document, like in YAML.

A common practice in Rails is to have the root hash of a YAML config file map environment names to another hash containing the actual configuration vars, which really benefits from referencing when you want the production and staging environments to have the same configuration save for one or two keys.

Proposed format:

[production]
key = "value"
key2 = "value2"

[production.database]
host = "123.123.123"
port = 234
database = "prod"

[staging]
&production
key2 = "othervalue2"

[staging.database]
&production.database
database = "stage"

This is a little different from YAML in that there is no anchor and that we're using the ampersand for reference instead of anchor, but I like it this way because the reference can be read as "and other_hash", which makes sense because we're defining the hash to have values X, Y and Z and those from the other hash.

The syntax is just an ampersand followed by the full key for the hash to reference. You should never reference the root hash, and without anchors you can't.

Just to clear something up: production.database was copied to staging.database, but when we then define staging.database we don't want to override the whole hash, just one key, so we reference production.database again.

[C0mkid]

Already addressed in #13

[DouweM]

@C0mkid Kinda, but not really. That issue tries to do a lot more than I am, and @mojombo's comments don't apply to my proposal.

[C0mkid]

@DouweM "You could always use your favorite templating language to do this already and generate the resulting file." You are probably better off doing something like that, I think this just brings unnecessary things to the spec.

[DouweM]

@C0mkid
His proposal was a lot more complex and I agree with @mojombo that full-on templating is a bit outside the scope for a configuration data format like TOML. I'm not trying to do anything as drastic as that guy was, so I don't think that comment applies here. @mojombo could have the same opinion on my proposal, but I'm going to wait for him to say so himself.

I think you're underestimating how useful what I'm proposing is in the context of config files as used in Rails and other environment-based projects. It's not some weird niche feature that's better left to an extra library, it really is a feature that's at home in a configuration format (see: YAML). I don't want templating like #13, just simple straight-forward references.

I think TOML could replace YAML as the de-facto standard Rails config file format, but that's not going to happen without support for references.

[ocxo]

👍 for references 👎 for templating

[mojombo]

I am intrigued by this proposal for references. I shall noodle upon it. Keep the arguments comin'.

[DouweM]

@mojombo GitHub is mainly a Rails shop, right? How do you see TOML working with your Rails apps without references? You'll pretty much need them if you want config = all_configs[Rails.env] to be practical, with a defaults or common group referenced by all envs, and staging referencing production. Or is TOML not meant to be used with Rails apps? If so, for what reason?

I really like TOML, and prefer it over YAML for configuration. With support for references, my config.yml, database.yml, mongoid.yml and newrelic.yml files would be promptly replaced by their .toml equivalents (with support in their respective libraries, of course).

[rossipedia]

I like this idea, but the proposed format is making me itch. TOML's main strength is it's immediate readability. You can quickly scan it and grok what the config is in one pass (very much like INI files). However, references kinda muddy that readability up. You lose that oh-so-elegant name = value format.

If what you're trying to do is effectively embed the values of one hash into another (with optional overriding), what about something like this:

[production]
key1 = "value1"
key2 = "value2"

[staging:production]
key2 = "overridevalue2"

It does look a bit more like inheritance than strict references, but it can still be implemented in terms of a reference.

[DouweM]

Hmm. I think that at a glance it looks too much like the a hash with a tree key: 'staging.production'. Also, the brackets are used to set the name of the hash, which is still a simple 'staging'; it doesn't really makes sense to have the name of the referenced hash in the same place. The reference should be with the key/value pairs, because that's what it is: a reference to key/value pairs.

—Douwe

On 25 feb. 2013, at 00:05, "Bryan J. Ross" notifications@github.com wrote:

I like this idea, but the proposed format is making me itch. TOML's main strength is it's immediate readability. You can quickly scan it and grok what the config is in one pass (very much like INI files). However, references kinda muddy that readability up. You lose that oh-so-elegant name = value format.

If what you're trying to do is effectively embed the values of one hash into another (with optional overriding), what about something like this:

[production]
key1 = "value1"
key2 = "value2"

[staging:production]
key2 = "overridevalue2"
It does look a bit more like inheritance than strict references, but it can still be implemented in terms of a reference.

—
Reply to this email directly or view it on GitHub.

[DouweM]

I think very important about TOML is its simplicity, also syntax wise: What goes into the hash is written below the hash-header. References are, of course, an example of the former.

[mojombo]

The more I stare at this, the more I like it, but there's one bit that makes my collar itch. The &production reference copies over [production] and [production.database] but when you want to override something in [staging.database] you have to re-initialize it again with &production.database. I'm guessing this would be a constant source of errors as people forget to do that, and I'm not sure what to do about it. Any ideas?

[DouweM]

Yeah, that's exactly why I included that in the example. The same problem exists in YAML, and I guess people have learned through trial by error, but that's clearly the worst-case scenario and it would be great if we were to come up with a more obvious solution for TOML, seeing it's in the name.

We could have keygroups like [staging.database] automatically inherit [production.database], even if the keygroup is explicitly 'opened'. This would cause issues, however, in the case of optional keys, where [staging.database] would not include them on purpose and where they'd be set through [production.database] anyway. If there are no optional keys, it would just work, because even though [production.database]'s values are copied over, we're overriding all of them in [staging.database].

While not as obvious as we'd like, I don't think having to explicitly reference the nested keygroup again is that much of a problem. What we're doing is merging hashes; when doing so programmatically, everybody knows (or quickly finds out) that unless you explicitly say so, nested hashes won't be merged recursively:

{ a: "b", c: { a: "b", c: "d" } }.merge({ c: { c: "e" } }) === { a: "b", c: { c: "e" } }

[goto-bus-stop]

Would this work?

[somedata]
&mixin
foo = "bar"

[mixin]
foo = "foo"
bar = "bar"

ie. do references before definitions count?

[DouweM]

That would complicate implementation because non-referencing keygroups would have to be parsed first, instead of being able to go through the file from top to bottom. What do you think, @mojombo ?

[mojombo]

@renekooi If we do references, I'd say the referenced keygroup would have to appear before the reference can be made. I don't think that's an unreasonable requirement, and makes the parser's life 100x easier.

[ocxo]

It also makes the document more human readable which is a nice side effect of TOML (if not a stated goal).

On Feb 27, 2013, at 11:21 AM, Tom Preston-Werner notifications@github.com wrote:

@renekooi If we do references, I'd say the referenced keygroup would have to appear before the reference can be made. I don't think that's an unreasonable requirement, and makes the parser's life 100x easier.

—
Reply to this email directly or view it on GitHub.

[DouweM]

Yeah, I agree re: referencing keygroups defined later.

@mojombo What's your current view on this proposal? Is having to reference &production.database again under the circumstances described above still a problem in your eyes?

[rcarver]

Though commonly used, I think this is one of the features that makes YAML confusing.

levels is another take on the defaults/overrides idea. It also supports computed values (value references) in its Ruby syntax, which I could see supporting on top of TOML as well - say, via ERB or Mustache or something.

Just throwing that out there if you'd prefer to keep TOML simple.

[mojombo]

I talked about this with @tnm IRL and we came to the conclusion that references add a layer of confusion and actually increase the odds of unexpected errors sneaking into config files. TOML is about obviousness and minimalism, neither of which are accomplished when references come into play. We looked through a variety of GitHub's rails apps' YAML config files and we rarely use YAML references. We much prefer duplicating config sections, even if that means a lot of duplication. This approach makes config files more greppable, easier to read by a human, and eliminates the risk of edits to one section unwittingly cascading to other sections.

[DouweM]

All right, I can see where you're coming from. Thanks for taking this into consideration anyway!

[prasannavl]

@mojombo, I see that one of prime reasons you ended up discarding this is with the analysis of your current real use cases.

But limitation with that analysis is that you used yaml, which by its very nature doesn't allow references to a single hash (It allows only to a node, which is very different and is barely useful unless you're writing luge lists of data - only suited for a particular taxonomy)

I find myself not using node references much as well, but direct variable references? I'd think that I'd be using it everywhere, and as a side effect it actually makes things more readable because I don't have to keep sticking long paths, or sentences over and over again, making it concise.

References get confusing when it has too much syntax. But with the idea proposed above, its one addition and remains the same. This enables for scenarios like these:

• http://stackoverflow.com/questions/2063616/how-to-reference-a-yaml-setting-from-elsewhere-in-the-same-yaml-file
• http://stackoverflow.com/questions/4150782/using-yaml-with-variables

The reason one can't compare that with current/past use cases is because almost none of the markup languages have good references (yaml complicates life, and even so doesn't provide this exact specific requirement). So, if one provided it, I'd think it'd be used everywhere. I'd suggest that this to be reconsidered - although more like how its done in the above stackoverflow questions.

rootDir = "C:\Workspace"
project1Dir = &rootDir + "\Project1"
project2Dir = &rootDir + "\Project2"

With basic and natural operators.. or perhaps without operators and letting the double quoted strings to handle the reference operator.

Note: This above complicates the parser a little bit - yes, but not the actual markup language.

[AndreiPashkin]

@mojombo, I'd like to express the opposite point of view.

If look at existing config-parsing libraries for different languages, we would see, that they all implement the same features:

1. Substitution (a.k.a. referencing).
2. Default values.
3. Validation.

Examples:

• ini4j - a Java library.
  Implements substitution.
• config - a Java library.
  Implements substitution.
• configparser - from Python's standard library.
  Implements substitution and default values.
• ConfigObj - a Python library.
  Implements substitution, default values and validation.
• go-ini - a Go library.
  Implements substitution and validation.

It shows, that this functionality is essential, otherwise people wouldn't create all those libraries and just use YAML or JSON.

On the other hand, all these libraries use non-standardized configuration language, and they all usually not well thought-out, not cross-platform and implement only part of popular use cases.

This is why I personally started researching for an alternative and found TOML.