Datetime literals are not appropriate for a minimal language

[thezerobit]

If this is to be taken at face value: "TOML aims to be a minimal configuration file format", then embedding all of RFC 3339 to get datetime literals makes no sense. Since the string values of TOML keys are restricted, I don't see TOML as being appropriate as a general data serialization protocol (in which a native datetime literal would be useful), so I assume it really is meant to be for human-edited config files. I've written/edited plenty of configuration files, and I can't remember when I ever needed to encode a datetime by hand.

[thezerobit]

What is more likely in a configuration file, an octal file permission (eg. 0644) or a datetime literal? There's no support for octal numbers in this spec, which would be trivially easy to implement, but there is support for datetime literals which are nontrivial to parse.

[ChristianSi]

TOML is not just meant for config files, it's useful for all kinds of structured data. A few use cases where datetime literals might be useful and TOML might be the right kind of format for storing the managed data:

• Calender app: start and end time of events
• Address manager: birthdays and other anniversaries
• Version or dependency manager: date when a version was released or installed

[cies]

I'd like to see the time removed as well. It is not needed for minimal, if you really need to do time--stuff you probably need something more powerful (timezones, ranges, repetition, etc.), which can then be put in strings which you parse how-ever you wish.

TL;DR RFC3339 is not in line with minimal.

[pnathan]

Disagree - having a standards conformant time selected allows a particular
time standard, as opposed to each app rolling their own.

[cies]

@pnathan indeed verrrry particular; as time usually needs timezones, ranges, etc.

opposed to each app rolling their own.

"each" app would be more like one-in-a-100 (or 1000) apps that almost certainly dont need interop.

[oliverjanik]

I vote to keep the datetime in. JSON doesn't have a datetime type and it's a sore every day.

[cies]

Do you use json for config files when you miss datetime?
On Jan 11, 2015 7:41 AM, "Oliver Janik" notifications@github.com wrote:

I vote to keep the datetime in. JSON doesn't have a datetime type and it's
a sore every day.

—
Reply to this email directly or view it on GitHub
#263 (comment).

[cies]

That is not easily a hieved when writing a parser.. Hence my point: dates
are not very minimal in a conf lang.

And when one needs dates in a conf lang, chances are big you need more
(durations, tzones, repetitions), or less (only year+month+day) then
currently in the toml spec.
On Jan 13, 2015 8:51 PM, "Barry Allard" notifications@github.com wrote:

It might be simpler to borrow from a standard like ISO 8601 only in the
UTC format of yyyy-MM-ddThh:mm:ss[.sssssss]Z for datetime. If a date type
were essential, exclude the T part.

—
Reply to this email directly or view it on GitHub
#263 (comment).

[mojombo]

There are some fair points raised here. When I originally wrote the spec, I included Date because it's a common language type and it didn't seem like it would add too much complexity. But everything added takes away from everything else. I do see TOML as a configuration format (not a serialization format), and I've also rarely needed to specify a date in a config file. They are indeed a bit pesky to parse (though not too bad, really), and tend to produce arguments around them. Honestly, I could be convinced to drop Date support, but I don't think I'm quite there yet.

@BurntSushi what are your thoughts?

[cies]

Thanks for having a look at this. For me the parsing is certainly not the
biggest problem. Its more the utility: they only serve a very narrow
purpose (non tzone, non duration, fixed precision). I cannot imagine i will
ever need exactly this datetime type in a config file; therefor imho not
part of a "minimal" config file format spec.
On Jan 15, 2015 11:31 PM, "Tom Preston-Werner" notifications@github.com
wrote:

There are some fair points raised here. When I originally wrote the spec,
I included Date because it's a common language type and it didn't seem like
it would add too much complexity. But everything added takes away from
everything else. I do see TOML as a configuration format (not a
serialization format), and I've also rarely needed to specify a date in a
config file. They are indeed a bit pesky to parse (though not too bad,
really), and tend to produce arguments around them. Honestly, I could be
convinced to drop Date support, but I don't think I'm quite there yet.

@BurntSushi https://github.com/BurntSushi what are your thoughts?

—
Reply to this email directly or view it on GitHub
#263 (comment).

[mojombo]

I asked on Twitter if anyone was using datetimes in config files, and the most useful response I got was this from @keithamus:

@mojombo They’re useful for fixture files!

Which is incredibly true. Rails fixtures often specify database fields. It would be nice for TOML to support common database types for such use cases. Which, I suppose is more of a serialization task, but still a common use case for things like YAML or JSON which TOML aims to supplant in many cases.

[oliverjanik]

There are only a few datetime formats that are useful IMO:

YYYY-MM-dd for just dates
YYYY-MM-ddThh:mm:ss.ffff for local datetimes
YYYY-MM-ddThh:mm:ss.ffff+zz for exact timestampts (zz is timezone offset)

There is not need to implement entire RFC3339. This would cover 99% of the cases from my experience.

[cies]

@mojombo: fixture files are indeed a good case, but they are certainly not
config files, and most likely better served by a more featureful language
like yaml or ruby itself (i use factorygirl since many years)

So no obvious use for datetimes in config files thus far. ;)
On Jan 16, 2015 8:41 PM, "Tom Preston-Werner" notifications@github.com
wrote:

I asked on Twitter if anyone was using datetimes in config files, and the
most useful response I got was this from @keithamus
https://github.com/Keithamus:

@mojombo https://github.com/mojombo They’re useful for fixture files!

Which is incredibly true. Rails fixtures often specify database fields. It
would be nice for TOML to support common database types for such use cases.
Which, I suppose is more of a serialization task, but still a common use
case for things like YAML or JSON which TOML aims to supplant in many cases.

—
Reply to this email directly or view it on GitHub
#263 (comment).

[groves]

We use datetimes in our feature flag config files e.g. feature x should be available for all users at this time. It's nice for tying product launches in with blog posts and other bits of marketing without having to remember to have someone there to throw the switch.

[ChristianSi]

I think there have been several examples for why datetimes can be useful, hence they should stay in.

I also think there would be some use cases for YYYY-MM-dd for just dates -- dates in version histories, birthdays etc. The additional overhead for implementors would be minimal.

I'm opposed to timezone-less dates since they would point to different points in time if read on different computers, or even on the same computer at different times (due to daylight saving time). I also think that +zz for encoding specific timezones would be an unnecessary overhead -- just convert to UTC instead.

I don't think that durations would be an important addition, since starttime + duration can always be easily replaced by starttime + endtime. There may be a few use cases for duration without startime (e.g. in recipes), but I think they are few enough to not warrant a new basic data type.

[cies]

I think there have been several examples for why datetimes can be useful

I have not seen a clear, broad usecase for datetimes in a config file. And since TOML is a format for config files that tries to be minimal.

Just an idea of the number of datatimes that we need to support in order to be at least facilitate for some (mostly non-config usecases):

YYYY-MM-dd for just dates
YYYY-MM-ddThh:mm:ss for local datetimes w/o msecs
YYYY-MM-ddThh:mm:ss.ffff for local datetimes
YYYY-MM-ddThh:mm:ss+zz for exact timestamps w/o msecs (zz is timezone offset)
YYYY-MM-ddThh:mm:ss.ffff+zz for exact timestamps (zz is timezone offset)

I added the "w/o msecs" to @oliverjanik's list. But still this is not "enough", it leaves a lot of corner cases: for instance dates also have timezones. And what about a time without a date? A proper spec for time/date/datetime, like RFC3339, is probably the size of the whole TOML spec.

We also should see what the pain is of not having a datetime literal in TOML... Well in that case you can simply put them in a string and specify how you parse them: any of the case above is easily facilitate in that way (you probably only need one of the list for your case).

But then: config files do almost never need datetimes, so you are probably using TOML beyond its intended purpose.

[BurntSushi]

I think there are definitely some valid use cases for dates/times in a config file, although they are not that common. For example, Hugo (a popular project using TOML for configuration to statically generate web sites/blogs) uses dates and times for publish dates. The interesting bit about this use case is that Hugo doesn't actually use TOML's datetime type! (Or, at least, doesn't require it.) I suspect this is because our type is inflexible: you have to specify the full date and time in UTC, although @spf13 may be able to elaborate more. Having to write a datetime in something other than your current timezone is an utter pain, and so is having to write the time when all you need is a date.

I also looked at some other projects using TOML like Heka and InfluxDB. Neither of those use TOML's datetimes either. And I'm pretty sure Cargo doesn't either.

With that said, I think we have two choices:

1. Extend the TOML datetime type to be more flexible so that people actually use it in the few use cases for datetimes.
2. Remove the datetime type and encourage users to just use strings and rely on existing tools in your ecosystem to parse that date. We can recommend that they follow some flavor of RFC3339.

At this point, given the small number of uses of TOML's datetime and the extra complexity required to actually make TOML's datetime type useful, I'm in favor of removing it altogether.

[cies]

@BurntSushi, I had a look at Hugo. It accepts TOML for as config file, and there are no dates/times/datetimes used there. It also accepts TOML for metadata on a blog item, but metadata is not config, it is a different usecase and TOML might not be the best options there (for limited cases it is probably quite good, but for heavier use something else is needed, maybe RDF/RDFS with turtle or n3).

So when TOML is used for metadata then dates/times/datetimes become useful, but as you mention, not their current shape. The much more extended shape that is needed for the use of metadata lies completely out of TOMLs intended domain of config files.

So TOML, are you truly minimal? Or are you also going to feature hog like the formats you try to replace? :)

[BurntSushi]

and there are no dates/times/datetimes used there. It also accepts TOML for metadata on a blog item, but metadata is not config, it is a different usecase and TOML might not be the best options there (for limited cases it is probably quite good, but for heavier use something else is needed, maybe RDF/RDFS with turtle or n3).

This seems like a distraction and a pretty pointless distinction. Whether you call it meta data or config, TOML is being used to simply specify the functionality of an application. That seems perfectly within the use case of TOML as a configuration format.

I'm not convinced that this argumentation to rule out all use cases of date times in a config format is fruitful. The data supports the conclusion that datetimes are rarely used in the wild (given the projects we've checked that use TOML). And even when they are used, TOML's datetime type isn't used. That is the line of reasoning that I used to suggest that the type be removed. (Rare + ineffective.)

[ChristianSi]

@BurntSushi I'd be in favor of checking the date/time formats actually accepted by Hugo (and maybe other projects that use TOML and need to store dates) and then decide whether it's reasonable to modify TOML's datetime type accordingly.

If it turns out that their date/time formats would significantly complicate parsing, I'm in favor dropping the datetime type altogether, otherwise I'd plead to leave it in and broaden it accordingly so that they could start to use it.

[BurntSushi]

We have the opportunity to remove something from TOML with minimal impact. IMO, we should seize that opportunity.

As a distant second choice, I agree that having a more effective datetime type that people actually use would be better than what we have today. But we could remove the datetime type and always add it back later if there ends up being real demand for one.

[mojombo]

Alright, I've re-read this thread, and I think it makes sense to remove Datetimes from TOML. At least for 1.0. Once real-world usage of TOML picks up, and if strong use cases arise for some Date/Time representations, we'll be much better equipped to deal with them at that point. I will be removing them from the spec shortly. Thanks for the feedback, everyone!

[spf13]

Oh, please don't do that. We benefit from them in Hugo quite a bit. Let me make a case for keeping them in, at least in some form. Sorry I've not responded before now. I'll make the case shortly.

[mojombo]

@spf13 Ok, I hope it's compelling! =)

[spf13]

So perhaps it's not the strongest case, but here goes.

A bit of background. Hugo is similar to Jekyll in which the top of a markup file is the meta data (frontmatter).

Unlike Jekyll, Hugo supports yaml, toml and json for the front matter. Hugo also uses a similar approach to data handling and configuration files (through Viper, a sister project).

In Hugo there are a few predefined fields like date & pubdate which we treat like dates regardless of how they are defined in the front matter. This works well with all formats as Json and Yaml don't have a date type. In Hugo the code that handles this conversion happens independent of the format the front matter is defined in... and it's true that we don't require the TOML date type here. See https://github.com/spf13/hugo/blob/master/hugolib/page.go#L469

However in Hugo you can define any arbitrary field in the front matter which can be then used in the site. Because of the static nature of types in Go, the only easy way to treat a value as a date is at the time it's being converted from the yaml/json/toml format. For example, It's not possible to call date format on a string value in the template. Since json and yaml don't have a native date format, it's not possible to define a date in either of those. The only way to add a date field to your front matter is to use TOML and make it a date type there. See https://github.com/spf13/hugo/blob/master/hugolib/page.go#L509

We've recently added support for data import functionality and the same idea applies there.

Adding a date type to TOML was a really beneficial thing. It enables Hugo users to do something that they couldn't do otherwise (define arbitrary date fields). I know a bunch of people that specifically use TOML because of this feature.

I think that a strong case could be made for making the date type a bit less strict. I myself have been tripped up on that a number of times, but to remove the type altogether seems like a step backwards from a useful progressive feature.

Does that make sense?

[mojombo]

@spf13 Cool, thanks for those thoughts. I do believe that some kind of date support in TOML would be beneficial, just not in its current format. I spent some time looking at how others deal with dates and came up with a few links:

• http://stackoverflow.com/questions/10286204/the-right-json-date-format
• http://yaml.org/type/timestamp.html

There are definitely use cases for dates in config files, and it would be nice for TOML to work really well for Jekyll, which has an almost identical situation. =) So perhaps the question becomes: can we cover 80% the date/time use cases with a simple and obvious mechanism? I think we could by allowing only three possibilities:

• Date (implicit 00:00:00 time and machine-local offset).
• Date + Time (implicit machine-local offset).
• Date + Time + Time Zone (RFC 3339).

For example:

• 2015-02-06
• 2015-02-06T11:55:12
• 2015-02-06T11:55:12-08:00

If we want to keep date support I think we have to do something like this, but it also creates more complexity in a project that aims for minimalism.

[BurntSushi]

If we have a real live use case in the wild, then I'm in favor of improving the datetime type and keeping it. (I was under the impression that Hugo wasn't really using it at all.) Allowing abbreviated datetimes certainly seems like a reasonable solution to me.

[spf13]

I like the proposal and can confirm that it is being used by Hugo's users who depend on it today.

[mojombo]

Cool, I'll draw up a PR and see where it goes from there.

[mojombo]

See #297 for a PR that covers the current discussion.

[eksortso]

Although I fear blowing the date/time format thing open again, would it be
worth discussing ISO 8601? It includes the three best formats mentioned
before, plus almost all of the stricter formats specified by RFC 3339. This
would pick up most practical use cases. But I don't know how extensive the
support for it is across languages.

-- Dave Ostroske
On Feb 6, 2015 4:22 PM, "Tom Preston-Werner" notifications@github.com
wrote:

Cool, I'll draw up a PR and see where it goes from there.

—
Reply to this email directly or view it on GitHub
#263 (comment).

[mojombo]

@eksortso That's a fair point, but ISO 8601 has an almost endless number of options, and if we're going to keep datetime support, I think it's important to keep the options to the absolute minimum.

[tunnckoCore]

'key, I've read this thread again and again, and can't realize what's the problem to just parse whatever datetime format strings?

[cies]

what's the problem to just parse whatever datetime format strings?

It's not a problem by itself, but TOML was intended as "minimal" (a parser of ISO 8601 is probably larger then the rest of the TOML spec).

Then TOML is intended for config files, that do not usually need timestamps/dates/times/etc. As you can read in this thread, the mentioned usecases for timestamps/dates/times/etc are not config files (but markup, meta data, log files). In the rare case a config file needs such a type, it is likely not the date+time type that TOML currently supports. Obvious solution: anyone who needs timestamps/dates/times/etc can use the string type for that. And TOML can stay minimal.

[tunnckoCore]

Yep. Im agree with you. And I dont see reason to have date type at all.

[eddelbuettel]

Please do not remove Date(time) formats. It is one of the key distinguishing features.

Our config files do need date objects as configs can be valid before/after a point in time, or have parameter values for given dates or date ranges.

[tunnckoCore]

@eddelbuettel whats the problem to be strings?

Please, give actual use cases next pretendends

[eddelbuettel]

@tunnckoCore:

• Strings are a problem for the same reason we prefer strongly typed int64_t, double, bool in the rest of the TOML spec: Typing adds value. If I wanted to mess around with strings and the "can only parse later but not when reading the file" approach I could stick with JSON or a dozen other (inferior) config formats. That TOML can parse (and hence validate) datetime objects is a core strength of TOML. Removing datetimes would greatly reduce the usefulness of TOML.
• I already did 20+ minutes ago. So let me repeat more explicitly: We have (large, complex) parameter estimation configurations which change through time and have before/after/during date(time) boundaries. Certain setups are valid, then change into other setups. I turned to TOML (and currently work on R support for it) because date(times) are native for TOML.

[oliverjanik]

Agree with date/time support being a core strength. What happens in JSON world, where datetimes are strings, everyone invents their own notation. And even if they are ISO8601 complaint that still leaves you with about million representations.

A nuance a lot of implementations miss is a distinction between an instant (exact unique point in time globally) and local time which doesn't carry timezone information e.g. to schedule something at 6:00pm no matter what timezone the deployment is in. The crucial part here is that the exact moment is interpreted at the very end of the system, not the config parser. Example would be a distributed build server with central TOML config and agents in different timezones that need to schedule builds after work hours.

[cies]

@eddelbuettel said: "It is one of the key distinguishing features."

I think TOML should not have distinguishing features. The basic motivation for TOML was: minimal and directly convertible into JSON. Dates and/or times are neither and a slippery slope. Now we're going to adopt a subset of a ISO8601, quite a beast by itself, meaning I cannot even use a ISO8601 parser!

Maybe a .mtoml will follow: a strict subset of TOML. Easy to write a parser for, easy to remember the syntax. :)

In the meantime we have a fully 0.4.0 compliant --as per the @BurntSushi tests (yay!)-- parser in Haskell, that makes painfully clear how this unJSONic notion of implicit tables complicates the parser further then necessary. Sorry for hijacking the thread a little in this last, rather rantish, paragraph.

[eddelbuettel]

Claim by @cies: " The basic motivation for TOML was: minimal and directly convertible into JSON. "

Nope. From the README: "TOML is designed to map unambiguously to a hash table." And nowhere does said README state that conversion to JSON was a goal. Older versions had much more colourful language reserved for JSON...

[Luthaf]

I am also in favour of removing Datetime type from TOML, and leaving it to the application to parse whatever date format suite there need.

Cons of Datetime:

• Having Datetime type introduce a lot of complexity in the spec (see the discussion about the size of RFC/ISO norms)
• Datetime handling is not a well defined problem. Should literal be allowed (Suggest noon for raw day datetimes. #361)? Should Date, Time, Datetime and Duration be separated types (Treat Dates and Datetimes as separate types #362)?
• Not really minimal. Why Datetime and not URL? And what about IP addresses?

Pro of Datetime:

• We get normalization of the format of datetime. I do not think this is important for a configuration format, as configurations files should not be shared between application. (Serialization formats is another story)
• Anything else I missed?

I can understand that datetime can be convenient sometimes (I read about database fixtures and blog posts), but I think that most of the time, Datetimes should be left to the application and not to the configuration parser.

[BurntSushi]

@Luthaf There are real users of TOML that find a datetime type helpful. I doubt it's getting removed at this point.

[Luthaf]

Yes, lot of types could be helpful for TOML. I am just defending the point that I do not see strong arguments (except from compatibility) to keep Datetime in the spec. I will still use TOML if there is Datetime in it, but I think it would be better without.

If the problem is sorted, maybe this issue should be closed to indicate that a decision has been made by the maintainers :)

[mojombo]

Even less minimal: #414.

[mojombo]

Alas, it turns out Dates and Times are not too minimal, at least for this guy (#414 has been merged). Thanks to everyone for a lively discussion!