Clean slate with Winnow 2 #86

dhilst · 2024-08-29T22:09:25Z

Continuing the rewrite started by @tertsdiepraam #61

The functions in lib.rs keep the same signature
I kept the tests in lib.rs too all they are passing
90% of the parsing was written by @tertsdiepraam (check Clean slate with Winnow #61)
Added timestamp with @\d+ format
Added timezone abbreviation names
Added 8 digits dates MMDDhhmm
Using winnow

dhilst · 2024-08-29T22:09:55Z

@tertsdiepraam I squashed your commits into a single commit, I hope you don't mind

tertsdiepraam · 2024-08-30T08:25:24Z

Looks cool! A few initial comments to send you in the right direction. One of the reasons for this rewrite is that chrono wasn't cutting it for adding all the items together. Instead we should add all the items together ourselves to really match GNU. It'll require a lot of experimentation, but it'll make the compatibility much better. I think most of the corner cases were around the ends of months. I believe GNU had some overflow behaviour there that chrono didn't. So I recommend first creating a custom datetime struct to manipulate and then mapping that to chrono.

Squashing the commit is absolutely fine by the way. Thanks for continuing this!

tertsdiepraam · 2024-09-11T20:34:34Z

Hi! I was looking at this again and I found an issue with how you're currently using chrono. Here's my theory:

It's currently 2024-09-11 and September is a month with 30 days.
I give as input: 2024-01-31
While setting the Date item, you first change the day, then the month and then the year.
At each of those, the validity is checked.
That means that you adjust the date first to 2024-09-31, then 2024-01-31.
However, 2024-09-31 is an invalid date and chrono will therefore yield and error even though I have entered a valid date.

We need to find a way to construct the times such that this never happens. Just changing the order won't cut it because that can still lead to invalid states. chrono might be more of a hindrance than a help here, unfortunately.

dhilst · 2024-09-14T00:08:56Z

Hi! I was looking at this again and I found an issue with how you're currently using chrono. Here's my theory:
1. It's currently 2024-09-11 and September is a month with 30 days.

2. I give as input: `2024-01-31`

3. While setting the `Date` item, you first change the day, then the month and then the year.

4. At each of those, the validity is checked.

5. That means that you adjust the date first to `2024-09-31`, then `2024-01-31`.

6. However, `2024-09-31` is an invalid date and `chrono` will therefore yield and error even though I have entered a valid date.
We need to find a way to construct the times such that this never happens. Just changing the order won't cut it because that can still lead to invalid states. chrono might be more of a hindrance than a help here, unfortunately.

Hi Terts,

I introduced a function to create dates from naive dates instead of using with_ method chains.

About the month behavior, if I understood correctly GNU gets the number of the days in the current month and adds it to the date, I implemented this behavior added your test and one more.

dhilst · 2024-09-14T02:46:57Z

@tertsdiepraam I was looking into the fuzzy testing. I added the code so it generates fuzzy dates with some sanity and caught some problems, I will look into them next.

dhilst · 2024-09-14T03:38:49Z

fuzz/fuzz_targets/parse_datetime.rs

+ // Testing GNU compatibility by calling date and
+ // comparing the inputs
+
+ // let fmt = "%Y-%m-%d %H:%M:%S";
+ // let mut output: process::Output = std::process::Command::new("date")
+ // .arg("-d")
+ // .arg(s)
+ // .arg(format!("+{fmt}"))
+ // .output()
+ // .expect("date command failed");
+ // io::stdout().write_all(&output.stdout).unwrap();
+ // io::stdout().write_all(&output.stderr).unwrap();
+ // output.stdout.pop(); // remove trailing \n
+ // assert_eq!(
+ // String::from_utf8(output.stdout).unwrap(),
+ // d.format(fmt).to_string()
+ // );


I was testing something like this to determine its compatibility with GNU. The test run GNU date in and compare the outputs

We could definitely do that with fuzzing! We should still rely on regular tests as well though.

dhilst · 2024-09-14T21:19:10Z

src/lib.rs

+ #[test]
+ fn chrono_date() {
+ const FMT: &str = "%Y-%m-%d %H:%M:%S";
+ let input = "262144-03-10 00:00:00";
+
+ // chrono rejects this specific date
+ assert!(NaiveDate::from_ymd_opt(262144, 3, 10).is_none());
+ assert!(chrono::DateTime::parse_from_str(input, FMT).is_err());
+ // the parsing works, but hydration fails
+ assert!(items::parse(&mut input.to_string().as_str()).is_ok());
+ assert!(parse_datetime(input).is_err());
+ // GNU date works
+ assert_eq!(make_gnu_date(input, FMT), input);
+ }


@tertsdiepraam I found a case where GNU accepts the date, we can parse it and chrono rejects it, more precisely chrono::NaiveDate::from_ymd_opt returns None for this date. I don't know what direction to take from there, chrono keeps getting in the way 😅 ideas? Maybe having a gnu-compatible datetime layer, but that will be a lot of work

Another problem is with formatting too. chrono puts a + in front of the year if the year is >9999. This breaks some tests:

sylvestre · 2024-09-17T07:39:39Z

sorry but some tests are failing:

failures:
    items::date::tests::test_year
    items::date::tests::with_year
    items::tests::date_and_time
    tests::offsets::invalid_offset_format
    tests::test_relative::chrono_date
    tests::test_relative::gnu_compat
    tests::test_relative::test_month

and needs some rebasing

dhilst · 2024-09-17T10:55:16Z

sorry but some tests are failing:

failures:
    items::date::tests::test_year
    items::date::tests::with_year
    items::tests::date_and_time
    tests::offsets::invalid_offset_format
    tests::test_relative::chrono_date
    tests::test_relative::gnu_compat
    tests::test_relative::test_month

and needs some rebasing

I will rebase it

But right now I have a problem with Chrono, I'm looking into alternatives like https://time-rs.github.io/book/how-to/create-dates.html

start parsing date with winnow

bf96790

dhilst mentioned this pull request Aug 29, 2024

[WIP] Add date format fallbacks uutils/coreutils#6667

Closed

dhilst force-pushed the clean-slate-2 branch 10 times, most recently from ff67b9e to 85d7574 Compare August 30, 2024 02:02

dhilst force-pushed the clean-slate-2 branch 16 times, most recently from 5516eed to 9388e45 Compare August 30, 2024 20:52

dhilst force-pushed the clean-slate-2 branch 2 times, most recently from 7592100 to 44a8141 Compare September 2, 2024 22:53

Add test_month to test relative dates

c76d675

dhilst force-pushed the clean-slate-2 branch from 44a8141 to c76d675 Compare September 2, 2024 22:54

dhilst requested review from sylvestre and tertsdiepraam September 5, 2024 13:39

dhilst force-pushed the clean-slate-2 branch 4 times, most recently from e858dfd to 417fde9 Compare September 14, 2024 00:01

Replace with_ method chains with new_date function, fix gnu compat issue

68498fa

dhilst force-pushed the clean-slate-2 branch from 417fde9 to 68498fa Compare September 14, 2024 00:19

dhilst commented Sep 14, 2024

View reviewed changes

dhilst force-pushed the clean-slate-2 branch 5 times, most recently from 2389d91 to 025808c Compare September 14, 2024 20:59

dhilst commented Sep 14, 2024

View reviewed changes

dhilst force-pushed the clean-slate-2 branch 4 times, most recently from 539a63f to 307c159 Compare September 15, 2024 00:56

WIP: Add fuzzy tests to get incompatibility issues

a434820

dhilst force-pushed the clean-slate-2 branch from 307c159 to a434820 Compare September 15, 2024 00:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean slate with Winnow 2 #86

Clean slate with Winnow 2 #86

dhilst commented Aug 29, 2024 •

edited

Loading

dhilst commented Aug 29, 2024

tertsdiepraam commented Aug 30, 2024

tertsdiepraam commented Sep 11, 2024

dhilst commented Sep 14, 2024

dhilst commented Sep 14, 2024

dhilst Sep 14, 2024

tertsdiepraam Sep 14, 2024

dhilst Sep 14, 2024

dhilst Sep 15, 2024

sylvestre commented Sep 17, 2024

dhilst commented Sep 17, 2024

Clean slate with Winnow 2 #86

Are you sure you want to change the base?

Clean slate with Winnow 2 #86

Conversation

dhilst commented Aug 29, 2024 • edited Loading

dhilst commented Aug 29, 2024

tertsdiepraam commented Aug 30, 2024

tertsdiepraam commented Sep 11, 2024

dhilst commented Sep 14, 2024

dhilst commented Sep 14, 2024

dhilst Sep 14, 2024

Choose a reason for hiding this comment

tertsdiepraam Sep 14, 2024

Choose a reason for hiding this comment

dhilst Sep 14, 2024

Choose a reason for hiding this comment

dhilst Sep 15, 2024

Choose a reason for hiding this comment

sylvestre commented Sep 17, 2024

dhilst commented Sep 17, 2024

dhilst commented Aug 29, 2024 •

edited

Loading