Merge pull request #30 from bxparks/develop

merge v1.1 into master

.github/workflows/ccpp.yml

@@ -19,10 +19,8 @@ jobs:
  git clone https://github.com/bxparks/AUnit
  - name: Verify examples
  run: |
- make -C examples clean
  make -C examples
  - name: Verify tests
  run: |
- make -C tests clean
  make -C tests
  make -C tests runtests

.github/workflows/pythonpackage.yml

@@ -0,0 +1,42 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Python package
+
+on:
+ push:
+ branches: [ develop ]
+ pull_request:
+ branches: [ develop ]
+
+jobs:
+ build:
+
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ python-version: [3.7, 3.8]
+
+ steps:
+ - uses: actions/checkout@v2
+ - name: Set up Python ${{ matrix.python-version }}
+ uses: actions/setup-python@v1
+ with:
+ python-version: ${{ matrix.python-version }}
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ # pip install -r requirements.txt
+ pip install flake8 mypy pytz python-dateutil
+ - name: Lint with flake8
+ run: |
+ cd tools
+ make flake8
+ - name: Type checking with MyPy
+ run: |
+ cd tools
+ make mypy
+ - name: Test with unittest
+ run: |
+ cd tools
+ make tests

.gitignore

@@ -65,3 +65,13 @@ wheels/
 *.egg
 MANIFEST
 *.app
+
+#-------------------------------------------------------------------------
+# Generated files and directories under tests/validation/*Test/
+#-------------------------------------------------------------------------
+
+tests/validation/*Test/validation_*.h
+tests/validation/*Test/validation_*.cpp
+tests/validation/*Test/validation_*.json
+tests/validation/*Test/zonedb*
+tests/validation/*Test/tzfiles

CHANGELOG.md

@@ -1,6 +1,21 @@
 # Changelog
 
 * Unreleased
+* 1.1 (2020-04-25, TZ DB version 2020a)
+ * Fix broken links in `README.md`.
+ * Fix typos in `USER_GUIDE.md` and update its version number to 1.0 as it
+ should have been back in October.
+ * Massive refactor of `./tools` processing pipeline and update
+ `validation/tests`.
+ * Add mypy strict type checking for Python scripts under `tools`.
+ * Funnel `validation*.{h,cpp}` code generation through a single program
+ using a `validation_data.json` intermediate file.
+ * Funnel processing of TZDB output from `transformer.py` into a single
+ `tzdbcollector.py` which can produce a `tzdb.json` output.
+ * Separate `validator.py` processing into a distinct section.
+ * Add validation tests against Python `dateutil` library (similar to
+ `pytz`).
+ * Update TZ Database version to 2020a that was released on 2020-04-23.
 * 1.0 (2019-10-02, TZ DB version 2019c)
  * Add initial support for GitHub actions to implement continuous integration
  using the unit tests that run under UnitHostDuino.

README.md

@@ -121,11 +121,13 @@ The zoneinfo files (and their associated `ZoneProcessor` classes) have a
 resolution of 1 minute, which is sufficient to represent all UTC offsets and DST
 shifts of all timezones after 1972 (Africa/Monrovia was the last timezone to
 switch to UTC time on Jan 7, 1972). These zoneinfo files and the algorithms in
-this library have been validated to match the UTC offsets calculated using 3
+this library have been validated to match the UTC offsets calculated using 4
 other date/time libraries:
 
 * the Python [pytz](https://pypi.org/project/pytz/) library from
  the year 2000 until 2037 (inclusive),
+* the Python [dateutil](https://pypi.org/project/python-dateutil/) library from
+ the year 2000 until 2037 (inclusive),
 * the Java JDK 11
  [java.time](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/package-summary.html)
  library from year 2000 to 2049 (inclusive),
@@ -209,9 +211,7 @@ Conversion from an epochSeconds to date-time components including timezone
 * 2.8 microseconds on an ESP32,
 * 6 microseconds on a Teensy 3.2.
 
-**Version**: 1.0 (2019-10-02, TZ DB version 2019c)
-
-**Status**: Graduated to v1.0.
+**Version**: 1.1 (2020-04-25, TZ DB version 2020a)
 
 ## Examples
 
@@ -508,15 +508,6 @@ I will occasionally test on the following hardware as a sanity check:
 * Teensy 3.2 (72 MHz ARM Cortex-M4)
 * Mini Mega 2560 (Arduino Mega 2560 compatible, 16 MHz ATmega2560)
 
-### Untested RTC hardware compatibility
-
-As DS1307 and DS3232 RTC chip have exactly same interface as DS3231 for used
-features except temperature on DS1307 (this one lacks this functionality),
-this class could be be used to control all of them.
-
-Any problem with these RTCs please, contact @Naguissa (https://github.com/Naguissa/AceTime)
-
-
 ## Changelog
 
 See [CHANGELOG.md](CHANGELOG.md).

USER_GUIDE.md

@@ -2,7 +2,7 @@
 
 See the [README.md](README.md) for introductory background.
 
-Version: 0.8.1 (2019-08-26, TZ DB version 2019b, beta)
+Version: 1.1 (2020-04-25, TZ DB version 2020a)
 
 ## Installation
 
@@ -68,9 +68,10 @@ usually have more precise dependency information:
 Various scripts in the `tools/` directory depend on:
 
 * [IANA TZ Database on GitHub](https://github.com/eggert/tz)
-* [pytz library](https://pypi.org/project/pytz/)
+* [Python pytz library](https://pypi.org/project/pytz/)
+* [Python dateutil library](https://pypi.org/project/python-dateutil)
 * [Hinnant date library](https://github.com/HowardHinnant/date)
-* Python 3.5 or greater
+* Python 3.6 or greater
 * Java OpenJDK 11
 
 If you want to run the unit tests or some of the command line examples using a
@@ -770,7 +771,7 @@ TimeZone <>-------- ZoneProcessor ------- ZoneProcessorCache
  .----- +----. .---- +-----.
  | | | |
  BasicZone ExtendedZone BasicZone ExtendedZone
- Processor Processor ProcessorCache ZoneProcessor
+ Processor Processor ProcessorCache ProcessorCache
 ```
 
 Here is the class declaration of `TimeZone`:
@@ -1620,11 +1621,6 @@ by zoneName or zoneId) into an index into the registry.
 The IANA TZ Database is updated continually. As of this writing, the latest
 stable version is 2019b. When a new version of the database is released, it is
 relatively easy to regenerate the `zonedb/` and `zonedbx/` zoneinfo files.
-However, it is likely that I would delay the release of a new version until the
-corresponding `pytz` package is updated to the latest TZ database version, so
-that the validation test suites pass (See Testing section below). Otherwise, I
-don't have a way to verify that the AceTime library with the new TZ Database
-version is correctly functioning.
 
 ## Mutations
 
@@ -2029,6 +2025,11 @@ void loop() {
 }
 ```
 
+It has been claimed that the DS1307 and DS3232 RTC chips have exactly same
+interface as DS3231 when accessing the time and date functionality. I don't have
+these chips so I cannot confirm that. Contact @Naguissa
+(https://github.com/Naguissa) for more info.
+
 ### System Clock
 
 The `SystemClock` is a special `Clock` that uses the Arduino built-in `millis()`
@@ -2344,9 +2345,10 @@ timezones.
 
 My next idea was to validate AceTime against a known, independently created,
 timezone library that also supports the TZ Database. Currently, I validate
-the AceTime library against 3 other timezone libraries:
+the AceTime library against 4 other timezone libraries:
 
 * Python [pytz](https://pypi.org/project/pytz/)
+* Python [dateutil](https://pypi.org/project/python-dateutil)
 * Java 11 [java.time](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/package-summary.html)
 * C++11/14/17 [Hinnant date](https://github.com/HowardHinnant/date)
 
@@ -2359,8 +2361,8 @@ found in the future.
 The Python pytz library was a natural choice since the `tzcompiler.py` was
 already written in Python. I created:
 
-* [BasicValidationUsingPythonTest](tests/validation/BasicValidationUsingPythonTest/)
-* [ExtendedValidationUsingPythonTest](tests/validation/ExtendedValidationUsingPythonTest/)
+* [BasicPythonTest](tests/validation/BasicPythonTest/)
+* [ExtendedPythonTest](tests/validation/ExtendedPythonTest/)
 
 The `pytz` library is used to generate various C++ source code
 (`validation_data.cpp`, `validation_data.h`, `validation_tests.cpp`) which
@@ -2377,31 +2379,45 @@ executed only on desktop-class Linux or MacOS machines through the use of the
 The `pytz` library supports [dates only until
 2038](https://answers.launchpad.net/pytz/+question/262216). It is also tricky to
 match the `pytz` version to the TZ Database version used by AceTime. The
-following combinations have been tested:
+following versions of `pytz` have been tested:
+
+* pytz 2019.1, containing TZ Datbase 2019a
+* pytz 2019.2, containing TZ Datbase 2019b
+* pytz 2019.3, containing TZ Datbase 2019c
+
+### Python dateutil
+
+Validation against the Python dateutil library is similar to pytz. I created:
+
+* [BasicDateUtilTest](tests/validation/BasicDateUtilTest/)
+* [ExtendedDateUtilTest](tests/validation/ExtendedDateUtilTest/)
+
+Similar to the `pytz` library, the `dateutil` library supports [dates only until
+2038](https://github.com/dateutil/dateutil/issues/462). The
+following versions of `dateutil` have been tested:
 
-* TZ Datbase: 2019a; pytz: 2019.1
-* TZ Datbase: 2019b; pytz: 2019.2
+* dateutil 2.8.1, containing TZ Datbase 2019c
 
 ### Java java.time
 
 The Java 11 `java.time` library is not limited to 2038 but supports years
-through the [year 1000000000
+through the [year 1,000,000,000
 (billion)](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/class-use/Instant.html).
-I wrote the [TestDataGenerator.java](tools/java/TestDataGenerator) program to
-generate a `validation_data.cpp` file in exactly the same format as the
-`tzcompiler.py` program, and produced data points from year 2000 to year 2050,
-which is the exact range of years supported by the `zonedb::` and `zonedbx::`
-zoneinfo files.
+I wrote the [TestDataGenerator.java](tools/compare_java/TestDataGenerator.java)
+program to generate a `validation_data.cpp` file in exactly the same format as
+the `tzcompiler.py` program, and produced data points from year 2000 to year
+2050, which is the exact range of years supported by the `zonedb::` and
+`zonedbx::` zoneinfo files.
 
 The result is 2 validation programs under `tests/validation`:
 
-* [BasicValidationUsingJavaTest](tests/validation/BasicValidationUsingJavaTest/)
-* [ExtendedValidationUsingJavaTest](tests/validation/ExtendedValidationUsingJavaTest/)
+* [BasicJavaTest](tests/validation/BasicJavaTest/)
+* [ExtendedJavaTest](tests/validation/ExtendedJavaTest/)
 
 The most difficult part of using Java is figuring out how to install it
 and figuring out which of the many variants of the JDK to use. On Ubuntu 18.04,
 I used `openjdk 11.0.4 2019-07-16` which seems to use TZ Database 2018g. I have
-no recollection how I installed, I think it was something like `$ sudo apt
+no recollection how I installed it, I think it was something like `$ sudo apt
 install openjdk-11-jdk:amd64`.
 
 The underlying timezone database used by the `java.time` package seems to be
@@ -2421,14 +2437,15 @@ powerful, complex and difficult to use. I managed to incorporate it into 2 more
 validation tests, and verified that the AceTime library matches the Hinnant date
 library for all timezones from 2000 to 2049 (inclusive):
 
-* [BasicValidationUsingHinnantDateTest](tests/validation/BasicValidationUsingHinnantDateTest/)
-* [ExtendedValidationUsingHinnantDateTest](tests/validation/ExtendedValidationUsingHinnantDateTest/)
+* [BasicHinnantDateTest](tests/validation/BasicHinnantDateTest/)
+* [ExtendedHinnantDateTest](tests/validation/ExtendedHinnantDateTest/)
 
-I have validated the AceTime library with the following TZ versions against
-the Hinnant date library with the same TZ version:
-
-* TZ Database: 2019a
-* TZ Database: 2019b
+I have validated the AceTime library with the Hinnant date library for the
+following TZ Dabase versions:
+* TZ DB version 2019a
+* TZ DB version 2019b
+* TZ DB version 2019c
+* TZ DB version 2020a
 
 ## Benchmarks
 
@@ -2630,10 +2647,10 @@ environment because:
  library.
 
 The Hinnant date libraries were invaluable for writing the
-[BasicValidationUsingHinnantDateTest](tests/validation/BasicValidationUsingHinnantDateTest/)
+[BasicHinnantDateTest](tests/validation/BasicHinnantDateTest/)
 and
-[ExtendedValidationUsingHinnantDateTest](tests/validation/ExtendedValidationUsingHinnantDateTest/)
-validation tests (in v0.7) which are the AceTime algorithms to the Hinnant Date
+[ExtendedHinnantDateTest](tests/validation/ExtendedHinnantDateTest/)
+validation tests which compare the AceTime algorithms to the Hinnant Date
 algorithms. For all times zones between the years 2000 until 2050, the AceTime
 UTC offsets (`TimeZone::getUtcOffset()`), timezone abbreviations
 (`TimeZone::getAbbrev()`), and epochSecond conversion to date components
@@ -2644,8 +2661,8 @@ libraries.
 
 The [cctz](https://github.com/google/cctz) library from Google is also based on
 the `<chrono>` library. I have not looked at this library closely because I
-assumed that it would fit inside an Arduino controller. Hopefully I will get
-some time to take a closer look in the future.
+assumed that it would *not* fit inside an Arduino controller. Hopefully I will
+get some time to take a closer look in the future.
 
 ## Bugs and Limitations
 
@@ -2703,10 +2720,11 @@ some time to take a closer look in the future.
  not load the entire TZ Database due to memory constraints of most Arduino
  boards.
 * `TimeZone`
- * It might be possible to use a Basic `TimeZone` created using a `zonedb::`
- zoneinfo file, and an Extended `TimeZone` using a `zonedbx::` zoneinfo
- file. However, this is not a configuration that is expected to be used
- often, so it has not been tested well, if at all.
+ * It might be possible to use both a Basic `TimeZone` created using a
+ `zonedb::` zoneinfo file, and an Extended `TimeZone` using a `zonedbx::`
+ zoneinfo file, together in a single program. However, this is not a
+ configuration that is expected to be used often, so it has not been tested
+ well, if at all.
  * One potential problem is that the equality of two `TimeZone` depends only
  on the `zoneId`, so a Basic `TimeZone` created with a
  `zonedb::kZoneAmerica_Los_Angeles` will be considered equal to an Extended
@@ -2743,8 +2761,11 @@ some time to take a closer look in the future.
  but this is sufficient to support the vast majority of timezones since
  2000.
  * The `zonedb/` files have been filtered to satisfy these constraints.
- * Tested again Python [pytz](https://pypi.org/project/pytz/) from
+ * Tested against Python [pytz](https://pypi.org/project/pytz/) from
  2000 until 2038 (limited by pytz).
+ * Tested against Python
+ [dateutil](https://pypi.org/project/python-dateutil/) from
+ 2000 until 2038 (limited by dateutil).
  * Tested against Java `java.time` from 2000 to until 2050.
  * Tested against C++11/14/17
  [Hinnant date](https://github.com/HowardHinnant/date) from 2000 until
@@ -2753,12 +2774,15 @@ some time to take a closer look in the future.
  * Has a 1-minute resolution for all time fields.
  * The `zonedbx/` files currently (version 2019b) do not have any timezones
  with 1-minute resolution.
- * Tested again Python [pytz](https://pypi.org/project/pytz/) from
+ * Tested against Python [pytz](https://pypi.org/project/pytz/) from
  2000 until 2038 (limited by pytz).
+ * Tested against Python
+ [dateutil](https://pypi.org/project/python-dateutil/) from
+ 2000 until 2038 (limited by dateutil).
  * Tested against Java `java.time` from 2000 to until 2050.
  * Tested against [Hinnant date](https://github.com/HowardHinnant/date)
  using 1-minute resolution from 1975 to 2050. See
- [ExtendedValidationUsingHinnantDateTest](tests/validation/ExtendedValidationUsingHinnantDateTest).
+ [ExtendedHinnantDateTest](tests/validation/ExtendedHinnantDateTest).
 * `zonedb/` and `zonedbx/` zoneinfo files
  * These statically defined data structures are loaded into flash memory
  using the `PROGMEM` keyword. The vast majority of the data structure
@@ -2831,12 +2855,3 @@ some time to take a closer look in the future.
  SAMD21 Mini Breakout` board. The `MKR Zero` could be using a different
  (more recent?) version of the GCC tool chain. I have not investigated
  this.
-
-
-## Untested RTC hardware compatibility
-
-As DS1307 and DS3232 RTC chip have exactly same interface as DS3231 for used
-features except temperature on DS1307 (this one lacks this functionality),
-this class could be be used to control all of them.
-
-Any problem with these RTCs please, contact @Naguissa (https://github.com/Naguissa/AceTime)

docs/doxygen.cfg

@@ -38,7 +38,7 @@ PROJECT_NAME = "AceTime"
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER = 1.0
+PROJECT_NUMBER = 1.1
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a