Sqlite3 Support #91
Merged
Sqlite3 Support #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sqlite3 is the default database in iOS, and generating custom formulas offline in javascript is great, but being able to query the offline cache directly is even better. General Changes - DatePart extraction moved to hooks to make it easier to reason with - Log & NaturalLog moved to hooks as well, since Sqlite doesn't have a standard implementation
Using ROUND(...,33) in sqlite causes more issues than it solves for exact to inexact conversion for ceil & floor. So allow the SqlHooks to turn off the rounding and control the precision Have special hooks for testing without the standard math functions, since the sqlite-jdbc doesn't use the standard math functions (see xerial/sqlite-jdbc#763) Turn off binding because the default driver uses numbers for date binding when testing More fixes for sqlite for date math
Add in goldfiles Duration, addMonths now work Simulate lpad/rpad using zeroblob trick FormatCurrency, IsoWeek/Month, InitCap still don't work
Fix goldfile difference in spanner due to error message change due to fallback on date/tstamp retrievable from the jdbc resultset
steventamm
added a commit
that referenced
this pull request
Oct 25, 2022
* Version 0.3.0
* Better support for multiple DBs
- Allow nested whyIgnoreSql elements in formulatests.xml instead of an attribute to make
merges, readability easier
- Allow specifying whether to ignore SQL because it's unimplemented or a specific number
of failure so that various sql issues aren't just swept under the rug.
- Be more precise about why SQL is failures and split formulatext.xml
- Do some refactoring around date/time logic to rely on the hooks for more
- Move math-like formulas to formulatests-math.xml
- Use Concat and not + for JsonValue
- DatePart extraction moved to hooks to make it easier to reason with
- Log & NaturalLog moved to hooks as well, since Sqlite doesn't have a standard implementation
* Add in support for Google Standard SQL
- Run tests against a Spanner local emulator
- Google Standard SQL has distinct date and timestamp, so some functions/hooks
needed to be split up between date and datetime since
google standard sql doesn't autoconvert date to datetime
- Google has no time type at all, so no reliance on any TIME at all
- Spanner has a maximum scale of 9 and lower precision than other DBs, so
the binding of BigDecimals needs to be customized to be only 9
- Implement numFailures for spanner.
- Fix goldfile difference in spanner due to error message change due to fallback on date/tstamp
retrievable from the jdbc resultset
* Mysql/MarisDB fixes
- Abstract Mysql and MariaDB testers to share time parsing stuff.
- MariaDB's TIME function doesn't parse correctly, so have a separate hook for that.
- MariaDB's JDBC driver uses time, so only convert in MySQL to integer
Note: MariaDB tests are not run by default for perf reasons
* Presto Support (for Amazon Athena and the like)
- Support for Presto/Trino with the formula engine to support Presto-style DBs
such as Amazon Athena. It uses TrinoDB to test with testcontainers, but
is restricted to Presto 0.277 syntax for backwards compatibility
- Note: These tests aren't run with -P db-tests. It's just too slow.
- But, like MariaDB, it works if you enable it or run it by hand.
* Presto Changes
- Query exceptions in Trino contain a UUID, so that has to be stripped with
getSqlExceptionMessage()
- Presto doesn't support unary +, so distance needed some work to remove it
- Presto is more fiddly with timestamp vs date than other DBs
- Date math more fully implemented
- Rounding mode is HALF_UP, not HALF_EVEN unlike other DBs. Causes some issues.
- Make superclass of Presto and TrinoContainerTester
- Presto with ' / 100.0' for percent was reducing the scale and making some
of the numbers wrong. Since you can't use binding, and have to use
literal because of the JDBC BD bind issue, so this improves things
- Use explicit decimal literals with Presto instead of 100.0 to prevent truncation
- Log/Ln need explicit decimal casting. Others may as well..
- Make presto TRUNC return DECIMAL(38,18)
- Use more decimals with the percent conversion and the conversion it in the SqlHooks.
- Presto can't format intervals, so don't bother with them. Do it using the java
format stuff that's in Presto/Trino
- Use trim when parsing dates to match java
- Use DECIMAL '100.' to prevent decimal overflow when using percents.
There's still precision differences in functions with it, however
- Ignore precision differences like in postgres
- Time formatting fixes
- Ignore some functions that aren't supported like "right"
* Sqlite3 Support (#91)
- Sqlite3 is the default database in iOS, and generating custom formulas offline in javascript is
great, but being able to query the offline cache directly is even better.
- Using ROUND(...,33) in sqlite causes more issues than it solves for exact to inexact conversion
for ceil & floor. So allow the SqlHooks to turn off the rounding and control the precision
- Have special hooks for testing without the standard math functions, since the sqlite-jdbc
doesn't use the standard math functions (see xerial/sqlite-jdbc#763)
- Turn off binding because the default driver uses numbers for date binding when testing
- Simulate lpad/rpad using zeroblob trick
- FormatCurrency, IsoWeek/Month, InitCap still don't work
- Mac and Linux are... different when it comes to their precision and implementations of certain
mathematical functions. So many of the tests have different results for Mac vs Linux which makes
CI/CD difficult. Paper over these difficulties by rounding the output to 12 decimals at most when
comparing the results.
- Remove test with Mac vs Linux differences from Sqlite testing
Linux throws an "Numerical result out of range" exception that Mac doesn't
Since we don't want different results for different architectures, paper over the difference by
returning null when doing an overflow, which the Mac version of sqlite doesIt's too variable to manage,
so just ignore the tests: testExponentiationOperator, testTruncUsesIf, and testTruncUsesTruncMinus
Will try again with 3.40
* Coverage Improvements
- Coverage for composite command and getFieldPathIdDirect...
- Test inaccessible field strategy
- And remove warnings for duplicate versions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sqlite3 is included in iOS and android devices, and just like javascript support was added to support offline display and
validation rules, adding in Sqlite3 support for doing formula queries against an offline cache seems quite useful.
Sqlite3 has many different compilation options, and the test drive used here
sqlite-jdbchas some functions that aredifferent from what would be available on those systems. So there's some hackery around the lack of a trunc function
that will be added when Sqlite3 hits 3.40.
Sqlite3 has weak typing, so invalid Datetime strings are treated as nulls and can't be fixed. So many of the date/time
tests fail because lack of a 2 digit month fail in Sqlite3, but work in other DBs.
Log, Ln, and Mod was moved to the hooks for similar reasons (since they are different in the sqlite-jdbc used for testing)
Greatest and Least are Max and Min in Sqlite3
Sqlite3 doesn't support negative scale for round, so Round was moved to a hook to support that.
InitCap, IsoWeek/IsoMonth are not supported at all.
General Changes:
Stuff around date extraction (Day,Minute,etc) was moved to the hooks to make it easier to handle at once
Real numbers use double precision in Sqlite, so ROUND(...,33) for Ceiling and Floor are worse than doing nothing at all.
To support fixing precision errors in other DB, it was moved to the hooks. In a second PR, this may be fixed
Extra parens when using Trunc were removed