Testing plan 18.2

[sebastian]

@Aircloak/developers, we need to test our 18.2 release so we can get it out the door.

• FuzzTest: @obrok could you please perform an extended run on the release version? Report back the errors you find. We can then prioritize and determine which have priority for this release and which can be moved to the next release
• (@sasa1977) Compliance test: our CI only runs the compliance test on a very small number of users to have the tests pass more quickly. We could have introduced edge cases in this release that we are unaware of. We should perform a more extensive run and validate that there are no unexpected results.
  • 1000 users
  • 10000 5000 users
• (@sebastian, @fjab) Interface validation: walk through interface manually and ensure everything is as expected
• Validate new components
  • Licensing schema
    • [x] Deploy new central @sebastian @obrok has already done this
    • Validate that it works in connected mode
    • Validate correct behaviour on license expiry for connected system
    • Validate correct behaviour on license expiry for non-connected systems
    • [x] Validate that the warning is shown is shown, but not quite satisfactory. Issue is made. Parts will be improved for the subsequent release.
  • Audit log has seen changes. It should be played with to ensure it behaves as expected
• Manually run the TeamBank queries against the TeamBank dataset and validate behaviour is as expected
• (@sebastian) Run performance test: ./performance.sh 1000 5000 10000 25000 50000 100000
• Validate the changelog contains the relevant changes
• Walk through the release checklist

I would like to make this release on Friday the 20th of April.

[sebastian]

If there are any particular things you would like to test, please go ahead and add your name to the corresponding item. Some are probably better done by me.

[obrok]

I wanted to extend the fuzzer some, compared to last milestone, before running the tests. Maybe give me till the end of this week to add more features to it, and I will do the extended run next week?

[sebastian]

Maybe give me till the end of this week to add more features to it, and I will do the extended run next week?

Sure, sounds good

[sebastian]

Ran make perftest

Query: SELECT COUNT(*) FROM notes_changes
18.1.2: AVERAGE duration: 1.216s, STDDEV: 0.023s
18.2.0: AVERAGE duration: 0.534s, STDDEV: 0.019s

Query: SELECT COUNT(*) FROM notes
18.1.2: AVERAGE duration: 0.368s, STDDEV: 0.005s
18.2.0: AVERAGE duration: 0.151s, STDDEV: 0.006s

In other words, despite being a small and locally executed test (and one can argue with how representative these queries are), the performance seems to have improved.

Also the data for the performance run in diffix0 is also more or less like last time we did the test. Unfortunately getting the same test to run on 18.1.2 didn't work out of the box. That would have been a more interesting comparison.

[sasa1977]

I was able to execute compliance tests for 1000 users. The test took almost one hour, and reported 10 errors. I think that some of them have the same cause, so I suspect the actual number of errors is small. I'll create a separate issues for these errors.

[sasa1977]

I created the corresponding issues and tagged them with compliance and bug labels. You can see them here.

[sasa1977]

I've encountered a lot of problems trying to run the compliance tests for 10k users. In the end I had to do a reduced scope over 5k users and without sql server odbc and tds, mongo 3.0 and 3.2, and sap hana.

In total there were 20 failures. I'll go through all of them and create issues for the errors I haven't seen in the previous run.

Even in such reduced scope, the test took 63 hours (for comparison, a test over 1k users took 1 hour). I can't explain why such a huge increase, but I think it's safe to say that 10k doesn't seem attainable at the moment. I think that if we want to run compliance for a larger user set, we should rethink our input. Perhaps having less records per-user, and using smaller input sets (e.g. less names) could help us here.

[obrok]

I ran the fuzz tests for 2000 queries on top of the runs I've been doing during last week while developing the fuzzer. I found quite a number of issues, most rather obscure.

I tested with postgres and mysql only. Currently the fuzzer is set to run as a compliance test and expects the results to be the same for all datasources. This is made somewhat difficult for the different datasources which have their own little idiosyncrasies. Even mysql handles booleans somewhat differently from postgres and is more permissive in terms of operations crashing in the DB.

The most useful work items I foresee for the fuzzer:

1. Making an effort to select a uid from the inner query - because this is only done randomly, many queries are rejected because of the lack of such and so the fuzzer is bad at exploring queries with subqueries.
2. Shrinking the failing queries - it takes quite some work to extract a useful minimal example from a reported query. Currently it seems to me that there is a bug in stream_data that makes this impossible - Shrinking doesn't seem to work as advertised for one_of/frequency/tree whatyouhide/stream_data#97

[sebastian]

Release has been made.