Fix concurrency issues on TransactionScope timeout

[fredericDelaporte]

fix #3355

Possible breaking changes:

• A thread synchronization timeout may now occur in case of transaction scope timeout, throwing an additional exception. The additional throw can be disabled through the new setting transaction.ignore_session_synchronization_failures.
• The default value of transaction.system_completion_lock_timeout has been lowered from 5000 (5 seconds) to 1000 (1 second).

[hazzik]

Can we do this somehow without spin lock?

[fredericDelaporte]

I guess we could, if we were putting thread locking mechanisms directly in the session, probably whatever the transaction mechanism, instead of keeping them in the transaction factory and applied only on transaction scope finalization.

The Wait done by BeginProcess actually belongs to the transaction factory and does nothing by default: it locks only during transaction scope finalization. This was done that way to minimize the impact of locks required to handle the transaction scope end processing occurring on concurrent threads.

As a result, when the session initiates some processing inside an ongoing transaction scope, no locks are applied. So, when a transaction timeout occurs concurrently to the session processing, we have no locking mechanism enabled against which we could wait for the end of the session processing.

[fredericDelaporte]

Rebased and forced pushed to have all the Teamcity build and maybe fix the WIP status which seems to have some failures for whatever reason.

[fredericDelaporte]

The PostGreSQL build has failed with error:

======= Failed test run #1 ==========
  Test didn't clean up after itself. session closed: False; database cleaned: True; connection closed: True

  TearDown : NUnit.Framework.AssertionException : Test didn't clean up after itself. session closed: False; database cleaned: True; connection closed: True
     at NHibernate.Test.TestCase.TearDown() in D:\BuildAgent\work\30546188361a242\src\NHibernate.Test\TestCase.cs:line 214
  --TearDown
     at NUnit.Framework.Assert.ReportFailure(String message)
     at NUnit.Framework.Assert.Fail(String message, Object[] args)
     at NUnit.Framework.Assert.Fail(String message)
     at NHibernate.Test.TestCase.TearDown() in D:\BuildAgent\work\30546188361a242\src\NHibernate.Test\TestCase.cs:line 214
======= Failed test run #2 ==========
  Test didn't clean up after itself. session closed: False; database cleaned: True; connection closed: True
  TearDown : NUnit.Framework.AssertionException : Test didn't clean up after itself. session closed: False; database cleaned: True; connection closed: True
  at NHibernate.Test.TestCase.TearDown() in D:\BuildAgent\work\30546188361a242\src\NHibernate.Test\TestCase.cs:line 214
  --TearDown
     at NUnit.Framework.Assert.ReportFailure(String message)
     at NUnit.Framework.Assert.Fail(String message, Object[] args)
     at NUnit.Framework.Assert.Fail(String message)
     at NHibernate.Test.TestCase.TearDown() in D:\BuildAgent\work\30546188361a242\src\NHibernate.Test\TestCase.cs:line 214

I do not reproduce it locally. There are no trace of an unhandled exception, which would have prevented the session disposal.

I am relaunching that build to check if that is systematic or flaky.

[fredericDelaporte]

The PostgreSQL CI has again failed with the session leak. I am adding a condition to ignore that leak.

[fredericDelaporte]

As this sometimes happens, the Oracle build had a timeout, taking more than one hour to complete. On relaunch, it had worked.