Very long string attributes used in indices cause exceptions

[aanastasiou]

The problem

If a string attribute happens to be too long (>>2kB) and also used as an index, then we may be hitting Neo4J's key length limit which is 4095 bytes.

If the string attribute is very long (e.g. ~8kB) then a very simple exception is thrown informing us about the problem. It goes along the lines of "Property value bytes length: 6198 is longer than 4095, which is maximum supported length of indexed property value".

But when the byte length of the string is close to the 4095 characters limit (but less than) then the transaction goes ahead without any complains but the server enters an unusable state from which it can only recover by a restart (possibly even involving completely wiping out the database directory).

A more detailed description and test case can be found in this repository

Impact

At the moment, saving single nodes with really long StringProperty only produces a safely handled exception from the database.

At the same time, neomodel's batch operations will fail completely if they contain 2 data items in the right conditions. This invalidates the complete batch operation and has the potential to also crash silently.

Fixing it

A first proposal for fixing the problem is available here.

This can be used during the validation step of StringProperty taking into account any available information about whether the property is supposed to be used as an index.

If the property is to be used as an index then it has to be trimmed.

If the property does get trimmed however we still run the danger of two different strings that happen to share the first N bytes (where N<4095) effectively being identified as the same string. In this case we need to also flag this with some kind of a warning during schema creation (i.e. within neomodel_install_labels.py perhaps (?)).

[aanastasiou]

The server side bug that caused this problem was fixed in 3.4.7 (I was using 3.4.5). Upgrading to 3.4.9 solved the problem.

However this still leaves us with a potential exception that we can catch at the StructuredNode level to avoid an exception from the database and potentially offer a user definable default behaviour of trimming the string to the key byte limit (?). This could go into the validation of StringProperty.