You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fluo's notifications are currently placed in a separate locality group. This means when scanning for notifications, user data is not read. Notifications can only be garbage collected with a full major compaction. However full major compactions do not happen often. This means that over time notification delete markers build up. This build up over time makes scanning for notifications more expensive.
Work was done for #457 to drop delete marker for notifications that were inserted and processed before an Accumulo minor compaction happens. The work in #457 also has a chance of dropping delete markers on partial compactions. However if a delete marker makes it to the largest files, it may be there for a long time. The effectiveness of #457 is inversely proportional to the average time it takes to process a notification. As the average time goes up, the effectiveness goes down. If too many delete markers have built up, then the best work around is a full compaction of the table.
If notifications were in a separate table, then the full compaction could be cheap. Also the work done for ACCUMULO-4500 could be leveraged to automatically make smart decisions about when to compact based on the number of notifications vs the number of delete markers.
The text was updated successfully, but these errors were encountered:
Fluo's notifications are currently placed in a separate locality group. This means when scanning for notifications, user data is not read. Notifications can only be garbage collected with a full major compaction. However full major compactions do not happen often. This means that over time notification delete markers build up. This build up over time makes scanning for notifications more expensive.
Work was done for #457 to drop delete marker for notifications that were inserted and processed before an Accumulo minor compaction happens. The work in #457 also has a chance of dropping delete markers on partial compactions. However if a delete marker makes it to the largest files, it may be there for a long time. The effectiveness of #457 is inversely proportional to the average time it takes to process a notification. As the average time goes up, the effectiveness goes down. If too many delete markers have built up, then the best work around is a full compaction of the table.
If notifications were in a separate table, then the full compaction could be cheap. Also the work done for ACCUMULO-4500 could be leveraged to automatically make smart decisions about when to compact based on the number of notifications vs the number of delete markers.
The text was updated successfully, but these errors were encountered: