Skip to content

Commit

Permalink
Increase journal insertion speed
Browse files Browse the repository at this point in the history
As the journal table is locked per #685 anyways,
we can guarantee that max(_new_serials_per_source)
is the same as running MAX(serial_nrtm) per source
after the first insert.

The MAX(serial_nrtm) query is actually really slow,
so avoiding this has significant benefits, and
reduces the cost of locking the table.
  • Loading branch information
mxsasha committed Jan 13, 2023
1 parent caf7b58 commit 96fe352
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 3 deletions.
4 changes: 4 additions & 0 deletions docs/releases/4.3.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,7 @@ Other changes
log file, before dropping the privileges. Previously, it would
drop privileges, then fail to write to the log file, and be unable
to report this error.
* Inserting the journal entries is now much faster when inserting
multiple entries per source. This happens in NRTM mirroring
or when users submit large sets of changes. IRRd will process
these changes faster up to an order of magnitude.
9 changes: 6 additions & 3 deletions irrd/storage/database_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -766,9 +766,12 @@ def record_operation(self, operation: DatabaseOperation, rpsl_pk: str, source: s
if self._is_serial_synchronised(source):
serial_nrtm = source_serial
else:
serial_nrtm = sa.select([sa.text('COALESCE(MAX(serial_nrtm), 0) + 1')])
serial_nrtm = serial_nrtm.where(RPSLDatabaseJournal.__table__.c.source == source)
serial_nrtm = serial_nrtm.as_scalar()
if source in self._new_serials_per_source and self._new_serials_per_source[source]:
serial_nrtm = max(self._new_serials_per_source[source]) + 1
else:
serial_nrtm = sa.select([sa.text('COALESCE(MAX(serial_nrtm), 0) + 1')])
serial_nrtm = serial_nrtm.where(RPSLDatabaseJournal.__table__.c.source == source)
serial_nrtm = serial_nrtm.as_scalar()
timestamp = datetime.now(timezone.utc)
stmt = RPSLDatabaseJournal.__table__.insert().values(
rpsl_pk=rpsl_pk,
Expand Down

0 comments on commit 96fe352

Please sign in to comment.