Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schedule ROA load each 15 minutes #1444

Merged
merged 58 commits into from
Apr 18, 2024
Merged
Changes from 50 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
111a76c
feat: create new warning for roas check and test for legacy
MiguelAHM Mar 27, 2024
2a78606
feat: add test for REST API calls and little refactor to specify rest…
MiguelAHM Mar 27, 2024
88a4b53
feat: fix ITs
MiguelAHM Mar 28, 2024
bafe2de
feat: add more tests
MiguelAHM Mar 28, 2024
e5493f1
feat: create the validator
MiguelAHM Mar 28, 2024
39b5818
feat: fix compile error due to previous code
MiguelAHM Mar 28, 2024
0b0fb2a
feat: renaming and deleting white spaces
MiguelAHM Mar 28, 2024
61db61e
feat: fix tests according to last changes
MiguelAHM Mar 28, 2024
c44316d
feat: fix conflicts
MiguelAHM Apr 3, 2024
c27dd40
feat: remove already migrated classes
MiguelAHM Apr 3, 2024
2db7bc5
feat: change generator by validator
MiguelAHM Apr 3, 2024
607a329
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 3, 2024
ea8abb6
feat: fix compilation errors
MiguelAHM Apr 3, 2024
095701f
feat: increase testperformance to 3s
MiguelAHM Apr 4, 2024
f386982
feat: put timeout back to 2s and fix compilation issue in whoisFixture
MiguelAHM Apr 4, 2024
24cbc26
feat: fix compilation issues
MiguelAHM Apr 4, 2024
14b95be
feat: fix ITs
MiguelAHM Apr 4, 2024
f45fc79
feat: refactor messages and PR comments
MiguelAHM Apr 4, 2024
e844ec2
feat: fix ITs
MiguelAHM Apr 5, 2024
169397b
feat: refactor
MiguelAHM Apr 5, 2024
ad5c241
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 5, 2024
72e3c34
feat: rename
MiguelAHM Apr 5, 2024
43b4b86
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 5, 2024
4f24215
feat: rename
MiguelAHM Apr 5, 2024
da65fed
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 8, 2024
4059fd4
Return 404 on invalid NRTMv4 notification file path (#1422)
maggarwal13 Apr 8, 2024
c40ee90
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 8, 2024
d58fe00
feat: fix logger issue (#1427)
MiguelAHM Apr 9, 2024
04c8bc3
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 9, 2024
ce0a9ee
remove_unused_code (#1429)
maggarwal13 Apr 9, 2024
11d66b2
Revert changes to thread configuration in tests (#1415)
eshryane Apr 9, 2024
af967b6
Handle Multiple Delivery Status Notification Failure Responses for th…
MiguelAHM Apr 10, 2024
83f0ab3
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 10, 2024
76d9535
Fix tests (#1433)
maggarwal13 Apr 10, 2024
56dbc2e
reduce mariadb innodb memory usage to 1GB (#1439)
eshryane Apr 11, 2024
a610dcb
Set memory limit for test elasticsearch instance (#1436)
fmurodov Apr 11, 2024
a4ca185
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 11, 2024
6bcf6f4
feat: renaming, revert ErrorMessages changes, change the query flag, …
MiguelAHM Apr 11, 2024
5590d8f
Merge branch 'master' into DB-5291-flag-route-roa-validator
MiguelAHM Apr 11, 2024
2aa2f02
feat: use a queryParameter instead of queryFlag
MiguelAHM Apr 11, 2024
ce6770c
feat: add stopwatch to monitor the performance of RPKI service
MiguelAHM Apr 12, 2024
f7b54de
feat: add missing return
MiguelAHM Apr 12, 2024
3d54e92
feat: rpki service should be a @Service and RpkiRoaChecker a componen…
MiguelAHM Apr 12, 2024
03361ed
Merge branch 'DB-5291-flag-route-roa-validator' into DB-5352-schedule…
MiguelAHM Apr 12, 2024
6362f65
feat: schedule roas
MiguelAHM Apr 12, 2024
cc05a1a
feat: merge master
MiguelAHM Apr 17, 2024
50cf9dd
feat: remove blank lines
MiguelAHM Apr 17, 2024
5c6c871
Merge branch 'master' into DB-5352-schedule_roa_load
MiguelAHM Apr 17, 2024
17aa594
use SynchonizedIntervalMap to avoid concurrent problems
MiguelAHM Apr 17, 2024
3663ec9
feat: remove unused import
MiguelAHM Apr 17, 2024
cfabfb7
feat: fix cron expresion
MiguelAHM Apr 17, 2024
a6fbe8a
feat: intantiate new interval instead using clean
MiguelAHM Apr 17, 2024
457f1ba
feat: fix each seconds issue
MiguelAHM Apr 17, 2024
c8d0120
feat: use nestedIntervalMap instead async map
MiguelAHM Apr 17, 2024
92e6fbb
feat: remove initial inizialisation
MiguelAHM Apr 17, 2024
7098bf5
feat: change message error and refactor
MiguelAHM Apr 18, 2024
8d9235a
feat: rename
MiguelAHM Apr 18, 2024
b8b33be
feat: avoid returning a pair
MiguelAHM Apr 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,15 @@

import com.google.common.collect.Sets;
import net.ripe.db.whois.common.collect.CollectionHelper;
import net.ripe.db.whois.common.etree.IntervalMap;
import net.ripe.db.whois.common.etree.NestedIntervalMap;
import net.ripe.db.whois.common.etree.SynchronizedIntervalMap;
import net.ripe.db.whois.common.ip.IpInterval;
import net.ripe.db.whois.common.ip.Ipv4Resource;
import net.ripe.db.whois.common.ip.Ipv6Resource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;

import java.util.Collection;
Expand All @@ -20,24 +23,35 @@ public class RpkiService {

private static final Logger LOGGER = LoggerFactory.getLogger(RpkiService.class);

private final NestedIntervalMap<Ipv4Resource, Set<Roa>> ipv4Tree = new NestedIntervalMap<>();
private final NestedIntervalMap<Ipv6Resource, Set<Roa>> ipv6Tree = new NestedIntervalMap<>();
final IntervalMap<Ipv4Resource, Set<Roa>> ipv4Tree = SynchronizedIntervalMap.synchronizedMap(new NestedIntervalMap<Ipv4Resource, Set<Roa>>());
final IntervalMap<Ipv6Resource, Set<Roa>> ipv6Tree = SynchronizedIntervalMap.synchronizedMap(new NestedIntervalMap<Ipv6Resource, Set<Roa>>());

private final RpkiDataProvider rpkiDataProvider;

public RpkiService(final RpkiDataProvider rpkiDataProvider) {
this.rpkiDataProvider = rpkiDataProvider;
loadRoas();
}

@Scheduled(cron = "* /15 * * * *")
private void loadRoas() {
final List<Roa> loadedRoas = rpkiDataProvider.loadRoas();
if (loadedRoas != null && !loadedRoas.isEmpty()){
final List<Roa> roas = loadedRoas.stream()
.filter(roa -> roa.getTrustAnchor() != TrustAnchor.UNSUPPORTED)
.collect(Collectors.toList());

ipv4Tree.clear();
ipv6Tree.clear();

LOGGER.info("downloaded {} roas from rpki", roas.size());
buildTrees(roas, ipv4Tree, ipv6Tree);
}
}

private void buildTrees(final List<Roa> roas,
final NestedIntervalMap<Ipv4Resource, Set<Roa>> ipv4Tree,
final NestedIntervalMap<Ipv6Resource, Set<Roa>> ipv6Tree) {
final IntervalMap<Ipv4Resource, Set<Roa>> ipv4Tree,
final IntervalMap<Ipv6Resource, Set<Roa>> ipv6Tree) {
for (Roa roa : roas) {
if (isIpv4(roa.getPrefix())) {
addRoaToTree(ipv4Tree, Ipv4Resource.parse(roa.getPrefix()), roa);
Expand All @@ -47,7 +61,7 @@ private void buildTrees(final List<Roa> roas,
}
}

private <T extends IpInterval<T>> void addRoaToTree(final NestedIntervalMap<T, Set<Roa>> tree,
private <T extends IpInterval<T>> void addRoaToTree(final IntervalMap<T, Set<Roa>> tree,
final T prefix,
final Roa roa) {
Set<Roa> roas = CollectionHelper.uniqueResult(tree.findExact(prefix));
Expand Down