Skip to content

Commit

Permalink
Applied formatting + removed Ignite #96
Browse files Browse the repository at this point in the history
Signed-off-by: Julien Nioche <julien@digitalpebble.com>
  • Loading branch information
jnioche committed Sep 13, 2024
1 parent bd4e209 commit b7a7d44
Show file tree
Hide file tree
Showing 11 changed files with 21,959 additions and 20,934 deletions.
3,365 changes: 1,982 additions & 1,383 deletions API/src/main/java/crawlercommons/urlfrontier/URLFrontierGrpc.java

Large diffs are not rendered by default.

38,542 changes: 19,968 additions & 18,574 deletions API/src/main/java/crawlercommons/urlfrontier/Urlfrontier.java

Large diffs are not rendered by default.

7 changes: 2 additions & 5 deletions service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,9 @@
[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/crawlercommons/url-frontier)](https://hub.docker.com/r/crawlercommons/url-frontier)
[![Docker Pulls](https://img.shields.io/docker/pulls/crawlercommons/url-frontier)](https://hub.docker.com/r/crawlercommons/url-frontier)

Implementations of the URL Frontier Service. There are currently 3 implementations available:
Implementations of the URL Frontier Service. There are currently 2 implementations available:
- a simple memory-based which was used primarily for testing
- the default one which is scalable, persistent and is based on [RocksDB](https://rocksdb.org/)
- a persistent and distributed one based on [Ignite](https://ignite.apache.org/)

The Ignite implementation is still in beta mode.

Web crawlers can connect to it using the gRPC code generated from the API. There is also a simple client available
which can do basic interactions with a Frontier.
Expand Down Expand Up @@ -37,7 +34,7 @@ the call above can have the following equivalent without the config file:

If no path is set explicitly for RocksDB, the default value _./rocksdb_ will be used.

For implementation supporting a cluster mode, like the Ignite one, it is required to use the parameter `-h xxx.xxx.xxx.xxx` with the private IP or hostname
For implementation supporting a cluster mode, it is required to use the parameter `-h xxx.xxx.xxx.xxx` with the private IP or hostname
on which it is running so that it can report its location with the heartbeat.

## Logging configuration
Expand Down
18 changes: 1 addition & 17 deletions service/config.ini
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ implementation = crawlercommons.urlfrontier.service.rocksdb.RocksDBService
# the default value is the number of processors / 4
# read.thread.num = 3

rocksdb.path = /data/crawl/rocksdb
rocksdb.path = ./rocksdb
# rocksdb.purge = true

# advanced rocksdb tuning
Expand All @@ -21,22 +21,6 @@ rocksdb.max_background_jobs = 4
rocksdb.max_subcompactions = 2
rocksdb.max_bytes_for_level_base = 536870912

#implementation = crawlercommons.urlfrontier.service.ignite.IgniteService

# Needed for the Ignite based frontiers to form a cluster
# ignite.seed.address = xxx.xxx.xxx.xxx

ignite.path = /data/crawl/ignite
ignite.workdir = /data/crawl/ignite
ignite.index = /data/crawl/lucene
# ignite.purge = true

ignite.backups = 3
# frequency in sec of when the frontiers should send a heartbeat
ignite.frontiers.heartbeat = 60
# ttl of hearbeats in sec
ignite.frontiers.ttl = 120




Expand Down
14 changes: 0 additions & 14 deletions service/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@

<properties>
<prometheus.version>0.16.0</prometheus.version>
<ignite.version>2.14.0</ignite.version>
<lucene.version>9.4.0</lucene.version>
<rocksdb.version>7.6.0</rocksdb.version>
</properties>

Expand Down Expand Up @@ -94,18 +92,6 @@
<version>${rocksdb.version}</version>
</dependency>

<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-core</artifactId>
<version>${ignite.version}</version>
</dependency>

<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>${lucene.version}</version>
</dependency>

<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
Expand Down

This file was deleted.

Loading

0 comments on commit b7a7d44

Please sign in to comment.