Big rewrite - distributed consumers for ingestion / scraping(scalable…

…) - single producer written in c#. Changed from page scraping to rss xml scraping Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours. Simplifies a lot of torrentio to deal with new data
knightcrawler-stremio · Feb 1, 2024 · ab17ef8 · ab17ef8
1 parent 6fb4ddc
commit ab17ef8
Show file tree

Hide file tree

Showing 255 changed files with 18,490 additions and 69,075 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,5 @@
-/.idea
 **/node_modules
-**.env
 .now
-/scraper/manual/examples.js
+.DS_Store
+.idea
+
diff --git a/README.md b/README.md
@@ -6,4 +6,10 @@
 ```
 docker-compose up -d
 ```
-Then open your browser to `127.0.0.1:7000`
+Then open your browser to `127.0.0.1:7000`
+
+If you'd like to enable crawling of RealDebridManager's shared hashlists which will massively boost your database cached entries, 
+enter a readonly github personal access token in 'env/producer.env' as the 'GithubSettings__PAT=<token_here>' value.
+
+You can scale the number of consumers, by changing the consumer deploy replica count in the compose file on line 87. This is currently set to 3.
+If you'd like to adjust the number of concurrent processed ingestions per consumer, thats the job concurrency setting within 'env/consumer.env'.
diff --git a/addon/Dockerfile b/addon/Dockerfile
diff --git a/addon/lib/configuration.js b/addon/lib/configuration.js
diff --git a/addon/lib/namedQueue.js b/addon/lib/namedQueue.js
diff --git a/addon/lib/sort.js b/addon/lib/sort.js
diff --git a/catalogs/.dockerignore b/catalogs/.dockerignore
diff --git a/catalogs/Dockerfile b/catalogs/Dockerfile
diff --git a/catalogs/addon.js b/catalogs/addon.js
diff --git a/catalogs/index.js b/catalogs/index.js