Unit tests and linting fixes.

… configuration istead of full python model names.

model_path parameter

…e.training.model refers to the full module path of the Model class(not the Enum key).

Config fix for Model

Country and Host features. Stratified sampling(parameter 'max_samples_per_host'). Support for nested features in JSON parser(geoip feature fix).

Activity 1.1 pipeline separation

License check

Fix for 'created_at' postgress issue.

Fill in prediction columns if no model

AttackDetection task. Anomaly and challenge thresholds. Anomaly metrics directly to Prometheus.

* Model logging. Spark-iforest-2.4.0.99.jar

* Preprocessing pipeline fix. Categorical features fixes for new pipeline. SendToKafka task. New kafka producer from spark dataframe.

* Preprocessing and Prediction pipelines tested with real logs and production kafka. Feature parsing fix. Carbon copy parameter for sending predictions to a separate client topic. Model path support for ServiceProvider.

* Working pipelines: preprocessing, predit, postprocessing.

* Merge branch 'develop' into model_logging

# Conflicts:
#	src/baskerville/models/pipeline_tasks/tasks.py

* Redis fix for timestamp.

* Redis fix2 for timestamp.

* Fix rs cache issue (#33)

* RefreshCache(config)

* created_at fix

* RequestSet cache is commented out since it's still failing.

* Extra comment removed from config validation.

* Removed F.broadcast from request_cache

* Fix for request cache update. Removing 'first_ever_request' column.

* Debug pdb line removed.

* Additional fix for cache update_self()

* Unit test and linting fixes.

* Linting fix.

* Original target column (#35)

Added `target_original` column

* New training pipeline

* Unit test fix.

* Fix for 'target_original' in update cache.

Co-authored-by: Maria Karanasou <karanasou@gmail.com>

…IP. (#37)

* Banjax reports. 
* Register banjax metrics only with -e argument.

* Redis password and config.

* Redis ip in the config. Additional banjax metrics.

* Using target instead of target_original in attack metrics.
* Traffic light metric
* Count anomalies metric removed. It should be directly queried from postgres.
* Redis merge warning for missing data.
* Sliding window for attack detection.
* Cache loading fix (wrong old isdir)
* Low rate attack detection
* White list
* Training loading schema fix.
* Casting features values fix

* Feature class 'host_country'

* Sliding window processing is disabled when attack_threshold is 0

All spark `persist` calls are commented out with a prefix `ppp`

* Low rate attack with 2 conditions

Greenhost (#62)

* IPCache class added
* no counts in logging
* Full cache warning instead of exception.
* IP cache persisting
* logs message corrected with total
* Thread safe singleton ip cache
* White list fix. Challenged default zero fix.
* Banned/passed reports from banjax
* Extra logging removed
* Two ip caches: passed and pending
* Saving ip_passed cache in the file.
* start report consumer even without -e
* Banjax thread moved into AttackDetectin task
* The new version of spark-iforest
* White list ips optimized.
* White list hosts added.
* Host white listing is moved to send_challenge()
* Spark standalone 3nodes deployment procedure.
* ats_log_schema.json added. IForest removed from requirements.txt. spark_standalone.md updated.
* count() is deleted everywhere
* Spark secret. Count() removed from postprocessing.
* spark encryption configuration added
* spark ssl for ui, standalone and history config.
* Sliding window in postprocesing is optional. Set sliding_window config param to zero to disable sliding_window(defaut is also zero now)
* Markdown dependency removed due to the confict with pdoc.
* model interpretation helpers and functional shapley with anomaly model
* improve shap calculation's udf
* OriginIPs class for getting the list of original ips for whitelisting in postprocessing pipeline.
* Hot model upgrade (#65)

notebook commits

Processing 'ip_passsed_challege2' from banjax

* reuse_dataframe parameter in model.predict()

* reuse spark workers to True

…ollect. (#76)

* S3 support
* memory only request_cahce
* second URL for OriginIPs

…s. (#81)