Anomaly detection working examples

[kingshukb]

Hey Team,
Could you please help me with the anomaly detection examples as the imports mentioned in the readme page isnt working properly for me: https://github.com/unit8co/darts#anomaly-detection . please help. thanks.

Also i am not able to install the latest release v0.24 of darts, getting the following error:

pip install darts==0.24.0.

ERROR: Could not find a version that satisfies the requirement darts==0.24.0 (from versions: 0.0.1, 0.0.1.post1, 0.6.1) ERROR: No matching distribution found for darts==0.24.0

pip freeze output:

absl-py==1.4.0
appnope @ file:///opt/concourse/worker/volumes/live/8b5bd665-fe9a-4b43-55ab-604475fcb838/volume/appnope_1606859465017/work
argon2-cffi @ file:///opt/concourse/worker/volumes/live/16540d95-8806-477c-6cbb-b0e3b6176d98/volume/argon2-cffi_1613037494634/work
async-generator==1.10
attrs @ file:///opt/conda/conda-bld/attrs_1642510447205/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
cachetools==4.2.4
certifi==2021.5.30
cffi @ file:///opt/concourse/worker/volumes/live/6ed3f91b-28f6-4d58-6a9b-7a5e864c6554/volume/cffi_1625814710543/work
charset-normalizer==2.0.12
cmdstanpy==0.4.0
convertdate==2.3.2
cramjam==2.5.0
cycler==0.11.0
Cython==0.29.17
darts==0.6.1
dataclasses==0.8
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
entrypoints==0.3
ephem==4.1.4
fastparquet==0.8.0
fbprophet==0.6
fsspec==2022.1.0
google-auth==1.35.0
google-auth-oauthlib==0.4.6
grpcio==1.48.2
holidays==0.10.2
idna==3.4
importlib-metadata==4.8.3
ipykernel @ file:///opt/concourse/worker/volumes/live/43eaf185-e9cd-4e78-5afa-c19accf1339d/volume/ipykernel_1596206683810/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython==7.15.0
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
jedi==0.17.0
Jinja2 @ file:///opt/conda/conda-bld/jinja2_1647436528585/work
joblib==1.1.1
jsonschema==3.0.2
jupyter-client @ file:///opt/conda/conda-bld/jupyter_client_1643638337975/work
jupyter-core @ file:///opt/concourse/worker/volumes/live/77bbbaa9-1646-4473-6813-2e409c0843a8/volume/jupyter_core_1633420134849/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
kiwisolver==1.3.1
korean-lunar-calendar==0.3.1
LunarCalendar==0.0.9
Markdown==3.3.7
MarkupSafe @ file:///opt/concourse/worker/volumes/live/16a820f0-3874-45b4-6b79-5313238ead21/volume/markupsafe_1621528157387/work
matplotlib==3.2.2
mistune==0.8.4
nbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/work
nbconvert @ file:///opt/concourse/worker/volumes/live/83354520-8b79-4a5c-771c-53c9f422edfc/volume/nbconvert_1601914843271/work
nbformat @ file:///tmp/build/80754af9/nbformat_1617383369282/work
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
notebook @ file:///opt/concourse/worker/volumes/live/09cd0e10-b843-4b15-417b-b731d09aaa95/volume/notebook_1629205797948/work
numpy==1.19.0
oauthlib==3.2.2
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
pandas==1.1.0
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
patsy==0.5.3
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==8.4.0
pmdarima==1.6.1
prometheus-client @ file:///opt/conda/conda-bld/prometheus_client_1643788673601/work
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1633440160888/work
protobuf==3.19.6
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pyarrow==6.0.1
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments @ file:///opt/conda/conda-bld/pygments_1644249106324/work
PyMeeus==0.5.12
pyparsing @ file:///tmp/build/80754af9/pyparsing_1635766073266/work
pyrsistent @ file:///opt/concourse/worker/volumes/live/d7ed6755-1956-4a16-5499-c03638db758d/volume/pyrsistent_1600141722278/work
pystan==2.19.1.1
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pytz==2023.3
pyzmq @ file:///opt/concourse/worker/volumes/live/9d7ac74a-c2e0-4f27-6bc8-960d9f4f8edf/volume/pyzmq_1628276025927/work
requests==2.27.1
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn==0.23.2
scipy==1.5.0
Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work
setuptools-git==1.2
six @ file:///tmp/build/80754af9/six_1644875935023/work
statsmodels==0.11.1
tensorboard==2.4.0
tensorboard-plugin-wit==1.8.1
terminado==0.9.4
testpath @ file:///tmp/build/80754af9/testpath_1624638946665/work
threadpoolctl==3.1.0
torch==1.10.2
torchaudio==0.10.2
torchvision==0.11.3
tornado @ file:///opt/concourse/worker/volumes/live/cd39df49-5c81-4c95-513c-5bb83298b7ef/volume/tornado_1606942310103/work
tqdm==4.46.1
traitlets @ file:///opt/concourse/worker/volumes/live/ab4b852a-8753-4f70-6083-433627bb8c3b/volume/traitlets_1632761183919/work
typing_extensions==4.1.1
urllib3==1.26.16
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
Werkzeug==2.0.3
zipp==3.6.0

pip version: 21.2.2
I am using conda environment.

Thanks

[madtoinou]

Hi @kingshukb,

The problem indeed come from the install as the anomaly detection module was introduced with release 0.23.0.

Can you try upgrading your pip version (ideally in the virtual environment to protect your system)?

As for the docker image, catboost changed their dependencies and depending on your system, it could require some special manipulations to be installed.

[kingshukb]

@madtoinou yes thanks. i was able to get the docker up and running. please help:

1. where can i find some more extensive examples of anomaly detection as per documentations in https://unit8co.github.io/darts/generated_api/darts.ad.html
2. how to reduce the build process, specially the pip install .
3. the jupyter kernel dies when i try to read parquet file by:
   df1 = pd.read_parquet('./0000_part_00.parquet', engine='fastparquet')
   this dies in docker installation though, working fine in host installation

[madtoinou]

1. the example notebook is not released yet, you can find it in PR Example/anomaly detection example #1518 (direct link to the notebook).
2. why would you try to reduce the build process? It should occur once, doesn't it?
3. the docker needs to have access to internet in order to download the dataset, could be that this is not properly configured?

[kingshukb]

Thanks again for quick support.

1. thanks, will check and use this
2. for devs making changes, eg: to requirements( i tried to add the fastparquet==0.8.0 in core.txt to support reading from parquet files in my local system. it took some time to build so was wondering if we can reduce this. it helps in faster development and release.
3. i uploaded the file in container, hence jupyter notebook ideally tried reading from its local fs.

[madtoinou]

Glad to be of help.

2. Since the install process is about pulling packages from pip/conda and darts is already offering various "flavours" in conda (without neural net/pytorch for example, see detailed instruction), I don't think we can do more. If developers want to change the requirements and tweak things, they can cache libraries to speed up things.
3. jupyter kernel's behavior is out of darts control, the line you're quoting does not involve at all.

[kingshukb]

Thanks. we were planning to implement darts's anomaly detection engine in our system. so any tentative time by when we planning to merge pt 1 for devs to playaround more for getting a handson before deep diving into implementation part?

[madtoinou]

The notebook should hopefully be finalized by the end of July and will certainly trigger a new minor version release.

[madtoinou]

Hi @kingshukb,

Sorry for the wait of 1 additional year, an example notebook for Anomaly Detection is finally officially available here thanks to #1477.