Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when threshold values are set via ScrapyCloud settings #374

Closed
marcosmadr opened this issue Dec 21, 2022 · 1 comment · Fixed by #378
Closed

Error when threshold values are set via ScrapyCloud settings #374

marcosmadr opened this issue Dec 21, 2022 · 1 comment · Fixed by #378
Assignees

Comments

@marcosmadr
Copy link

The following error occurs when the spider is running on ScrapyCloud and the monitor threshold value is set via ScrapyCloud settings. The threshold retrieved from ScrapyCloud settings is a string, while the value retrieved from the job stats is an integer, therefore the comparisson fails.

======================================================================
ERROR: Extracted Items Monitor/test_stat_monitor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/app/python/lib/python3.10/site-packages/spidermon/contrib/scrapy/monitors.py", line 144, in test_stat_monitor
    assertion_method(
  File "/usr/local/lib/python3.10/unittest/case.py", line 1248, in assertGreaterEqual
    if not a >= b:
TypeError: '>=' not supported between instances of 'int' and 'str'

If the monitor threshold is set only in the code (and not in ScrapyCloud), the value is retrieved as integer/float and no error is threw.

It seems it started when this PR was merged.

The method used to get the threshold value is get(), while it was using getint() before. It affects only monitors that are inheriting from the BaseStatMonitor class and don't implement a custom get_threshold() method.

@mrwbarg
Copy link
Contributor

mrwbarg commented Jan 12, 2023

Looking into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants