Skip to content

Commit

Permalink
Change format of content of _validation field (#425)
Browse files Browse the repository at this point in the history
The content of _validation field must to be the string representation of a Python dict instead defaultdict when SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS setting is True.
  • Loading branch information
rochamatcomp committed Feb 16, 2024
1 parent 5214606 commit 12f63bc
Show file tree
Hide file tree
Showing 4 changed files with 8 additions and 4 deletions.
3 changes: 1 addition & 2 deletions docs/source/getting-started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -478,8 +478,7 @@ The resulted item will look like this:
.. code-block:: js
{
'_validation': defaultdict(
<class 'list'>, {'author_url': ['Invalid URL']}),
'_validation': {'author_url': ['Invalid URL']},
'author': 'Mark Twain',
'author_url': 'not_a_valid_url',
'quote': 'Never tell the truth to people who are not worthy of it.',
Expand Down
2 changes: 1 addition & 1 deletion docs/source/item-validation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ You can change the name of the field by assigning a name to `SPIDERMON_VALIDATIO
.. code-block:: js
{
'_validation': defaultdict(<class 'list'>, {'author_url': ['Invalid URL']}),
'_validation': {'author_url': ['Invalid URL']},
'author': 'C.S. Lewis',
'author_url': 'invalid_url',
'quote': 'Some day you will be old enough to start reading fairy tales '
Expand Down
5 changes: 4 additions & 1 deletion spidermon/contrib/scrapy/pipelines.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,13 +129,16 @@ def find_validators(self, item):

def _add_errors_to_item(self, item: ItemAdapter, errors: Dict[str, str]):
errors_field_instance = get_nested_attribute(item, self.errors_field)

if errors_field_instance is None:
errors_field_instance = defaultdict(list)
set_nested_attribute(item, self.errors_field, errors_field_instance)

for field_name, messages in errors.items():
errors_field_instance[field_name] += messages

# change defaultdict to dict for errors_field_instance
set_nested_attribute(item, self.errors_field, dict(errors_field_instance))

def _drop_item(self, item, errors):
"""
This method drops the item after detecting validation errors. Note
Expand Down
2 changes: 2 additions & 0 deletions tests/contrib/validation/test_item_validation_pipeline.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from collections import defaultdict
from dataclasses import dataclass

import pytest
Expand Down Expand Up @@ -87,6 +88,7 @@ def test_jsonschema_validation(dummy_schema):
result_item = pipeline.process_item(dict(item), None)
assert item != result_item
assert "_validation" in result_item
assert not isinstance(result_item["_validation"], defaultdict)
assert result_item["_validation"]["foo"] == ["Missing required field"]


Expand Down

0 comments on commit 12f63bc

Please sign in to comment.