834 additional iati checks #856

edugomez · 2017-10-03T16:25:30Z

…:tag

…penag_tag.feature

…et_errors

Bjwebb · 2017-10-09T11:56:55Z

cove_iati/lib/iati.py

@@ -11,7 +11,7 @@
 from cove.lib.exceptions import CoveInputDataError, UnrecognisedFileTypeXML


-def common_checks_context_iati(context, upload_dir, data_file, file_type, api=False):
+def common_checks_context_iati(context, upload_dir, data_file, file_type, api=False, openag=False, orgids=False):


I think orgids should probably be on by default - I think we want to see it in the web interface.

I agree that we'd like it in the web interface, but I don't want the additional overhead (UI design, etc) to slow down anything else in this sprint. So, if it's easy/cheap/free to add to the web UI, amazing, but if there's any real work involved, let's leave it and prioritise it against other work in non-sprint or our next IATI sprint.

It won't be too difficult to add to the web interface, but I'd rather get this merged and leave that for another sprint or for the non-sprint board. If we agree on that, then I think it should be off by default.

Ah okay, makes sense.

Bjwebb · 2017-10-09T14:01:49Z

cove_iati/rulesets/iati_openag_ruleset/openag_location.feature

+    Given an IATI activity
+     Then at least one `location` element is expected
+
+  Scenario Outline: location.location-id must be present


Why not location/location-id?

As discussed, the scenario outline will be the name of a generated file (with / giving problems in Unix)

Bjwebb · 2017-10-09T14:21:55Z

cove_iati/rulesets/iati_orgids_ruleset/steps/orgids_step_definitions.py

+    for xpath in get_xobjects(context.xml, context.xpath_expression):
+        attr_id = xpath.attrib.get(attribute, '')
+        for prefix in ORGIDS_PREFIXES:
+            if re.match('^%s' % prefix, attr_id):


Since prefixes aren't regexes, would attr_id.startswith(prefix) be clearer here?

Yes, I just forgot that startswith exists

Bjwebb · 2017-10-09T14:27:34Z

cove_iati/rulesets/iati_openag_ruleset/steps/openag_ruleset_step_definitions.py

+
+@then('every `{xpath_expression1}` must include `{xpath_expression2}` element')
+@register_ruleset_errors(['openag'])
+def step_openag_location_id_expected(context, xpath_expression1, xpath_expression2):


Should this function name be more generic?

Bjwebb

Just noticed that this branch leaves the web interface failing for a lot of files, e.g. https://raw.githubusercontent.com/IATI/IATI-Extra-Documentation/version-2.02/en/activity-standard/activity-standard-example-annotated.xml

Looks to be a JSON decode error associated with rulesets. I'd be happy to have a go fixing this tomorrow.

edugomez · 2017-10-09T16:09:26Z

I'd be happy to have a go fixing this tomorrow

@Bjwebb that would be great.

Some context. The problem is that file that it tries to read does not translate to JSON, it contains the following:

_Argument 'element' has incorrect type (expected lxml.etree._Element, got lxml.etree.ElementUnicodeResult)

This is the rule it tries to apply (in https://github.com/OpenDataServices/cove/blob/834-additional-iati-checks/cove_iati/rulesets/iati_standard_v2_ruleset/mandatory_elements.feature):

  Scenario Outline: participating-org.@ref attribute or participating-org.narrative must be present
    Given `participating-org` organisations
    Then either `participating-org/@ref` or `participating-org/narrative/text()` is expected

And processed by this step function https://github.com/OpenDataServices/cove/blob/834-additional-iati-checks/cove_iati/rulesets/iati_standard_v2_ruleset/steps/standard_ruleset_step_definitions.py#L86

Caveat: I have amended the latest commit on this branch, please discard your local branch if you have one already and re-pull the branch.

#856 (comment)

Bjwebb · 2017-10-10T12:14:14Z

@edugomez Since you're around tomorrow, it would be good if you could check my changes before we merge this.

edugomez · 2017-10-11T11:06:43Z

cove_iati/rulesets/utils.py

+    elif hasattr(child_xobj, 'is_text') and child_xobj.is_text:
+        return '{}/text()'.format(tree.getpath(child_xobj.getparent()))
+    else:
+        return tree.getpath(child_xobj)


edugomez added 30 commits September 25, 2017 19:52

[#834] cove_iati: add missing backtick in feature

54a4589

[#834] cove_iati: add initial Open Ag ruleset

23d98d0

[#834] iati_openag: add steps to check for right attributes in openag…

f64c445

…:tag

[#834] iati_openag: rename openag_tag_agrovoc_attributes.feature as o…

1f33927

…penag_tag.feature

[#834] iati_openag: refactor openag steps to add 'And' step to features

6f3e525

[#834] iati_openag: add openag_location.features

c038504

[#834] iati_openag: add org-ids feature

b4e245b

[#834] Add @get_org_ids_prefixes

cddf2bc

[#834] iati_openag: edit opena_orgids.feature

bf8f0fc

[#834] iati_openag: add step for org_ids ref id check

2882cda

Merge branch 'master' into 834-additional-iati-checks

a273960

[#834] Raise requests exception if first request org-ids file fails

549c6a2

[#834] iati rulesets: reuse given steps

aa0ef82

[#834] iati ruleset: refactor actual dates features and steps

ce5943a

[#834]: iati_ruleset: refactor start-date end-date features

f0ba01e

[#834]: iati_ruleset: refactor sector features

101c321

[#834]: iati_ruleset: refactor element valid format feature

9dcd539

[#834] iati_openag: delete ruleset_openag dir from output dir

28f486e

[#834] iati_ruleset: edit error messages

07414ea

[#834] iati_ruleset: add rulesets utils

03d6a79

[#834] iati_ruleset: use register_ruleset_errors decorator

56c0766

[#834] iati_ruleset: rename steps

40fa7ee

[#834] iati_ruleset: edit scenario outlines

edadf3c

[#834] iati_ruleset: check for namespaces defined in data

de2c2b1

[#834] cove_iati: fix tests

2092b07

[#834] cove_iati: add --openag option to iati-cli

bccff32

[#834] iati_ruleset: make @register_ruleset_errors a decorator factory

d28124e

[#834] iati_openag: add test for openag cli output

98f985b

cove_ocds: fix broken test

d0e7c07

[#834] iati_openag: fix bug location-id step

0f81218

edugomez changed the title ~~[WIP] 834 additional iati checks~~ 834 additional iati checks Oct 5, 2017

[#834] iati_orgids: add orgids check option to cli

fbb540a

edugomez force-pushed the 834-additional-iati-checks branch from a0eb69c to fbb540a Compare October 5, 2017 22:12

edugomez added 4 commits October 6, 2017 08:23

Merge branch 'master' into 834-additional-iati-checks

ebf7892

[#834] iati_ruleset: add tests for rulesets utils

b05006c

Merge branch 'master' into 834-additional-iati-checks

3c8e061

[#834] Add orgids_url kwarg to @get_orgids_prefixes

0e4e129

edugomez force-pushed the 834-additional-iati-checks branch from 1ca23c2 to 0e4e129 Compare October 7, 2017 13:16

[#834] iati_ruleset: pass iterable with namespaces to @register_rules…

d06fe21

…et_errors

Bjwebb reviewed Oct 9, 2017

View reviewed changes

edugomez added 2 commits October 9, 2017 15:52

[#834] iati_openag: rename openag ruleset step

4af9641

[#834] iati_orgids: use startwith instead of regex

c054cdc

Bjwebb approved these changes Oct 9, 2017

View reviewed changes

edugomez force-pushed the 834-additional-iati-checks branch 2 times, most recently from 4122967 to d6a2a45 Compare October 9, 2017 15:34

[#834] iati_cli: add more info to README.rst

815fc80

edugomez force-pushed the 834-additional-iati-checks branch from d6a2a45 to 815fc80 Compare October 9, 2017 15:36

Bjwebb requested changes Oct 9, 2017

View reviewed changes

Bjwebb added 2 commits October 10, 2017 11:55

[#834] Fix broken ruleset check

d5250d2

#856 (comment)

[#834] Only sector should be an exclusive or

9653270

Bjwebb approved these changes Oct 10, 2017

View reviewed changes

edugomez commented Oct 11, 2017

View reviewed changes

Merge branch 'master' into 834-additional-iati-checks

dfdd697

edugomez merged commit f36ac9e into master Oct 11, 2017

edugomez deleted the 834-additional-iati-checks branch October 11, 2017 12:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

834 additional iati checks #856

834 additional iati checks #856

edugomez commented Oct 3, 2017 •

edited by Bjwebb

Loading

Bjwebb Oct 9, 2017

robredpath Oct 9, 2017

edugomez Oct 9, 2017

Bjwebb Oct 9, 2017

Bjwebb Oct 9, 2017 •

edited

Loading

edugomez Oct 9, 2017 •

edited

Loading

Bjwebb Oct 9, 2017

edugomez Oct 9, 2017

Bjwebb Oct 9, 2017

edugomez Oct 9, 2017

Bjwebb left a comment

edugomez commented Oct 9, 2017 •

edited

Loading

Bjwebb commented Oct 10, 2017

edugomez Oct 11, 2017

834 additional iati checks #856

834 additional iati checks #856

Conversation

edugomez commented Oct 3, 2017 • edited by Bjwebb Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bjwebb Oct 9, 2017 • edited Loading

Choose a reason for hiding this comment

edugomez Oct 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bjwebb left a comment

Choose a reason for hiding this comment

edugomez commented Oct 9, 2017 • edited Loading

Bjwebb commented Oct 10, 2017

Choose a reason for hiding this comment

edugomez commented Oct 3, 2017 •

edited by Bjwebb

Loading

Bjwebb Oct 9, 2017 •

edited

Loading

edugomez Oct 9, 2017 •

edited

Loading

edugomez commented Oct 9, 2017 •

edited

Loading