chore: Update cicd, Allow pypi publish, move to pyproject.toml

[duttonw]

• Updated Readme to have dependancy table like xloader
• Updated CICD to run 2.9, 2.10 and 2.11, plus optional master ckan
• Added CICD Pypi publish (based off xloader)
• Replaced setup.py to pyproject.toml config

[duttonw]

Hi @amercader or @wardi

In qld-gov-au we have a non-privileged pypi account added which lets dependabot provide us updates. (since 3 years ago).

Does the CKAN org have similar?

https://github.com/organizations/ckan/settings/secrets/dependabot
PYTHON_INDEX_PYPI_ORG_USERNAME
PYTHON_INDEX_PYPI_ORG_PASSWORD

or if you have them in different names do let me know to update or if its via an api key.

[duttonw]

todo: fix tests when swapping from: okfn container image to ckan image for 2.9;

https://github.com/ckan/ckanext-validation/actions/runs/12245486310?pr=103

131 passed, 4 failed
Test failures:
  test_resource_validation_wrong_format: ckanext.validation.tests.test_logic.TestResourceValidationRun
AssertionError: assert 'Unsupported resource format' in '<ExceptionInfo ValidationError(None) tblen=4>'
 +  where '<ExceptionInfo ValidationError(None) tblen=4>' = str(<ExceptionInfo ValidationError(None) tblen=4>)
self = <ckanext.validation.tests.test_logic.TestResourceValidationRun object at 0x7f54417ef2e0>
def test_resource_validation_wrong_format(self):

    resource = factories.Resource(format=&quot;pdf&quot;)

    with pytest.raises(t.ValidationError) as e:

        call_action(&quot;resource_validation_run&quot;, resource_id=resource[&quot;id&quot;])

    
  

>       assert "Unsupported resource format" in str(e)
E       AssertionError: assert 'Unsupported resource format' in '<ExceptionInfo ValidationError(None) tblen=4>'
E        +  where '<ExceptionInfo ValidationError(None) tblen=4>' = str(<ExceptionInfo ValidationError(None) tblen=4>)


ckanext/validation/tests/test_logic.py:50: AssertionError

  test_resource_validation_no_url_or_upload: ckanext.validation.tests.test_logic.TestResourceValidationRun
AssertionError: assert 'Resource must have a valid URL' in '<ExceptionInfo ValidationError(None) tblen=4>'
 +  where '<ExceptionInfo ValidationError(None) tblen=4>' = str(<ExceptionInfo ValidationError(None) tblen=4>)
self = <ckanext.validation.tests.test_logic.TestResourceValidationRun object at 0x7f54417ef820>
def test_resource_validation_no_url_or_upload(self):

    resource = factories.Resource(url=&quot;&quot;, format=&quot;csv&quot;)

    with pytest.raises(t.ValidationError) as e:

        call_action(&quot;resource_validation_run&quot;, resource_id=resource[&quot;id&quot;])

    
  

>       assert "Resource must have a valid URL" in str(e)
E       AssertionError: assert 'Resource must have a valid URL' in '<ExceptionInfo ValidationError(None) tblen=4>'
E        +  where '<ExceptionInfo ValidationError(None) tblen=4>' = str(<ExceptionInfo ValidationError(None) tblen=4>)


ckanext/validation/tests/test_logic.py:60: AssertionError

  test_validation_fails_on_upload: ckanext.validation.tests.test_logic.TestResourceValidationOnCreate
AssertionError: assert 'missing-cell' in '<ExceptionInfo ValidationError(None) tblen=5>'
 +  where '<ExceptionInfo ValidationError(None) tblen=5>' = str(<ExceptionInfo ValidationError(None) tblen=5>)
self = <ckanext.validation.tests.test_logic.TestResourceValidationOnCreate object at 0x7f5440567790>
@pytest.mark.usefixtures(&quot;mock_uploads&quot;)
def test_validation_fails_on_upload(self):

    invalid_file = get_mock_file(INVALID_CSV)

    mock_upload = MockFieldStorage(invalid_file, &quot;invalid.csv&quot;)

    dataset = factories.Dataset()

    with pytest.raises(t.ValidationError) as e:

        call_action(
            &quot;resource_create&quot;,
            package_id=dataset[&quot;id&quot;],
            format=&quot;CSV&quot;,
            upload=mock_upload,
        )

    assert &quot;validation&quot; in e.value.error_dict

    
  

>       assert "missing-cell" in str(e)
E       AssertionError: assert 'missing-cell' in '<ExceptionInfo ValidationError(None) tblen=5>'
E        +  where '<ExceptionInfo ValidationError(None) tblen=5>' = str(<ExceptionInfo ValidationError(None) tblen=5>)


ckanext/validation/tests/test_logic.py:533: AssertionError

  test_validation_fails_on_upload: ckanext.validation.tests.test_logic.TestResourceValidationOnUpdate
AssertionError: assert 'missing-cell' in '<ExceptionInfo ValidationError(None) tblen=5>'
 +  where '<ExceptionInfo ValidationError(None) tblen=5>' = str(<ExceptionInfo ValidationError(None) tblen=5>)
self = <ckanext.validation.tests.test_logic.TestResourceValidationOnUpdate object at 0x7f543feef5b0>
@pytest.mark.usefixtures(&quot;mock_uploads&quot;)
def test_validation_fails_on_upload(self):

    dataset = factories.Dataset(resources=[{&quot;url&quot;: &quot;https://example.com/data.csv&quot;}])

    invalid_file = get_mock_file(INVALID_CSV)

    mock_upload = MockFieldStorage(invalid_file, &quot;invalid.csv&quot;)

    invalid_stream = io.BufferedReader(io.BytesIO(INVALID_CSV.encode(&apos;utf8&apos;)))

    with mock.patch(&quot;io.open&quot;, return_value=invalid_stream):

        with pytest.raises(t.ValidationError) as e:

            call_action(
                &quot;resource_update&quot;,
                id=dataset[&quot;resources&quot;][0][&quot;id&quot;],
                format=&quot;CSV&quot;,
                upload=mock_upload,
            )

    assert &quot;validation&quot; in e.value.error_dict

    
  

>       assert "missing-cell" in str(e)
E       AssertionError: assert 'missing-cell' in '<ExceptionInfo ValidationError(None) tblen=5>'
E        +  where '<ExceptionInfo ValidationError(None) tblen=5>' = str(<ExceptionInfo ValidationError(None) tblen=5>)


ckanext/validation/tests/test_logic.py:629: AssertionError

This test report was produced by the test-summary action.  Made with ❤️ in Cambridge.

[ThrawnCA]

Better to just delete it than comment it out.

[ThrawnCA]

Looks like our Jinja version can't handle Python 3.10, which moved Mapping from collections to collections.abc

[duttonw]

closing out since toml was released by 2.9. The work now on this is 2.10, 2.11 uplift from the qld-gov-au fork and will be handled by a different pr.