datapusher submit ... -> all resources, also unsuitable formats

[ChristianF88]

Hi everyone,

thanks for developing datapusher-plus! Great job!

I have set up a docker container as a drop in replacement for my "old" datapusher container using dp+ version 0.16.4. After a little debugging everything everything works fine! For testing I want to re-submit all my resource, to identify possible issues, before using dp+ in production.

When resubmitting single package I experienced some unexpected behavior. Instead of only submitting files in formats specified in:

CKAN__DATAPUSHER__FORMATS="csv xls xlsx xlsm xlsb tsv ods application/csv application/vnd.ms-excel application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"

All resources are passed to datapusher, causing a lot of unnecessary computation and errors. The package contains mainly incompatible files:

I checked the formats and they're set correctly in CKAN.

How come all files are submitted? Is this by design or is it a bug?

[ChristianF88]

I believe I have found the answer. The submitting process is still controlled by the the standard datapusher implementation included in ckan, where simply all resources are submitted. I am using ckan 2.9.11 (ckan/ckanext/datapusher/cli.py).

[tino097]

hey @ChristianF88 , yeah, the DP+ 0.x and 1.0.x versions are still dependent of the datapusher ckan core implementation. But we are planning to move completely away from it. I would suggest to try the dev-1.0 branch as the DP+ is moving to work as a simple extension and you wouldn't need to spin up a separate container for Datapusher

[ChristianF88]

Hi @tino097,
thanks for you feedback. Would it also be possible to trigger specific jobs with a GET request to the endpoint /job/{resource-id} of the dp+ container? Or for a POST request, what would the data be?

[ChristianF88]

I found a way, it's a bit hacky... So if there's a proper API / Endpoint for this, I'd love to get a hint on where I can find the documentation. :) Cheers!

[tino097]

Those endpoints are part of old unmaintained extension https://github.com/ckan/ckan-service-provider/tree/master

We are removing this from DP and all job related actions should be done via the ckan jobs command