Why do we need to pass an URL when uploading a resource?

[vitorbaptista]

According to https://github.com/ckan/ckanapi docs, we should be able to upload a new resource with:

import ckanapi

mysite = ckanapi.RemoteCKAN('http://myckan.example.com',
    apikey='real-key',
    user_agent='ckanapiexample/1.0 (+http://example.com/my/website)')
mysite.action.resource_create(
    package_id='my-dataset-with-files',
    upload=open('/path/to/file/to/upload.csv'))

This doesn't work, because resource_create requires an url parameter. Blaming the create.py file, I found out that this validation was added by 072b62b5 (which, curiously, was made by me 😅 ). It is quite old (Apr/2014).

Probably the correct solution at the time would be to fix the docs, which says that the URL is required.

[CarlQLange]

this is really broken for me. Uploading resources simply doesn't work. If I supply no url, CKAN cracks out on me and won't let me upload my resource. If I supply a url like ' ', the view resource button goes to 'http://' and doesn't work. If I supply a url like '', it's treated as not existing.

I simply can't believe this is a bug in production CKAN right now. Uploading files is surely an important feature.

[wardi]

@CarlQLange are you saying that the change in ckan/ckanapi#74 doesn't work? Doesn't an uploaded file cause the url value to be replaced for you?

[mouillerart]

No it doesn’t – it experienced it with version 2.4.1 and 2.5.1 as well.

De : Ian Ward [mailto:notifications@github.com]
Envoyé : jeudi 21 janvier 2016 17:08
À : ckan/ckan
Cc : MOUILLERON Marc-Antoine IMT/OLPS
Objet : Re: [ckan] Why do we need to pass an URL when uploading a resource? (#2769)

@CarlQLangehttps://github.com/CarlQLange are you saying that the change in ckan/ckanapi#74ckan/ckanapi#74 doesn't work? Doesn't an uploaded file cause the url value to be replaced for you?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2769#issuecomment-173618946.

---

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

[CarlQLange]

I remembered the fix for this. I'm using requests:

r = requests.post('%s/api/action/resource_create' % (API_HOST),
        data={
            "package_id": dataset_name,
            "type": "file.upload",
            "url": "anything",
            "name": resource_name,
            "format": resource_format
        },
        headers={"Authorization": API_KEY},
        files=[('upload', file(resource_path))]) #HERE

The trick is to be sure the name of the uploaded file is 'upload', and only that. Anything else will cause this to break. Sorry about my saltiness, just under deadline and having something simple break :)

[wardi]

@CarlQLange don't worry about it.

You might want to give ckanapi a try (when not under deadline). It's a very small library and it gives you real python exceptions when there are errors in your requests.

[CarlQLange]

Just in case I forget how I fixed this in future:

r = requests.post('%s/api/action/resource_create' % (API_HOST),
        data={
            "package_id": dataset_name,
            "type": "file.upload",
            "url": "",
            "name": resource_name,
            "format": resource_format
        },
        headers={"Authorization": API_KEY},
        files={'upload': (resource_name, file(resource_path))})

The tuple in files is super important. You can't just do {'upload': (file(resource_path))} because it'll call the file 'upload' and that's no good.

[mouillerart]

Hi there,

Actually it would appear that the correct parameters are

r = requests.post('%s/api/action/resource_create' % (API_HOST),

    data={

        "package_id": dataset_name,

        "url_type": " upload",

        "url": "anything",

        "name": resource_name,

        "format": resource_format

    },

    headers={"Authorization": API_KEY},

    files=[('upload', file(resource_path))]) #HERE

I just tested it from Java client, with dummyfile.txt for upload. The API generates a new URL for download such as
http://localhost:5000/dataset/8bed5e50-8a17-4a44-b95c-e2d0fd3bec36/resource/a769204a-4c7c-4154-a50c-40145a7fc7e7/download/dummyfile.txt"

Marc-Antoine

De : Carl Lange [mailto:notifications@github.com]
Envoyé : jeudi 21 janvier 2016 17:31
À : ckan/ckan
Cc : MOUILLERON Marc-Antoine IMT/OLPS
Objet : Re: [ckan] Why do we need to pass an URL when uploading a resource? (#2769)

I remembered the fix for this. I'm using requests:

r = requests.post('%s/api/action/resource_create' % (API_HOST),

    data={

        "package_id": dataset_name,

        "type": "file.upload",

        "url": "anything",

        "name": resource_name,

        "format": resource_format

    },

    headers={"Authorization": API_KEY},

    files=[('upload', file(resource_path))]) #HERE

The trick is to be sure the name of the uploaded file is 'upload', and only that. Anything else will cause this to break. Sorry about my saltiness, just under deadline and having something simply break :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2769#issuecomment-173625344.

---

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

[wardi]

fixed by #2844