Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tasks API POST request does not accept csrf token #6742

Closed
NeBTrumpf opened this issue Aug 24, 2023 · 12 comments
Closed

Tasks API POST request does not accept csrf token #6742

NeBTrumpf opened this issue Aug 24, 2023 · 12 comments

Comments

@NeBTrumpf
Copy link

NeBTrumpf commented Aug 24, 2023

My actions before raising this issue

Steps to Reproduce (for bugs)

  1. In python script login to cvat and receive login cookies, containing csrftoken
  2. Send /api/tasks POST request, trying to create a new task

Expected Behaviour

I expect a new task to be created and status code 201 to be the response to my POST request.

Current Behaviour

Status code 403 is sent back, with the error in the response's content stating:
"CSRF Failed: CSRF token missing."
Curiously, downloading annotations from an existing task works with my login cookies, so i don't know why the csrf token would be missing when trying to create a new task.

Context

I am trying to automate the creation of new tasks and uploading data to them with the rest-api.

Your Environment

:
  • Docker version: 23.0.1
  • Operating System and version: 18.04.1 Ubuntu (running cvat server), Win10 making request
  • Code example to reproduce problem:
import requests
import json

login_data='{"username":"my_user","email":"","password":"my_password"}'
login_json = json.loads(login_data)
cvat_base_url = "http://myurl.com:8080"

login_response = requests.post(cvat_base_url+'/api/auth/login', json=login_json)

payload = '{"name": "test_rest_create_task","project_id": 18}'
payload = json.loads(payload)
post_response = requests.post(cvat_base_url+'/api/tasks', json=payload, cookies=login_response.cookies)
@zhiltsov-max
Copy link
Contributor

Hi, please check this answer about the authentication parameters.

@NeBTrumpf
Copy link
Author

Hi, thank you for your response.

Sorry to ask again, but I'm not sure I understand your previous answer correctly, so I will ask again. I found a workaround for the previous issue of creating tasks, but the csrf token error is still causing problems. Now I'm trying to upload annotations from my local machine to an existing task. From my understanding there are two ways to do that. Either using the sdk or making requests directly, I've tried both.

Using the sdk, I tried:

with ApiClient(configuration) as api_client:
    try:
        (_, response) = api_client.tasks_api.create_annotations(id=85, filename='annotations_test.zip', format='Segmentation mask 1.1', location='local', use_default_location=False)
        print(response.status, response.headers)
    except exceptions.ApiException as e:
            print("Exception when calling tasks_api.retrieve(): %s\n" % e)

Which yields status code 400 and the HTTP response body: "No such file were uploaded", similar to #6287. I've tried using the cvat-sdk version corresponding to my server version (2.4.5) and using the newest cvat-sdk on pip.

Since using the sdk didn't work for me after some attempts to fix it, I tried making requests directly:

login_cookies = get_login_cookies(USERNAME, PASSWORD, CVAT_URL)
post_response = requests.post(CVAT_URL+'/api/tasks/85/annotations?org=MyOrg&use_default_location=false&location=local&format=Segmentation+mask+1.1&filename=annotations_test.zip&conv_mask_to_poly=false', cookies=login_cookies)

Which yields status code 403 and Response content "CSRF Failed: CSRF token missing."
I also tried using sessionid and csrftoken copied from a browser session, as suggested in your previous response getting the same results.

I can upload the annotations via the UI without problems. The URL for my post request is copied from my browser, from successfully uploading annotations via the UI.

@zhiltsov-max
Copy link
Contributor

zhiltsov-max commented Aug 29, 2023

Hi.

Now I'm trying to upload annotations from my local machine to an existing task.

Unfortunately, as we discovered in the linked issue #6287, annotations now can only be uploaded via the TUS file uploading protocol. Both UI and high-level SDK use this protocol. Manual use via plain requests or low-level SDK is quite complicated with this protocol, so I recommend using the high-level SDK instead:

import os
import sys

from cvat_sdk import make_client
from cvat_sdk.core.helpers import DeferredTqdmProgressReporter

API_KEY = os.getenv("CVAT_API_KEY", "<authorization token>")
API_SESSIONID = os.getenv("CVAT_API_SESSIONID", "<session id>")
API_CSRFTOKEN = os.getenv("CVAT_API_CSRFTOKEN", "<csrf token>")

def main(argv=None):
    assert API_KEY and API_SESSIONID and API_CSRFTOKEN

    with make_client("https://app.cvat.ai", port=443) as client:
        client.api_client.set_default_header("Authorization", f"Token {API_KEY}")
        client.api_client.cookies["sessionid"] = API_SESSIONID
        client.api_client.cookies["csrftoken"] = API_CSRFTOKEN
        # client.organization_slug = ""
        client.config.status_check_period = 2

        task_id = # fill in your task id
        task = client.tasks.retrieve(task_id)

        filename = f"task-{task_id}-cvat_for_images.zip"
        task.export_dataset("CVAT for video 1.1", filename=filename,
            pbar=DeferredTqdmProgressReporter(), include_images=False
        )
        task.import_annotations("CVAT 1.1", filename=filename,
            pbar=DeferredTqdmProgressReporter()
        )

main(sys.argv)

@NeBTrumpf
Copy link
Author

Thank you very much, that kind of worked.

A few issues:
1)
client.tasks.retrieve() takes one parameter, obj_id. It ultimately calls _from_openapi_data() which throws an error: missing 1 required positional argument: 'validation'
I fixed it by editing /core/proxies/model_proxy.py in line 139, adding _validate_outputs=False as a parameter to self.api.retrieve() but that was a shot in the dark...

When uploading annotations to a new task for the first time I get the following error:

Uploading data: 100%| [00:00<00:00, 490kB/s]
Traceback (most recent call last):
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 135, in
upload_annotations(124, 'annotations_test.zip')
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 133, in upload_annotations
task.import_annotations("Segmentation mask 1.1", filename=an_filename, pbar=DeferredTqdmProgressReporter())
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\proxies\tasks.py", line 174, in import_annotations
AnnotationUploader(self.client).upload_file_and_wait(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 290, in upload_file_and_wait
rq_id = json.loads(response.data).get("rq_id")
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\json_init
.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

When I run the same code again, the error turns into:

Uploading data: 100%|| 174k/174k [00:00<00:00, 1.96MB/s]
Traceback (most recent call last):
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 135, in
upload_annotations(124, 'annotations_test.zip')
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 133, in upload_annotations
task.import_annotations("Segmentation mask 1.1", filename=an_filename, pbar=DeferredTqdmProgressReporter())
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\proxies\tasks.py", line 174, in import_annotations
AnnotationUploader(self._client).upload_file_and_wait(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 286, in upload_file_and_wait
response = self.upload_file(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 190, in upload_file
real_filename = self._upload_file_data_with_tus(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 236, in _upload_file_data_with_tus
tus_uploader = self._make_tus_uploader(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 232, in _make_tus_uploader
return _MyTusUploader(client=client, api_client=api_client, **kwargs)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 71, in init
super().init(*_args, **_kwargs)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\tusclient\uploader.py", line 124, in init
self.url = url or self.get_url()
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\tusclient\uploader.py", line 216, in get_url
return self.create_url()
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\tusclient\uploader.py", line 22, in _wrapper
return func(args, **kwargs)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 92, in create_url
resp = self._api_client.rest_client.POST(self.client.url, headers=headers)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\api_client\rest.py", line 256, in POST
return self.request("POST", url,
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\api_client\rest.py", line 211, in request
raise ApiException(http_resp=r)
cvat_sdk.api_client.exceptions.ApiException: Status Code: 409
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Access-Control-Allow-Headers': 'Tus-Resumable,upload-length,upload-metadata,Location,Upload-Offset,content-type', 'Access-Control-Allow-Methods': 'PATCH,HEAD,GET,POST,OPTIONS', 'Access-Control-Allow-Origin': '
', 'Access-Control-Expose-Headers': 'Tus-Resumable,upload-length,upload-metadata,Location,Upload-Offset', 'Allow': 'GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS', 'Cache-Control': 'no-store', 'Content-Length': '36', 'Content-Type': 'application/vnd.cvat+json', 'Cross-Origin-Opener-Policy': 'same-origin', 'Date': 'Tue, 29 Aug 2023 08:15:10 GMT', 'Referrer-Policy': 'same-origin', 'Server': 'Apache', 'Tus-Extension': '', 'Tus-Max-Size': '26843545600', 'Tus-Resumable': '1.0.0', 'Tus-Version': '1.0.0', 'Vary': 'Accept,Origin', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'DENY', 'X-Request-Id': 'f0009d6c-b582-41c7-aa48-3d54f8708b29'})
HTTP response body: b'"File with same name already exists"'

..indicating that I cannot upload the same file twice?

So I tried changing the name of the annotation file, which led to:

Uploading data: 100%|| 174k/174k [00:00<00:00, 647kB/s]
Traceback (most recent call last):
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 135, in
upload_annotations(124, 'annotations_testingf.zip')
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 133, in upload_annotations
task.import_annotations("Segmentation mask 1.1", filename=an_filename, pbar=DeferredTqdmProgressReporter())
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\proxies\tasks.py", line 174, in import_annotations
AnnotationUploader(self._client).upload_file_and_wait(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 286, in upload_file_and_wait
response = self.upload_file(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 194, in upload_file
return self._tus_finish_upload(url, query_params=query_params, fields=fields)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 269, in _tus_finish_upload
File "C:\Users\me\Dokumente\cvat_venv\cvat_utils\cvat_utils.py", line 133, in upload_annotations
task.import_annotations("Segmentation mask 1.1", filename=an_filename, pbar=DeferredTqdmProgressReporter())
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\proxies\tasks.py", line 174, in import_annotations
AnnotationUploader(self._client).upload_file_and_wait(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 286, in upload_file_and_wait
response = self.upload_file(
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 194, in upload_file
return self._tus_finish_upload(url, query_params=query_params, fields=fields)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\uploading.py", line 269, in _tus_finish_upload
expect_status(202, response)
File "C:\Users\me\Dokumente\cvat_venv\lib\site-packages\cvat_sdk\core\helpers.py", line 154, in expect_status
raise exceptions.ApiException(
cvat_sdk.api_client.exceptions.ApiException: Status Code: 201
Reason: Created
HTTP response headers: HTTPHeaderDict({'Allow': 'GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS', 'Content-Length': '0', 'Cross-Origin-Opener-Policy': 'same-origin', 'Date': 'Tue, 29 Aug 2023 08:22:16 GMT', 'Referrer-Policy': 'same-origin', 'Server': 'Apache', 'Vary': 'Accept,Origin', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'DENY', 'X-Request-Id': '807157ea-d95c-4b3a-bfb1-cba99afdedee'})

Which luckily created the annotations in my task! Nice!
One problem, however, remains: It converts my masks to polygons... I tried to find a parameter to change that behaviour but in /core/proxies/tasks.py, which is called by task.import_annotations(), I couldn't find one.
Also I don't really understand why I get the json decoding error and why it fixes itself by calling the same code three times.

@zhiltsov-max
Copy link
Contributor

zhiltsov-max commented Aug 29, 2023

Hi,

The errors about the validation field, failing rq_id = json.loads(response.data).get("rq_id") and auto-fixed errors on the following requests indicate that the SDK/server versions might be mismatching, and the server doesn't include commit from PR #5909. Please check the installed SDK version.

It converts my masks to polygons

There is no high-level API option for this, but you can change this by adding "conv_mask_to_poly": True/False in https://github.com/opencv/cvat/blob/213bcf0b360d0e09fcfee1ff71ccf0d4e7a7b2b0/cvat-sdk/cvat_sdk/core/proxies/tasks.py#L178

@NeBTrumpf
Copy link
Author

Thank you for your help, I will update the server.

@YaoJusheng
Copy link

Hi.

Now I'm trying to upload annotations from my local machine to an existing task.

Unfortunately, as we discovered in the linked issue #6287, annotations now can only be uploaded via the TUS file uploading protocol. Both UI and high-level SDK use this protocol. Manual use via plain requests or low-level SDK is quite complicated with this protocol, so I recommend using the high-level SDK instead:

import os
import sys

from cvat_sdk import make_client
from cvat_sdk.core.helpers import DeferredTqdmProgressReporter

API_KEY = os.getenv("CVAT_API_KEY", "<authorization token>")
API_SESSIONID = os.getenv("CVAT_API_SESSIONID", "<session id>")
API_CSRFTOKEN = os.getenv("CVAT_API_CSRFTOKEN", "<csrf token>")

def main(argv=None):
    assert API_KEY and API_SESSIONID and API_CSRFTOKEN

    with make_client("https://app.cvat.ai", port=443) as client:
        client.api_client.set_default_header("Authorization", f"Token {API_KEY}")
        client.api_client.cookies["sessionid"] = API_SESSIONID
        client.api_client.cookies["csrftoken"] = API_CSRFTOKEN
        # client.organization_slug = ""
        client.config.status_check_period = 2

        task_id = # fill in your task id
        task = client.tasks.retrieve(task_id)

        filename = f"task-{task_id}-cvat_for_images.zip"
        task.export_dataset("CVAT for video 1.1", filename=filename,
            pbar=DeferredTqdmProgressReporter(), include_images=False
        )
        task.import_annotations("CVAT 1.1", filename=filename,
            pbar=DeferredTqdmProgressReporter()
        )

main(sys.argv)

How to solve the 403 problem with Rest API requests?

I tried adding X-CSRFTOKEN to the headers, this usually works but not always.

@zhiltsov-max
Copy link
Contributor

@YaoJusheng, hi, we need some information about how to obtain such an error. To our knowledge, it can be some server-side firewall problem or (corporate) proxy problem, so these are the first to be checked. If you can provide additional information, it will be possible for us to try help you.

@YaoJusheng
Copy link

@YaoJusheng, hi, we need some information about how to obtain such an error. To our knowledge, it can be some server-side firewall problem or (corporate) proxy problem, so these are the first to be checked. If you can provide additional information, it will be possible for us to try help you.

Similar to the example code above:

import requests
import json

login_data='{"username":"admin_user","email":"", "password":"admin_password"}'
login_json = json.loads(login_data)
cvat_base_url = "http://cvat-server:8080"

login_response = requests.post(cvat_base_url+'/api/auth/login', json=login_json)
data={"role": "worker", "email": "user@example.com"}

response = requests.post(
       cvat_base_url+'/api/invitations?org_id=1', 
       data=data, 
       headers={
           "accept": "application/vnd.cvat+json",
           "content-type": "application/json",
           "Authorization": "Token " + login_response.json().get("key", ""), 
           "X-CSRFTOKEN": login_response.cookies.get("csrftoken", "")
       },
       cookies=login_response.cookies
)

Result :

403-Forbidden-{"detail":"You do not have permission to perform this action."}

Environmental information:
cvat version: 2.12.0
Linux: ubuntu20.04 container
net: The request client shares a docker network with cvat

@zhiltsov-max
Copy link
Contributor

@YaoJusheng , are you sure the problem is with the authorization method? Is it possible that this user actually has no rights to invite users into the org? If the problem persists, consider switching to just the basic auth in each request.

@YaoJusheng
Copy link

@zhiltsov-max
This user is a super administrator and has all permissions. When using the above example method to initiate a post request, a 403 error will often occur.
My solution is as mentioned in the reply. Re-authentication after each request fails. In most cases will be resolved.

Is there a more reliable solution without repeated login verification?

@zhiltsov-max
Copy link
Contributor

@YaoJusheng,

My solution is as mentioned in the reply. Re-authentication after each request fails. In most cases will be resolved.

That's very nice to see you've found a workaround.

Is there a more reliable solution without repeated login verification?

We need more information about the problem. If you see the UI is working without problems, please try to find the difference in the requests between the UI and SDK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants