Tutorial rest_framework part cannot be completed using curl

[wibou7]

I performed the first 3 part (https://django-oauth-toolkit.readthedocs.org/en/0.7.0/tutorial/tutorial.html) of the tutorial and I'm now trying to perform the "Django Rest Framework" (https://django-oauth-toolkit.readthedocs.org/en/0.7.0/rest-framework/getting_started.html) but that part cannot be completed as it is due to curl issue.

At step 4 of this part of the tutorial (https://django-oauth-toolkit.readthedocs.org/en/0.7.0/rest-framework/getting_started.html#step-4-get-your-token-and-use-your-api), it says:
At this point we’re ready to request an access_token. Open your shell
curl -X POST -d "grant_type=password&username=<user_name>&password=<password>" http://<client_id>:<client_secret>@localhost:8000/o/token/

The problem here is that "client_id" and "client_secret" aren't alpha-numeric string.
In my case, I got:
client_id: .EW6rNN0pkmfDWFaM.vtlCuMbD-W6RgRJjeASrVK
client_secret: ?LZSZzO5SNWm?ul5!9XGyLFqfI2AZ@??Y_UXWXTnfQ3y;!Q@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB!@Diu3gzk8sSy6t:gW0lMbL!@DxJoBfyRcFXzeYiQlHo

I tried and tried, read bunch of curl documentation, but there doesn't seem to be a way to pass such weird string with curl. Special characters (e.g. ! ? ;) will be interepreted by the shell so need to be escaped... but even though, escaped characters and fake "control" characters (e.g. @ : / ) will confuse the hell out of curl.

I'm not the typical web developer type, I was able to work my way around (see work around below) using perl to URI encode the strings. But this was super frustrating, took me 2 hours and I doubt any beginner web developer would ever be able to do that.

For the record:
OS: Linux CentOS 6.5
curl version: curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Shell (bash) version: GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)

Various tests details

If I try to pass it straight to curl (as seen in the tutorial) I get:
[wbourque@localhost ~]$ curl -X POST -d "grant_type=password&username=wibou7&password=123456" http://.EW6rNN0pkmfDWFaM.vtlCuMbD-W6RgRJjeASrVK:?LZSZzO5SNWm?ul5!9XGyLFqfI2AZ@??Y_UXWXTnfQ3y;!Q@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB!@Diu3gzk8sSy6t:gW0lMbL!@DxJoBfyRcFXzeYiQlHo@127.0.0.1:8000/o/token/
bash: !Q@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB!@Diu3gzk8sSy6t: event not found

Curl documentation suggest putting the URL into double quote (i.e. " ") when the link contains special characters... However, doing so here won't do us much good:
[wbourque@localhost ~]$ curl -X POST -d "grant_type=password&username=wibou7&password=123456" "http://.EW6rNN0pkmfDWFaM.vtlCuMbD-W6RgRJjeASrVK:?LZSZzO5SNWm?ul5!9XGyLFqfI2AZ@??Y_UXWXTnfQ3y;!Q@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB!@Diu3gzk8sSy6t:gW0lMbL!@DxJoBfyRcFXzeYiQlHo@127.0.0.1:8000/o/token/"
bash: !9: event not found

Obviously, the ! characters (and all the others special characters) is interpreted by the shell so it needs to be escaped.
If I try just that:
[wbourque@localhost ~]$ curl -X POST -d "grant_type=password&username=wibou7&password=123456" http://.EW6rNN0pkmfDWFaM.vtlCuMbD-W6RgRJjeASrVK:\?LZSZzO5SNWm\?ul5\!9XGyLFqfI2AZ\@\?\?Y_UXWXTnfQ3y\;\!Q\@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB\!\@Diu3gzk8sSy6t\:gW0lMbL\!\@DxJoBfyRcFXzeYiQlHo@127.0.0.1:8000/o/token/
curl: (6) Couldn't resolve host ''

Ok, host is empty, that's weird..?
Let's try to put all that into double quote, maybe?
[wbourque@localhost ~]$ curl -X POST -d "grant_type=password&username=wbourque&password=123456" "http://.EW6rNN0pkmfDWFaM.vtlCuMbD-W6RgRJjeASrVK:?LZSZzO5SNWm\?ul5\!9XGyLFqfI2AZ\@\?\?Y_UXWXTnfQ3y\;\!Q\@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB\!\@Diu3gzk8sSy6t\:gW0lMbL\!\@DxJoBfyRcFXzeYiQlHo@172.16.180.204:8000/o/token/"
curl: (6) Couldn't resolve host '\'
Ok... obviously curl is just super confused with anything I send him here.

Work around

Ok, what if I use another language (e.g. perl) to URI Encode that part of the URL?
Let's try:

#!/bin/bash

CLIENT=".EW6rNN0pkmfDWFaM.vtlCuMbD-W6RgRJjeASrVK"
SECRET="?LZSZzO5SNWm?ul5!9XGyLFqfI2AZ@??Y_UXWXTnfQ3y;!Q@B2jA3k1YX7QFLPk8Iyiy95ZRUfutlfIpoEnB!@Diu3gzk8sSy6t:gW0lMbL!@DxJoBfyRcFXzeYiQlHo"

value_client="$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);' "$CLIENT")"
value_secret="$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);' "$SECRET")"

curl -X POST -d "grant_type=password&username=wibou7&password=123456" "http://$value_client:$value_secret@127.0.0.1:8000/o/token/"

Now when I execute that, I get:
[wbourque@localhost ~]$ ./bug_that.sh
{"access_token": "j71tyLppbX2kjebvtCH34g2tYzsDms", "token_type": "Bearer", "expires_in": 36000, "refresh_token": "JEfVGq30uaPIFmRhl8bh8yth6cYdPq", "scope": "read write groups"}

IT WORKS!!!

So in conclusion:
This will NEVER work using shell only... The only way seems to use something else to URI encode before calling curl.

[poswald]

I've run into this problem before as well... My solution was to register a new BaseHashGenerator that picks from from oauthlib.common import UNICODE_ASCII_CHARACTER_SET. At this key length, I don't believe the additional entropy of having punctuation in the keys to be worth the cost to developers sanity. I'd suggest that as a default for this project.

The oauthlib.common.generate_token function seems to be a good choice to base the secret generation on as well.

Also as an aside, I think you can issue the Basic auth with a command line like this (sorry, I didn't test it but you get the idea) to put the auth data in the headers instead of the url parameters. Both work but I think this way is preferred:

$ curl -v https://example.com/user/oauth2/token/ \
-X POST \
-u "username:password" \
-d "grant_type= password"

[poswald]

Looks like another reason not to use these chars by default is reported in issue #170. The ; messes with url parsers.

[peterldowns]

I agree that it's strange to allow : and ; in the legal character set for client IDs and secrets. It's true that the specification allows for any character to appear in the ID or secret, but obviously that's not practical (see the discussion above for reasons why – some characters have special meaning in the shell or when making authenticated http(s) requests).

When I implemented client ID / secret generation, I restricted the set of legal characters to the set of characters legal for inclusion in a bearer token, which is defined by the specification:

The syntax for Bearer credentials is as follows:

b64token = 1_( ALPHA / DIGIT / "-" / "." / "_" / "~" / "+" / "/" ) *"="
credentials = "Bearer" 1_SP b64token

Does that seem like a reasonable choice? It's been working fine for me.

In addition, with regards to the issues using the curl request, I suggest that the documentation be updated to change the example curl request from:

curl -X POST -d "grant_type=password&username=<user_name>&password=<password>" http://<client_id>:<client_secret>@localhost:8000/o/token/

to

curl -X POST -d 'grant_type=password&username=<username>&password=<password>' --user '<client_id>:<client_secret>' 'http://localhost:8000/o/token/'

which is semantically equivalent to the current solution but removes any issues with shell expansion. This will only break in cases where the client_id contains a :, which the HTTP Authentication Specification disallows. From section 2 of that document:

To receive authorization, the client sends the userid and password,
separated by a single colon (":") character, within a base64 [7]
encoded string in the credentials.

basic-credentials = base64-user-pass
base64-user-pass = <base64 [4] encoding of user-pass, except not limited to 76 char/line>
user-pass = userid ":" password
userid = *<TEXT excluding ":">
password = *TEXT

So when using curl's --user flag, everything after the first occurrence of : will be treated as the password. For that reason, I recommend changing the legal charset of client IDs (and for ease of reasoning / remembering, changing the legal charset for client secrets to be the same).

[poswald]

Does that seem like a reasonable choice? It's been working fine for me.

It's a reasonable choice but I don't really see why it would be a better choice than UNICODE_ASCII_CHARACTER_SET which is defined in the underlying oauth library this project uses. Alphanumeric removes any need for having to deal with character encoding in any context.

To be clear, I believe the test suite should use the set of all possible allowed characters however I think the settings out of the box should be alphanumeric. I think the user can override it to whatever they want if that doesn't work for their needs and that should be supported.

As for the recommended curl command line I think you're basically proposing the same thing I did. I said username:password instead of client_id:client_secret because that's what Basic auth calls them. So yeah, I'm ok with that.

[peterldowns]

@poswald +1 on from oauthlib.common import UNICODE_ASCII_CHARACTER_SET and oauthlib.common.generate_token. I didn't read your response as carefully as I should have before posting my own, sorry to duplicate your recommendation.

[Chitrank-Dixit]

also , I am confused with oauth implementation on front end ,
because before I was able to implement this using

'use strict';

app.factory('oauth', function ($http, $q) {
  return {
    create_or_refresh_token : function (url, authData) {
      return $http({
        method: 'POST',
        url:   url,
        headers : { "Content-Type": "application/x-www-form-urlencoded"},
        transformRequest: function(obj) {
          var str = [];
          for(var p in obj)
            str.push(encodeURIComponent(p) + "=" + encodeURIComponent(obj[p]));
          return str.join("&");
        },
        data: authData
      });
    }
  }

});

but now I am getting the problem with the above snippet ,

status 401

{
  "error": "invalid_grant",
  "error_description": "Invalid credentials given."
}