Add scope parsing to Scope object

[sirosen]

I'm not sure that it's all perfect, but I wrote a parser and I can't seem to find a valid scope which breaks it, so that seems like a good sign.

I did strongly consider using pyparsing for this. I've had good experiences using pyparsing and it seemed like it might fit this use-case. There were a couple of reasons I decided not to use it here.

1. Overkill. If it turns out that we can write a correct lexer trivially, and a correct parser easily in pure python without defining it as a recursive descent parser.
2. Dependency handling. We'd either add a dependency (possibility of conflicts), or add it as an optional dependency (more complicated codepaths).
3. Globally scoped ParserElement changes. The whitespace sensitivity of pyparsing can be adjusted on a global basis by certain method calls, and I know of at least one other bit of global state (enable_packrat). I didn't like the idea that our parsing behavior could change depending on what a user application might be doing with pyparsing independently.
4. Whitespace-sensitivity is messy. pyparsing defines all whitespace as non-semantic by default. This is fine when it fits your language, but Globus Auth requires that an optional marker be adjacent to the scope string (*foo, but not * foo). It seemed like this might lead to us using Combine() in ways that don't naturally fit our desired grammar.

---

The core of this change is that MutableScope has been renamed to Scope and now has a pair of parsing functions as classmethods: Scope.parse and Scope.deserialize. The first parses any valid scope string to request, meaning it may be a space delimited list of scopes, and the second parses any valid scope string with a single top-level scope.

As an inverse of deserialize(), the stringifier has been renamed to serialize(), and __str__() now just calls serialize().

The parsing itself requires several new internal components. A tokenizer handles converting strings into lists of tokens, e.g. converting foo[bar *baz] into
["foo", "[", "bar", "*", "baz", "]"].
The tokens are small objects pairing strings with a type, which makes it simpler for the parser to handle various cases. The parser is minimally stateful, has no recursion or backtracking, operates with limited lookahead, and is generally meant to be the smallest and simplest thing which can do the job that it does.

The parse result, a Scope, can now represent the true tree-like structure of scopes. Scope.dependencies is a list of child nodes in the tree, a list of Scopes. This has implicitations for Scope.add_dependency in that a Scope is now a valid input. As a result of this introduction, the use of
add_dependency("foo", optional=True) is considered deprecated and now emits a DeprecationWarning. Building dependencies in the current version should be done either with (newly parseable) strings or with Scope objects. i.e. The aforementioned usage can be replaced with either add_dependency("*foo") or
add_dependency(Scope("foo", optional=True)).

Some related changes included here:

• globus_sdk.scopes is now a subpackage (it was getting a bit long)
• the internal type alias for ScopeCollectionType is now kept in _types for simpler universal access within the SDK
• many more unit tests are added for the scope parsing and serialization functionality
• documentation is updated to refer to these as "Scope objects" rather than "MutableScopes"
• various doc fixes, including missing whitespace, missing scope data in the scopes docs, and a bug in the listknownscopes custom directive

[kurtmckee]

I don't see the following possibilities (which might not all be invalid):

• "foo * " (space after '*')
• "foo[ bar]" (space after '[')
• " foo" (leading space)
• "foo]" (unmatched closing bracket 1)
• "foo bar]" (unmatched closing bracket 2)
• "foo[bar] baz]" (unmatched closing bracket 3)

The reason there are multiple unmatched closing brackets examples is because each might exercise different types of "reset" failures.

Note that if plus-separated string parsing is added after reviewing a comment above, these tests will need to also be updated.

[sirosen]

I just checked, and there are two of these which are valid: " foo" (equivalent to "foo") and "foo[ bar]". The others are all definitely invalid, and I'm more than happy to add them as test cases; thanks for enumerating them!
I'll also add those two valid cases to our tests, to document/enforce their validity.

At this point, I'm setup with a copy of Auth's scope parser to play with, so I can check these against the service to ensure we agree on things to accept or reject. So if you have other questionable cases that you think up, we can evaluate them and add as positive or negative tests.

[kurtmckee]

I just realized there's one more: *foo. I don't think this variant is accounted for yet.

[sirosen]

I just pushed 6a97871, which adds a few test cases for "can parse" as opposed to these for "can't parse".
It includes *foo and gives us a place for any future interesting cases which are valid.

[sirosen]

This work is something I'm considering ready to merge in its current state.

We discussed the differences between the parse tree and the true processing and representation of consents in Globus Auth. The end result of that discussion was that we agreed that parsing is still a very valuable utility to provide, but that one of the pieces here is something which we aren't ready to expose -- __contains__. For the immediate, short term, we are keeping this functionality, but under the internal name _contains. And as an extra measure, the docstring for that internal method declares it "non-authoritative", since it could always be misleading to look at a saved scope string rather than at consents in Auth.

scope1 in scope2 has been removed from public docs, including the changelog.

Also, per a discussion around the add_dependency interface, we are not changing its chaining behavior, but that usage no longer appears in the docs. Removing the chaining behavior is slated for SDK v4.0, to ensure it won't be forgotten when we are next liberated to change interfaces.

[derek-globus]

Overall looks good

The only additional piece I'd ask for before merging would be to include data in the unit tests that performs deserializing/serializing on real (or real-formatted) url & urn scope strings.

I doubt it would cause issues by reading the parser myself, but I wouldn't want us to unintentionally break in a way that we could successfully parse the scope string "foo" but not "urn:globus:auth:scope:auth.globus.org:foo" or "https://auth.globus.org/scopes/auth.globus.org/foo"