Improve typing of view decorators #2164

Viicos · 2024-05-15T16:42:13Z

A new type variable is defined, matching (async) callables taking a request and returning a response.

Some decorators are defined as views to comply with pyright (see microsoft/pyright#5821 (comment) for an explanation).

Note that some decorators allow *args, **kwargs to exist on the decorated views, e.g. never_cache:

def never_cache(view_func):
    """
    Decorator that adds headers to a response so that it will never be cached.
    """

    if iscoroutinefunction(view_func):

        async def _view_wrapper(request, *args, **kwargs):
            _check_request(request, "never_cache")
            response = await view_func(request, *args, **kwargs)
            add_never_cache_headers(response)
            return response

    else:

        def _view_wrapper(request, *args, **kwargs):
            _check_request(request, "never_cache")
            response = view_func(request, *args, **kwargs)
            add_never_cache_headers(response)
            return response

    return wraps(view_func)(_view_wrapper)

However, I think it's good to actually enforce views to take a single request argument. What do you think?

sobolevn · 2024-05-15T17:50:53Z

tests/typecheck/views/decorators/test_csrf.yml

+
+      @csrf_protect
+      def good_view(request: HttpRequest) -> HttpResponse:
+          return HttpResponse()


pls, add reveal_type(good__view)

flaeppe · 2024-05-15T19:59:15Z

However, I think it's good to actually enforce views to take a single request argument. What do you think?

What about path parameters?

flaeppe · 2024-05-15T20:01:08Z

It's probably fine to keep the first positional argument a request, but it needs to allow an arbitrary amount of args and kwargs after that

intgr

Thanks, I like creating a common shared definition for _ViewFuncT.

But some changes are needed.

intgr · 2024-05-16T13:48:32Z

django-stubs/views/decorators/__init__.pyi

+
+from django.utils.deprecation import _AsyncGetResponseCallable, _GetResponseCallable
+
+_ViewFuncT = TypeVar("_ViewFuncT", bound=_AsyncGetResponseCallable | _GetResponseCallable)  # noqa: PYI018


~~Would make sense to keep _AsyncGetResponseCallable, _GetResponseCallable definitions together in the same file with _ViewFuncT.~~

~~Maybe move those 2 into this file? Or put all of them in django-stubs/views/__init__.pyi?~~

Waitaminute. Views aren't necessarily compatible with _GetResponseCallable.

Views can have extra arguments from URL parameters, e.g. def good_view2(request: HttpRequest, url_param: str) -> HttpResponse

This causes error

error: Value of type variable "_ViewFuncT" of "csrf_protect" cannot be "Callable[[HttpRequest, str], HttpResponse]" [type-var]

Yep as suggested by flaeppe. I got mislead because some of these decorators are defined using decorator_from_middleware, which will call get_response (i.e. the view function) only by passing the request. So probably we need another view func type for those

Ok my bad, decorator_from_middleware also supports extra arguments specified in a view function. So in the end I only need to update the definition of _ViewFuncT.

django-stubs/views/decorators/debug.pyi

tests/typecheck/views/decorators/test_csrf.yml

intgr · 2024-05-16T14:20:36Z

However, I think it's good to actually enforce views to take a single request argument. What do you think?

I believe it would break the hack that our FAQ suggests to constrain request.user type for authenticated views? https://github.com/typeddjango/django-stubs?tab=readme-ov-file#how-can-i-create-a-httprequest-thats-guaranteed-to-have-an-authenticated-user

I never liked this hack to begin with (#2046 (comment)), but I think lots of users are relying on it, we shouldn't break it, unless there's strong consensus to do so.

A new type variable is defined, matching (async) callables taking a request and returning a response. Some decorators are defined as views to comply with pyright

Viicos · 2024-05-28T17:55:27Z

tests/assert_type/views/decorators/test_csrf.py

I went for the assert types. Unfortunately I can't do:

assert_type(good_view, Callable[[HttpRequest], HttpResponse])

As Callable does not have information about the name of the arguments, while good_view is inferred as "(request: HttpRequest) -> HttpResponse" (which is a good thing).

Considering the decorators are defined with the type vars in a straightforward way I don't think it is that much of an issue to not be able to test this

I say that we should prefer the type over the name of an argument and thus add an assert_type to ensure the return type of the decorator.

You should be able to use assert_type(good_view, Callable[[HttpRequest], HttpResponse]) if your good_view forces the request argument to be positional e.g.

@csrf_protect -def good_view(request: HttpRequest) -> HttpResponse: +def good_view(request: HttpRequest, /) -> HttpResponse: return HttpResponse() +assert_type(good_view, Callable[[HttpRequest], HttpResponse])

intgr · 2024-05-28T18:25:24Z

django-stubs/views/decorators/__init__.pyi

+    bound=Callable[Concatenate[HttpRequest, ...], HttpResponseBase]
+    | Callable[Concatenate[HttpRequest, ...], Awaitable[HttpResponseBase]],


Unfortunately this still breaks the AuthenticatedHttpRequest hack, #2164 (comment)

Have we tried/do you think we can use callable protocols here? IIRC Concatenate only works in function scope(?)

What if we add a callable protocol for each type? e.g. View and AsyncView

class View(Protocol): def __call__(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponseBase: ... class AsyncView(Protocol): async def __call__(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponseBase: ... _ViewFuncT = TypeVar("_ViewFuncT", bound=View | AsyncView)

Also, can we incorporate the AuthenticatedHttpRequest hack in the test suite? Unless we haven't done so already

__call__(self, request: HttpRequest, *args: Any, **kwargs: Any) will require every view function to have *args, **kwargs int their signature. Callable[Concatenate[HttpRequest, ...], HttpResponseBase] is currently the only way to express what we want here iirc.

I'll take a look at the AuthenticatedHttpRequest hack

__call__(self, request: HttpRequest, *args: Any, **kwargs: Any) will require every view function to have *args, **kwargs int their signature. Callable[Concatenate[HttpRequest, ...], HttpResponseBase] is currently the only way to express what we want here iirc.

From experimenting with mypy playground, I don't think that *args and **kwargs need to be included in the signature. Check this playground link: https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=e5d6cc2baf2095c4eab2863ea645faf3

Let me know if I missed something

Edit: Here's a different playground link that includes a TypeVar and decorator, much similar to what we have in this PR; https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=277b229672c75316b1b9473c07d4712a

Interesting, this seems to have changed really recently in the typing spec:

spec: Expanded "Callables" chapter to include additional details. python/typing#1723

mypy: This behavior was implemented from the beginning apparently.

pyright: (*args: Any, **kwargs: Any) should be treated as ... in function definition microsoft/pyright#7848.

tests/assert_type/views/decorators/test_csrf.py

Viicos · 2024-05-30T20:06:51Z

The AuthenticatedHttpRequest think is annoying :/ Maybe it could be explicitly exported in django-stubs-ext or in the stubs itself, and added as an overload?

intgr · 2024-06-06T09:48:55Z

tests/assert_type/views/decorators/test_csrf.py

+@csrf_protect
+def good_view(request: HttpRequest) -> HttpResponse:


Ignoring AuthenticatedHttpRequest for the moment, the added test_csrf.py is failing for me with mypy... Do you know what's up with that?

tests/assert_type/views/decorators/test_csrf.py:19: error: Value of type variable "_ViewFuncT" of "csrf_protect" cannot be "Callable[[HttpRequest], HttpResponse]" [type-var]
tests/assert_type/views/decorators/test_csrf.py:24: error: Value of type variable "_ViewFuncT" of "csrf_protect" cannot be "Callable[[HttpRequest], Coroutine[Any, Any, HttpResponse]]" [type-var]
tests/assert_type/views/decorators/test_csrf.py:29: error: Value of type variable "_ViewFuncT" of "csrf_protect" cannot be "Callable[[HttpRequest, int, str], HttpResponse]" [type-var]
tests/assert_type/views/decorators/test_csrf.py:34: error: Value of type variable "_ViewFuncT" of "csrf_protect" cannot be "Callable[[HttpRequest, int, str], Coroutine[Any, Any, HttpResponse]]" [type-var]

I wanted to look into what could be done about AuthenticatedHttpRequest, but this is blocking that.

Problem is probably the forced positional of request in the protocols

Indeed. Removing the /, from _View signature fixes that, but then makes line 9 fail.

We can fix this by duplicating both View protocols for both positional/non-positional cases. I think that would be fine. Unless there are better solutions?

class _View(Protocol): def __call__(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponseBase: ... class _ViewPositionalRequest(Protocol): def __call__(self, request: HttpRequest, /, *args: Any, **kwargs: Any) -> HttpResponseBase: ... class _AsyncView(Protocol): async def __call__(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponseBase: ... class _AsyncViewPositionalRequest(Protocol): async def __call__(self, request: HttpRequest, /, *args: Any, **kwargs: Any) -> HttpResponseBase: ... _ViewFuncT = TypeVar( "_ViewFuncT", bound=_View | _ViewPositionalRequest | _AsyncView | _AsyncViewPositionalRequest ) # noqa: PYI018

I don't know what is best to do here really.. Django runs all this with a first, "forced" positional, HttpRequest argument. But it's annoying that everyone then have to add the forced positional marker(/) to avoid violating the protocol.

An additional thing to keep in mind with the protocols here is that if the request argument isn't a forced positional it dictates the name of the argument to be "request".

i.e. trying something like below will yield an error:

@csrf_protect def someview(req: HttpRequest) -> HttpResponse: ...

An additional approach here could be to dig deeper into what attributes the decorator expects. For instance, I think, @csrf_protect "just" requires an object of something has the attributes session and COOKIES(went looking again and found there was a bunch of more attributes/methods, but you get the point) instead of the HttpRequest type. But it requires it as the first argument.

I'm not sure how much or if that can help, I think we still would have to decide on forced positional or not, but just wanted to mention it as an alternate approach to the HttpRequest type

intgr · 2024-06-06T09:51:36Z

django-stubs/views/decorators/__init__.pyi

+class _AsyncView(Protocol):
+    async def __call__(self, request: HttpRequest, /, *args: Any, **kwargs: Any) -> HttpResponseBase: ...
+
+_ViewFuncT = TypeVar("_ViewFuncT", bound=_View | _AsyncView)  # noqa: PYI018


btw django.db.transaction.non_atomic_requests decorator expects views as well, and could re-use this.

intgr · 2024-06-06T10:25:34Z

django-stubs/views/decorators/__init__.pyi

+
+# `*args: Any, **kwargs: Any` means any extra argument(s) can be provided, or none.
+class _View(Protocol):
+    def __call__(self, request: HttpRequest, /, *args: Any, **kwargs: Any) -> HttpResponseBase: ...


Looks like defining a Protocol for request wouldn't work either.

I'm open to just hinting as request: Any to solve the AuthenticatedHttpRequest hack. That's basically the approach we had until now.

intgr · 2024-06-06T10:27:55Z

tests/assert_type/views/decorators/test_csrf.py

+
+@csrf_protect  # type: ignore[type-var]  # pyright: ignore[reportArgumentType, reportUntypedFunctionDecorator]
+def bad_view_no_arguments() -> HttpResponse:
+    return HttpResponse()


Let's add a test for the hack as well:

from django.contrib.auth.models import User class AuthenticatedHttpRequest(HttpRequest): user: User @csrf_protect def view_hack_authenticated_request(request: AuthenticatedHttpRequest) -> HttpResponse: return HttpResponse()

intgr · 2024-06-06T10:29:08Z

django-stubs/views/decorators/__init__.pyi

+from django.http.response import HttpResponseBase
+
+# `*args: Any, **kwargs: Any` means any extra argument(s) can be provided, or none.
+class _View(Protocol):


Add @type_check_only to all protocols.

Viicos force-pushed the view-decorators branch 2 times, most recently from f709670 to 726654a Compare May 15, 2024 17:39

sobolevn reviewed May 15, 2024

View reviewed changes

intgr requested changes May 16, 2024

View reviewed changes

Viicos added 3 commits May 28, 2024 19:16

Improve typing of view decorators

3d7a572

A new type variable is defined, matching (async) callables taking a request and returning a response. Some decorators are defined as views to comply with pyright

Use str for key_prefix

a7827f9

Add test

40e0a36

Viicos force-pushed the view-decorators branch 2 times, most recently from c88356b to 126664a Compare May 28, 2024 17:53

Viicos commented May 28, 2024

View reviewed changes

Feedback

a65d50d

Viicos force-pushed the view-decorators branch from 126664a to a65d50d Compare May 28, 2024 17:56

intgr reviewed May 28, 2024

View reviewed changes

flaeppe reviewed May 28, 2024

View reviewed changes

tests/assert_type/views/decorators/test_csrf.py Outdated Show resolved Hide resolved

Viicos and others added 2 commits May 30, 2024 22:04

Use callback protocols, apply feedback

7b47826

[pre-commit.ci] auto fixes from pre-commit.com hooks

74e07c0

intgr reviewed Jun 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve typing of view decorators #2164

Improve typing of view decorators #2164

Viicos commented May 15, 2024

sobolevn May 15, 2024

flaeppe commented May 15, 2024

flaeppe commented May 15, 2024 •

edited

Loading

intgr left a comment

intgr May 16, 2024 •

edited

Loading

intgr May 16, 2024

Viicos May 16, 2024

Viicos May 28, 2024

intgr commented May 16, 2024

Viicos May 28, 2024

flaeppe May 28, 2024

intgr May 28, 2024

flaeppe May 28, 2024

flaeppe May 28, 2024

Viicos May 28, 2024

flaeppe May 28, 2024 •

edited

Loading

Viicos May 29, 2024

Viicos commented May 30, 2024

intgr Jun 6, 2024

intgr Jun 6, 2024

flaeppe Jun 6, 2024 •

edited

Loading

intgr Jun 6, 2024 •

edited

Loading

This comment was marked as outdated.

flaeppe Jun 6, 2024 •

edited

Loading

intgr Jun 6, 2024

intgr Jun 6, 2024

intgr Jun 6, 2024 •

edited

Loading

intgr Jun 6, 2024


		from django.utils.deprecation import _AsyncGetResponseCallable, _GetResponseCallable

		_ViewFuncT = TypeVar("_ViewFuncT", bound=_AsyncGetResponseCallable \| _GetResponseCallable) # noqa: PYI018

		bound=Callable[Concatenate[HttpRequest, ...], HttpResponseBase]
		\| Callable[Concatenate[HttpRequest, ...], Awaitable[HttpResponseBase]],

		@csrf_protect
		def good_view(request: HttpRequest) -> HttpResponse:

Improve typing of view decorators #2164

Are you sure you want to change the base?

Improve typing of view decorators #2164

Conversation

Viicos commented May 15, 2024

Choose a reason for hiding this comment

flaeppe commented May 15, 2024

flaeppe commented May 15, 2024 • edited Loading

intgr left a comment

Choose a reason for hiding this comment

intgr May 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

intgr commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flaeppe May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Viicos commented May 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flaeppe Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

intgr Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

flaeppe Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

intgr Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flaeppe commented May 15, 2024 •

edited

Loading

intgr May 16, 2024 •

edited

Loading

flaeppe May 28, 2024 •

edited

Loading

flaeppe Jun 6, 2024 •

edited

Loading

intgr Jun 6, 2024 •

edited

Loading

flaeppe Jun 6, 2024 •

edited

Loading

intgr Jun 6, 2024 •

edited

Loading