Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add drf like serializer #32

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 46 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,25 @@

[![CircleCI](https://circleci.com/gh/gluk-w/django-grpc.svg?style=svg)](https://circleci.com/gh/gluk-w/django-grpc)


Easy way to launch gRPC server with access to Django ORM and other handy stuff.
gRPC calls are much faster that traditional HTTP requests because communicate over
persistent connection and are compressed. Underlying gRPC library is written in C which
makes it work faster than any RESTful framework where a lot of time is spent on serialization/deserialization.

Note that you need this project only if you want to use Django functionality in gRPC service.
Note that you need this project only if you want to use Django functionality in gRPC service.
For pure python implementation [read this](https://grpc.io/docs/languages/python/quickstart/)

* Supported Python: 3.4+
* Supported Django: 2.X, 3.X and 4.X
- Supported Python: 3.4+
- Supported Django: 2.X, 3.X and 4.X

## Installation

```bash
pip install django-grpc
```
```

Update settings.py

```python
INSTALLED_APPS = [
# ...
Expand All @@ -41,6 +41,7 @@ GRPCSERVER = {
```

The callback that initializes "servicer" must look like following:

```python
import my_pb2
import my_pb2_grpc
Expand All @@ -57,26 +58,42 @@ class MYServicer(my_pb2_grpc.MYServicer):
```

## Usage

```bash
python manage.py grpcserver
```

For developer's convenience add `--autoreload` flag during development.


## Signals

The package uses Django signals to allow decoupled applications get notified when some actions occur:
* `django_grpc.signals.grpc_request_started` - sent before gRPC server begins processing a request
* `django_grpc.signals.grpc_request_finished` - sent when gRPC server finishes delivering response to the client
* `django_grpc.signals.grpc_got_request_exception` - this signal is sent whenever RPC encounters an exception while
processing an incoming request.

Note that signal names are similar to Django's built-in signals, but have "grpc_" prefix.
- `django_grpc.signals.grpc_request_started` - sent before gRPC server begins processing a request
- `django_grpc.signals.grpc_request_finished` - sent when gRPC server finishes delivering response to the client
- `django_grpc.signals.grpc_got_request_exception` - this signal is sent whenever RPC encounters an exception while
processing an incoming request.

Note that signal names are similar to Django's built-in signals, but have "grpc\_" prefix.

## Serializers

There is an easy way to serialize django model to gRPC message using `django_grpc.serializers.serialize_model`.

Or you can use `django_grpc.serializers.drf.GrpcSerializer` to serialize and deserialize like django rest framework, need `djangorestframework` dependency.

```python
from django_grpc.serializers.drf import GrpcSerializer

class MySerializer(GrpcSerializer):
class Meta:
model = MyModel
proto_class = MyProtoClass
fields = '__all__'

proto_data = MySerializer(instance=MyModel.objects.first()).message
```

## Helpers

### Ratelimits
Expand All @@ -89,11 +106,12 @@ from django_grpc.helpers import ratelimit


class Greeter(helloworld_pb2_grpc.GreeterServicer):

@ratelimit(max_calls=10, time_period=60)
def SayHello(self, request, context):
return helloworld_pb2.HelloReply(message='Hello, %s!' % request.name)
```

> When limit is reached for given time period decorator will abort with status `grpc.StatusCode.RESOURCE_EXHAUSTED`

As storage for state of calls [Django's cache framework](https://docs.djangoproject.com/en/4.0/topics/cache/#django-s-cache-framework)
Expand All @@ -102,6 +120,7 @@ is used. By default `"default"` cache system is used but you can specify any oth
#### Advanced usage

Using groups

```python
@ratelimit(max_calls=10, time_period=60, group="main")
def foo(request, context):
Expand All @@ -111,14 +130,17 @@ def foo(request, context):
def bar(request, context):
...
```

`foo` and `bar` will share the same counter because they are in the same group

Using keys

```python
@ratelimit(max_calls=5, time_period=10, keys=["request:dot.path.to.field"])
@ratelimit(max_calls=5, time_period=10, keys=["metadata:user-agent"])
@ratelimit(max_calls=5, time_period=10, keys=[lambda request, context: context.peer()])
```

Right now 3 type of keys are supported with prefixes `"request:"`, `"metadata:"` and as callable.

- `"request:"` allows to extract request's field value by doted path
Expand All @@ -129,9 +151,10 @@ Right now 3 type of keys are supported with prefixes `"request:"`, `"metadata:"`
> and can cause sharing of ratelimits between different RPCs in the same group

> TIP: To use the same configuration for different RPCs use dict variable
>
> ```python
> MAIN_GROUP = {"max_calls": 5, "time_period": 60, "group": "main"}
>
>
> @ratelimit(**MAIN_GROUP)
> def foo(request, context):
> ...
Expand All @@ -141,15 +164,17 @@ Right now 3 type of keys are supported with prefixes `"request:"`, `"metadata:"`
> ...
> ```


## Testing
Test your RPCs just like regular python methods which return some

Test your RPCs just like regular python methods which return some
structure or generator. You need to provide them with only 2 parameters:
request (protobuf structure or generator) and context (use `FakeServicerContext` from the example below).

### Fake Context

You can pass instance of `django_grpc_testtools.context.FakeServicerContext` to your gRPC method
to verify how it works with context (aborts, metadata and etc.).

```python
import grpc
from django_grpc_testtools.context import FakeServicerContext
Expand All @@ -160,13 +185,13 @@ servicer = Greeter()
context = FakeServicerContext()
request = HelloRequest(name='Tester')

# To check metadata set by RPC
# To check metadata set by RPC
response = servicer.SayHello(request, context)
assert context.get_trailing_metadata("Header1") == '...'

# To check status code
try:
servicer.SayHello(request, context)
servicer.SayHello(request, context)
except Exception:
pass

Expand All @@ -175,6 +200,7 @@ assert context.abort_message == 'Cannot say hello to John'
```

In addition to standard gRPC context methods, FakeServicerContext provides:
* `.set_invocation_metadata()` allows to simulate metadata from client to server.
* `.get_trailing_metadata()` to get metadata set by your server
* `.abort_status` and `.abort_message` to check if `.abort()` was called

- `.set_invocation_metadata()` allows to simulate metadata from client to server.
- `.get_trailing_metadata()` to get metadata set by your server
- `.abort_status` and `.abort_message` to check if `.abort()` was called
74 changes: 74 additions & 0 deletions django_grpc/serializers/drf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
try:
from google.protobuf.json_format import MessageToDict, ParseDict
from rest_framework import serializers
yswtrue marked this conversation as resolved.
Show resolved Hide resolved

def _is_field_optional(field):
"""
Checks if a field is optional.

Under the hood, Optional fields are OneOf fields with only one field with the name of the OneOf
prefixed with an underscore.
"""

if not (co := field.containing_oneof):
return False

return len(co.fields) == 1 and co.name == f"_{field.name}"

def message_to_dict(message, **kwargs):
"""
Converts a protobuf message to a dictionary.
Uses the default `google.protobuf.json_format.MessageToDict` function.
Adds None values for optional fields that are not set.
"""

kwargs.setdefault("including_default_value_fields", True)
kwargs.setdefault("preserving_proto_field_name", True)

result_dict = MessageToDict(message, **kwargs)
optional_fields = {
field.name: None
for field in message.DESCRIPTOR.fields
if _is_field_optional(field)
}

return {**optional_fields, **result_dict}

def parse_dict(js_dict, message, **kwargs):
kwargs.setdefault("ignore_unknown_fields", True)
return ParseDict(js_dict, message, **kwargs)

class GrpcSerializer(serializers.BaseSerializer):
def __init__(self, *args, **kwargs):
message = kwargs.pop("message", None)
if message is not None:
self._message = message
kwargs["data"] = self.message_to_data(message)
super().__init__(*args, **kwargs)

@property
def message(self):
if not hasattr(self, "_message"):
self._message = self.data_to_message(self.data)
return self._message

def message_to_data(self, message):
"""Protobuf message -> Dict of python primitive datatypes."""
return message_to_dict(message)

def data_to_message(self, data):
"""Protobuf message <- Dict of python primitive datatypes."""
assert hasattr(
self, "Meta"
), 'Class {serializer_class} missing "Meta" attribute'.format(
serializer_class=self.__class__.__name__
)
assert hasattr(
self.Meta, "proto_class"
), 'Class {serializer_class} missing "Meta.proto_class" attribute'.format(
serializer_class=self.__class__.__name__
)
return parse_dict(data, self.Meta.proto_class())

except ImportError:
pass