Skip to content

Commit

Permalink
Merge pull request #4 from liviomendonca/main
Browse files Browse the repository at this point in the history
add `JSONFileChildSeeder` class
  • Loading branch information
suliman-99 authored Sep 2, 2024
2 parents 451de1f + 35e0bfa commit 776e8a3
Show file tree
Hide file tree
Showing 14 changed files with 562 additions and 37 deletions.
21 changes: 21 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
FROM python:3.10.14-bookworm
WORKDIR /app

# Set environment variables to avoid prompts during package installations
# caso não dê certo, voltar `&& export DEBIAN_FRONTEND=noninteractive \`
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
# Remove imagemagick due to https://security-tracker.debian.org/tracker/CVE-2019-10131
&& apt-get purge -y imagemagick imagemagick-6-common

RUN python3 -m pip install --upgrade \
setuptools==69.0.3 \
gitpython==3.1.41

# django-seeding specifics
RUN pip install pipenv
RUN apt-get install -y sqlite3 libsqlite3-dev

# finalizing..
RUN apt-get clean
20 changes: 20 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "dev env",
"dockerComposeFile": "docker-compose.yml",
"service": "backend",
"workspaceFolder": "/workspace", // /${localWorkspaceFolderBasename}"

"customizations": {
"vscode": {
"extensions": [
"ms-azuretools.vscode-docker",
"ms-python.python"
],
"settings": {
"python.analysis.nodeExecutable": ""
}
}
},

"postStartCommand": "pipenv sync"
}
12 changes: 12 additions & 0 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
services:
backend:
build:
context: ..
dockerfile: .devcontainer/Dockerfile
image: python3.10-bookworm
container_name: seeding_PR

volumes:
- ..:/workspace

command: sleep infinity
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
-->
# Contributing to Django Seeding

Thank you for your interest in contributing to Django Seeding We welcome contributions from anyone, regardless of their level of experience or expertise.
Thank you for your interest in contributing to Django Seeding! We welcome contributions from anyone, regardless of their level of experience or expertise.

## Getting Started

Expand Down
175 changes: 157 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
- [Full Examples](#full-examples)
- [CSVFileModelSeeder (Recommended)](#csvfilemodelseeder-recommended)
- [JSONFileModelSeeder (Recommended)](#jsonfilemodelseeder-recommended)
- [JSONFileChildlSeeder](#jsonfilechildseeder)
- [CSVFileSerializerSeeder](#csvfileserializerseeder)
- [JSONFileSerializerSeeder](#jsonfileserializerseeder)
- [EmptySeeder (Recommended)](#emptyseeder-recommended)
Expand All @@ -46,7 +47,7 @@

## Introduction

This package helps develpoers to `fill` the database with `real data` instead of filling it manually.
This package helps developers to `fill` the database with `real data` instead of filling it manually.

Data can be presented as `CSV File` , `JSON File` or `in-code`.

Expand Down Expand Up @@ -104,7 +105,7 @@ t1,d1
t2,d2
```

Now you just need to run this commande:
Now you just need to run this command:

```
python manage.py seed
Expand All @@ -116,6 +117,7 @@ Now lets go deeper into the different Seeders types with its details:
### Seeders List:
- [CSVFileModelSeeder (Recommended)](#csvfilemodelseeder-recommended)
- [JSONFileModelSeeder (Recommended)](#jsonfilemodelseeder-recommended)
- [JSONFileChildlSeeder](#jsonfilechildseeder)
- [CSVFileSerializerSeeder](#csvfileserializerseeder)
- [JSONFileSerializerSeeder](#jsonfileserializerseeder)
- [EmptySeeder (Recommended)](#emptyseeder-recommended)
Expand Down Expand Up @@ -177,8 +179,8 @@ JSONFile..Seeder needs `json_file_path` class-attribute
* `@SeederRegistry.register` is the decorator that register the seeder, so, if this decorator is not applied then the seeder will not be applied
* Model seeders use bulk_create method, so, they are faster than Serializer seeders
* CSV file reader is using pandas for a better performance and less bugs
* Using Model seeders means the fields names must match the fields you have defined in your model
* Using Serializer seeders means the fields names must match the fields you have defined in your serializer
* Using Model seeders means the field names must match the fields you have defined in your model
* Using Serializer seeders means the field names must match the fields you have defined in your serializer
* you can define `get_` class-methods instead of class-attributes as below:

```
Expand Down Expand Up @@ -232,7 +234,7 @@ Here we will go deeper in the seeders classes and its details

Fast `bulk_create` seeder

notice that the titles in the `csv-file` have to match the fields name in the `model`
notice that the titles in the `csv-file` have to match the field names in the `model`

models.py
```
Expand All @@ -246,7 +248,7 @@ seeders.py
@SeederRegistry.register
class M1Seeder(seeders.CSVFileModelSeeder):
id = 'M1Seeder'
priopity = 1
priority = 1
model = M1
csv_file_path = 'django_seeding_example/seeders_data/M1Seeder.csv'
```
Expand All @@ -264,7 +266,7 @@ t2,d2

Fast `bulk_create` seeder

notice that the keys in the `json-file` have to match the fields name in the `model`
notice that the keys in the `json-file` must match the field names in the `model`

models.py
```
Expand All @@ -278,7 +280,7 @@ seeders.py
@SeederRegistry.register
class M2Seeder(seeders.JSONFileModelSeeder):
id = 'M2Seeder'
priopity = 2
priority = 2
model = M2
json_file_path = 'django_seeding_example/seeders_data/M2Seeder.json'
```
Expand All @@ -299,12 +301,149 @@ seeders_data/M2Seeder.json



#### JSONFileChildSeeder

Blinky-fast `bulk-create` seeder implemented with caching strategy.

This seeder was concieved to seed child models, i.e. models that at least one
field is a foreign key (`models.ForeignKey`), but can be used instead of
`JSONFileModelSeeder` for general models as well.

Notice that the keys in the `json-file` must match the field names in the `model`
and also the structure. Parent models are represented as inner dicts.

models.py

```python
class Father(models.Model):
name = models.TextField()

class Son(models.Model):
name = models.TextField()
father = models.ForeignKey(Father, on_delete=models.CASCADE)
```

seeders.py

```python
@SeederRegistry.register
class SonSeeder(seeders.JSONFileChildSeeder):
id = 'SonSeeder'
model = Son
priority = 10
json_file_path = 'django_seeding_example/seeders_data/SonSeeder.json'
```

seeders_data/SonSeeder.json

```json
[
{
"name": "json son 1",
"father": { "name": "json father 1" }
},
{
"name": "json son 2",
"father": { "name": "json father 2" }
}
]
```

Notice that child priority must be greater than parent priority in order to the
parent model be seeded before. Not seeding parent before will raise errors!
Each field that is a FK must be a dictionary with field names same as its related model.

This seeder class can handle pretty complex relations between models.
Let's expand the family (pun intended):

models.py

```python
class Mother(models.Model):
name = models.TextField()

class Daughter(models.Model):
name = models.TextField()
father = models.ForeignKey(Father, on_delete=models.CASCADE)
mother = models.ForeignKey(Mother, on_delete=models.CASCADE)

class Meta:
constraints = [
UniqueConstraint (
fields=['name', 'father', 'mother'],
name='unique_parentage'
)]

class Grandson(models.Model):
name = models.TextField()
parentage = models.ForeignKey(Daughter, on_delete=models.CASCADE)
```

seeders.py

```python
@SeederRegistry.register
class DaughterSeeder(seeders.JSONFileChildSeeder):
id = 'DaughterSeeder'
priority = 10
model = Daughter
json_file_path = 'django_seeding_example/seeders_data/DaughterSeeder.json'


@SeederRegistry.register
class GrandsonSeeder(seeders.JSONFileChildSeeder):
id = 'GrandsonSeeder'
model = Grandson
json_file_path = 'django_seeding_example/seeders_data/GrandsonSeeder.json'
```

seeders_data/DaughterSeeder.json

```json
[
{
"name": "json daughter 1",
"father": { "name": "json father 1" },
"mother": { "name": "json mother 1" }
},
{
"name": "json daughter 2",
"father": { "name": "json father 2" },
"mother": { "name": "json mother 2" }
}
]
```

seeders_data/GrandsonSeeder.json

```json
[
{
"name": "json grandson 1",
"parentage": {
"name": "json daughter 1",
"father": { "name": "json father 1" },
"mother": { "name": "json mother 1" }
}
},
{
"name": "json grandson 2",
"parentage": {
"name": "json daughter 2",
"father": { "name": "json father 2" },
"mother": { "name": "json mother 2" }
}
}
]
```



#### CSVFileSerializerSeeder:

Slow one-by-one seeder

notice that the titles in the `csv-file` have to match the fields name in the `serializer`
notice that the titles in the `csv-file` have to match the field names in the `serializer`

<b> This seeder is used to inject a serializer to implement custom create logic </b>

Expand Down Expand Up @@ -333,7 +472,7 @@ seeders.py
@SeederRegistry.register
class M3Seeder(seeders.CSVFileSerializerSeeder):
id = 'M3Seeder'
priopity = 3
priority = 3
serializer_class = M3Serializer
csv_file_path = 'django_seeding_example/seeders_data/M3Seeder.csv'
```
Expand All @@ -352,7 +491,7 @@ t2,d2

Slow one-by-one seeder

notice that the keys in the `json-file` have to match the fields name in the `serializer`
notice that the keys in the `json-file` have to match the field names in the `serializer`

<b> This seeder is used to inject a serializer to implement custom create logic </b>

Expand Down Expand Up @@ -381,7 +520,7 @@ seeders.py
@SeederRegistry.register
class M4Seeder(seeders.JSONFileSerializerSeeder):
id = 'M4Seeder'
priopity = 4
priority = 4
serializer_class = M4Serializer
json_file_path = 'django_seeding_example/seeders_data/M4Seeder.json'
```
Expand Down Expand Up @@ -419,7 +558,7 @@ seeders.py
@SeederRegistry.register
class M5Seeder(seeders.EmptySeeder):
id = 'M5Seeder'
priopity = 5
priority = 5
model = M5
records_count = 2
```
Expand All @@ -431,7 +570,7 @@ class M5Seeder(seeders.EmptySeeder):

Fast `bulk_create` seeder

notice that the keys in the `data` class-attribute have to match the fields name in the `model`
notice that the keys in the `data` class-attribute have to match the field names in the `model`

models.py
```
Expand All @@ -445,7 +584,7 @@ seeders.py
@SeederRegistry.register
class M6Seeder(seeders.ModelSeeder):
id = 'M6Seeder'
priopity = 6
priority = 6
model = M6
data = [
{
Expand All @@ -466,7 +605,7 @@ class M6Seeder(seeders.ModelSeeder):

Slow one-by-one seeder

notice that the keys in the `data` class-attribute have to match the fields name in the `serializer`
notice that the keys in the `data` class-attribute have to match the field names in the `serializer`

<b> This seeder is used to inject a serializer to implement custom create logic </b>

Expand Down Expand Up @@ -495,7 +634,7 @@ seeders.py
@SeederRegistry.register
class M7Seeder(seeders.SerializerSeeder):
id = 'M7Seeder'
priopity = 7
priority = 7
serializer_class = M7Serializer
data = [
{
Expand Down Expand Up @@ -532,7 +671,7 @@ seeders.py
@SeederRegistry.register
class CustomSeeder(seeders.Seeder):
id = 'CustomSeeder'
priopity = 8
priority = 8
def seed(self):
post1 = Post.objects.create(content='post1')
Expand Down
Loading

0 comments on commit 776e8a3

Please sign in to comment.