Skip to content

Commit 7381975

Browse files
andrewdoddandrewdodd
committed
Documentation
Filling out examples mainly.
1 parent 77bd8fa commit 7381975

File tree

3 files changed

+216
-21
lines changed

3 files changed

+216
-21
lines changed

‎AUTHORS.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,4 @@ Development Lead
1010
Contributors
1111
------------
1212

13-
None yet. Why not be the first?
13+
* Chris Snell

‎CONTRIBUTING.rst

+3-2
Original file line numberDiff line numberDiff line change
@@ -77,11 +77,12 @@ Now you can make your changes locally.
7777
5. When you're done making changes, check that your changes pass flake8 and the
7878
tests, including testing other Python versions with tox::
7979

80-
$ flake8 nsync tests
80+
$ flake8 src tests
8181
$ python setup.py test
8282
$ tox
8383

84-
To get flake8 and tox, just pip install them into your virtualenv.
84+
To get flake8 and tox, just pip install them into your virtualenv. NB: Don't
85+
worry about flake8 issues in the migrations files or if URLs are too long.
8586

8687
6. Commit your changes and push your branch to GitHub::
8788

‎docs/examples.rst

+212-18
Original file line numberDiff line numberDiff line change
@@ -2,34 +2,228 @@
22
Examples
33
========
44

5-
MORE COMING SOON
5+
The following are some examples of using the Nsync functionality. The
6+
following Django models will be used as the target models::
67

7-
Plain file
8-
Plain file with external system
8+
class Person(models.Model):
9+
first_name = models.CharField(
10+
blank=False,
11+
max_length=50,
12+
verbose_name='First Name'
13+
)
14+
last_name = models.CharField(
15+
blank=False,
16+
max_length=50,
17+
verbose_name='Last Name'
18+
)
19+
age = models.IntegerField(blank=True, null=True)
920

10-
Two or more systems
21+
hair_colour = models.CharField(
22+
blank=False,
23+
max_length=50,
24+
default="Unknown")
1125

12-
Referential fields
1326

14-
Delete tricks
27+
class House(models.Model):
28+
address = models.CharField(max_length=100)
29+
country = models.CharField(max_length=100, blank=True)
30+
floors = models.IntegerField(blank=True, null=True)
31+
owner = models.ForeignKey(TestPerson, blank=True, null=True)
1532

1633

17-
Example - Plain file
18-
--------------------
34+
Example - Basic
35+
---------------
1936

20-
.. example-persons-noexternal-txt-begin
21-
::
22-
first_name,last_name,employee_id,action_flags,match_field_name
23-
Andrew,Dodd,E1234,cu,employee_id
24-
Some,Other-Guy,E4321,d,employee_id
37+
Using this file:
2538

26-
.. example-persons-noexternal-txt-end
39+
.. csv-table:: persons.csv
40+
:header: "action_flags", "match_field_name", "first_name", "last_name", "employee_id"
41+
42+
"cu","employee_id","Andrew","Dodd","EMP1111"
43+
"d*","employee_id","Some","Other-Guy","EMP2222"
44+
"cu","employee_id","Captain","Planet","EMP3333"
45+
"u*","employee_id","C.","Batman","EMP1234"
46+
47+
And running this command::
48+
49+
> python manage.py syncfile TestSystem myapp Person persons.csv
50+
51+
Would:
52+
- Create and/or update ``myapp.Person`` objects with Employee Ids EMP1111 & EMP3333. However, it would not update the two name fields if the objects already existed with non-blank fields.
53+
- Delete any ``myapp.Person`` objects with Employee Id EMP2222.
54+
- If a person with Employee Id EMP1234 exists, then it will forcibly update the name fields to 'C.' and 'Batman' respectively.
55+
56+
NB it would also:
57+
- Create an ``nsync.ExternalSystem`` object with the name TestSystem, as the default is to create missing external systems. However, because there are no ``external_key`` values, no ``ExternalKeyMapping`` objects would be created.
58+
59+
Example - Basic with External Ids
60+
---------------------------------
61+
62+
Using this file:
2763

28-
.. example-persons-external-csv-begin
2964
.. csv-table:: persons.csv
3065
:header: "external_key", "action_flags", "match_field_name", "first_name", "last_name", "employee_id"
3166

32-
1221228,"cu","employee_id","Andrew","Dodd","EMP1111"
33-
4371928,"d","employee_id","Some","Other-Guy","EMP2222"
34-
.. example-persons-external-csv-end
67+
12212281,"cu","employee_id","Andrew","Dodd","EMP1111"
68+
43719289,"d*","employee_id","Some","Other-Guy","EMP2222"
69+
99999999,"cu","employee_id","Captain","Planet","EMP3333"
70+
11235813,"u*","employee_id","C.","Batman","EMP1234"
71+
72+
And running this command::
73+
74+
> python manage.py syncfile TestSystem myapp Person persons.csv
75+
76+
Would:
77+
- Perform all of steps in the 'Plain file' example
78+
- Delete any ``ExternalKeyMapping`` objects that are for the 'TestSystem' and have the external key '43719289' (i.e. the record for Some Other-Guy).
79+
- Create or update ``ExternalKeyMapping`` objects for each of the other three ``myapp.Person`` objects, which contain the ``external_key`` value.
80+
81+
82+
Example - Two or more systems
83+
-----------------------------
84+
This is probably the main purpose of this library: the ability to
85+
synchronise from multiple systems.
86+
87+
Perhaps we need to synchronise from two data sources on housing information,
88+
one is the 'when built' information and the other is the 'renovations'
89+
information.
90+
91+
As-built data:
92+
93+
.. csv-table:: AsBuiltDB_myapp_House.csv
94+
:header: "external_key", "action_flags", "match_field_name", "address", "country", "floors"
95+
96+
111,"cu","address","221B Baker Street","England",1
97+
222,"cu","address","Wayne Manor","Gotham City",2
98+
99+
Renovated data:
100+
101+
.. csv-table:: RenovationsDB_myapp_House.csv
102+
:header: "external_key", "action_flags", "match_field_name", "address", "floors"
103+
104+
ABC123,"u*","address","221B Baker Street",2
105+
ABC456,"u*","address","Wayne Manor",4
106+
FOX123,"u*","address","742 Evergreen Terrace",2
107+
108+
109+
And running this command::
110+
111+
> python manage.py syncfiles AsBuiltDB_myapp_House.csv RenovationsDB_myapp_House.csv
112+
113+
Would:
114+
- Use the **mutliple file command**, ``syncfiles``, to perform multiple updates in one command
115+
- Create the two houses from the 'AsBuilt' file
116+
- Only update the ``country`` values of the two houses from the 'AsBuilt' file IFF the objects already existed but they did not have a value for ``country``
117+
- Forcibly set the ``floors`` attribute for the first two houses in the 'Renovations' file.
118+
- Create 4 ``ExternalKeyMapping`` objects:
119+
120+
+---------------+--------+----------------------+
121+
| External | Ext. | House Object |
122+
| System | Key | |
123+
+===============+========+======================+
124+
| AsBuiltDB | 111 | |
125+
+---------------+--------+ 212B Baker Street |
126+
| RenovationsDB | ABC123 | |
127+
+---------------+--------+----------------------+
128+
| AsBuiltDB | 222 | |
129+
+---------------+--------+ Wayne Manor |
130+
| RenovationsDB | ABC456 | |
131+
+---------------+--------+----------------------+
132+
- Only update the ``floors`` attribute for "742 Evergreen Terrace" if the house already exists (and would then also create an ``ExternalKeyMapping``)
133+
134+
135+
Example - Referential fields
136+
----------------------------
137+
You can also manage referential fields with Nsync. For example, if you had the following people:
138+
139+
.. csv-table:: Examples_myapp_Person.csv
140+
:header: "external_key", "action_flags", "match_field_name", "first_name", "last_name", "employee_id"
141+
142+
1111,"cu*","employee_id","Homer","Simpson","EMP1"
143+
2222,"cu*","employee_id","Bruce","Wayne","EMP2"
144+
3333,"cu*","employee_id","John","Wayne","EMP3"
145+
146+
You could set their houses with a file like this:
147+
148+
.. csv-table:: Examples_myapp_House.csv
149+
:header: "external_key", "action_flags", "match_field_name", "address", "owner=>first_name"
150+
151+
ABC456,"cu*","address","Wayne Manor","Bruce"
152+
FOX123,"cu*","address","742 Evergreen Terrace","Homer"
153+
154+
The **"=>"** is used by Nsync to follow the the related field on the provided object.
155+
156+
Example - Referential field gotchas
157+
-----------------------------------
158+
The referential field update will ONLY be performed if the referred-to-fields target a single object. For example, if you had the following list of people:
159+
160+
.. csv-table:: Examples_myapp_Person.csv
161+
:header: "external_key", "action_flags", "match_field_name", "first_name", "last_name", "employee_id"
162+
163+
1111,"cu*","employee_id","Homer","Simpson","EMP1"
164+
2222,"cu*","employee_id","Homer","The Greek","EMP2"
165+
3333,"cu*","employee_id","Bruce","Wayne","EMP3"
166+
4444,"cu*","employee_id","Bruce","Lee","EMP4"
167+
5555,"cu*","employee_id","John","Wayne","EMP5"
168+
6666,"cu*","employee_id","Marge","Simpson","EMP6"
169+
170+
The ``owner=>first_name`` from the previous example is insufficient to pick out a single person to link a house to (there are 2 Homers and 2 Bruces). Using just the ``employee_id`` field would work, but that piece of information may not be available in the system for houses.
171+
172+
Nsync allows you to specify multiple fields to use in order to 'filter' the correct object to create the link with. In this instance, this file would perform correctly:
173+
174+
.. csv-table:: Examples_myapp_House.csv
175+
:header: "external_key", "action_flags", "match_field_name", "address", "owner=>first_name", "owner=>last_name"
176+
177+
ABC456,"cu*","address","Wayne Manor","Bruce","Wayne"
178+
FOX123,"cu*","address","742 Evergreen Terrace","Homer","Simpson"
179+
180+
181+
Example - Complex Fields
182+
------------------------
183+
If you want a more complex update you can:
184+
- Write an extension to Nsync and submit a Pull Request! OR
185+
- Extend your Django model with a custom setter
186+
187+
If your Person model has a photo ImageField, then you could add a custom handler to update the photo based on a provided file path::
188+
189+
class Person(models.Model):
190+
...
191+
photo = models.ImageField(
192+
blank = True,
193+
null = True,
194+
max_length = 200,
195+
upload_to = 'person_photos',
196+
)
197+
...
198+
199+
@photo_filename.setter
200+
def photo_filename(self, file_path):
201+
...
202+
Do the processing of the file to update the model
203+
204+
And then supply the photos with a file sync file like:
205+
206+
.. csv-table:: persons.csv
207+
:header: "action_flags", "match_field_name", "first_name", "last_name", "employee_id", "photo_filename"
208+
209+
"cu*","employee_id","Andrew","Dodd","EMP1111","/tmp/photos/ugly_headshot.jpg"
210+
211+
212+
Example - Delete tricks
213+
-----------------------
214+
This is a list of tricky / gotchas to be aware of when deleting objects.
215+
216+
When syncing from external systems that have external key mappings, it is probably best to use the 'unforced delete'. This ensures that an object is not removed until all of the external systems think it should be removed.
217+
218+
If using 'forced delete', beware that (depending on which sync policy you use) you may end up with different systems fighting over the existence of an object (i.e. one system creating the object, then another deleting it in the same sync).
219+
220+
A system without external key mappings cannot delete objects if it uses an 'unforced delete'. The reason for this is that the 'unforced delete' only removes the model object IF AND ONLY IF it is the last remaining external key mapping. Thus, if a system without external key mappings is the source-of-truth for the removal of an object, you must use the 'forced delete' for it to be able to remove the objects.
221+
222+
223+
Alternative Sync Policies
224+
-------------------------
225+
The out-of-the-box sync policies are pretty straightforward and are probably worth a read (see the ``policies.py`` file). The system is made so that it is pretty easy for you to define your own custom policy and write a command (similar to the ones in Nsync) to use it.
35226

227+
Some examples of alternative policies might be:
228+
- Run deletes before creates and updates
229+
- Search and execute certain actions before all others

0 commit comments

Comments
 (0)