IATI partner types

[kardan]

akvo/akvo-product-design#84

[KasperBrandt]

@zzgvh taking another look at this

[zzgvh]

Notes on the migrations

Above are three migrations managing the keywords.

• First duplicates are removed according to this list:

Dutch WASH Alliance -> WASH Alliance
C4C -> Connect4Change
SRHR -> SRHR Alliance
4winternational -> Football for Water
IGG-Water -> IGG-water program

• Then keywords that are not used for any project are removed.

• Finally a new keyword is generated for each organisation that is a sponsor partner to at least one project, and those projects are tagged with this keyword. Those keywords start with a number and a ":" to distinguish then from regular keywords. You can see them at the start of the list below.

  After the migrations have run the keywords look like this:

Keyword distribution after migration:
1060:Football for Water 17
1061:WASH Liberia 104
1099:Cordaid Food Security 105
113:Wandelen voor Water  95
1151:WvW 2014 23
1241:Cordaid Security/Justice 56
1443:Walking for Water 2014 15
1684:YEP Water 107
2150:WvW 2015 30
2151:Walking for Water 2015 19
272:Connect4Change 85
275:Dutch WASH Alliance 70
2925:WvW 2016 1
444:WvW 2012 16
487:Walking for Water 2012 1
815:WvW 2013 27
829:Walking for Water 2013 13
912:SRHR Alliance 9
946:Cordaid Urban Matters 57
949:Cordaid Healthcare 261
950:Cordaid Entrepreneurship 39
953:Cordaid Investments 80
955:Cordaid Womens Leadership 110
959:Cordaid Child & Education 75
960:Cordaid Extractives 104
961:Cordaid Disaster Response 325
962:Private Initiatives 180
Akvo/AsiaRegion 22
Akvo/Chum 107
Akvo/East Africa 61
Akvo/West Africa 14
AkvoFLOW 60
AkvoRSR 43
Akvopedia 3
BTC/CTB 1
Connect4Change 101
Football for Water 41
Football for Water Ghana 15
Football for Water Kenya 14
Football for Water Mozambique 4
GNWP 20
ICCOAsia 25
IGG-water program 89
PPP3 4
Plan Finland 14
SRHR Alliance 10
WASH Alliance 90
WASH Liberia 104
YEP 108
ginks 8
ice 14
inetwork 2
rain4food 84
ttc 45
ttc-c4c 40
wfw2012 2
wfw2013 14
wfw2014 14
wfw2015 22
wump3r 1
wvw2012 20
wvw2013 26
wvw2014 31
wvw2015 37
wvw2016 1

The number at the end of each line being the number of projects tagged with that keyword.

We can see discrepancies between the sponsor partner keywords and the corresponding regular ones. Example: 272:Connect4Change 85 and Connect4Change 101.
So there are projects that are tagged "normally" but aren't connected via the sponsor partner. To me it looks like the "organisation" keywords like 272:Connect4Change are unneeded where there is a regular corresponding keyword.
This leaves the Cordaid keywords (and possibly some more?) that I think we will use to migrate Cordaid to use keywords rather than sponsor partners.

[zzgvh]

Test plan

This is a fairly large change in how projects are organized and how we differentiate partner types.

Changes:

• The PartnerType model is removed, meaning we don't check what kind of partner an organisation can be.
• The new field Partnership.iati_organisation_role replaces Partnership.partner_type. There is a mapping between the two (see here), the main change being that the iati_organisation_role doesn't really have a value corresponding to the old Akvo partner type Sponsor partner. The data migration includes the sponsor partners for now as the value 100 to clearly distinguish it as outside the proper IATI code list. The plan is to remove it when there are no legacy dependencies on that data.
• The keywords for projects has been cleaned up and extended, see above "Notes on the migrations" for details.

Since much of this work is related to the live data set, I think testing this requires a fresh copy of it. It might be best to compare a the test database before and after applying the migrations in this branch.

Testing the migration of partner_type to iati_organisation_role

GIVEN a fresh copy of the live RSR DB
AND a migrated version of same
WHEN running http://rsr.uat.akvo.org/rest/v1/partnership/?partner_type=[partner_type] pre-migration
AND http://rsr.uat.akvo.org/rest/v1/partnership/?iati_organisation_role=[iati_role] post-migration
THEN the "count" field of the API responses should be the same
SUBSTITUTE [partner_type] : [iati_role] WITH
funding:1
support:2
extending:3
field:4
sponsor:100

Testing of keywords

This is harder to "pin down". In the Notes on the migrations post above you can see a sample list of keywords and the number of projects for each after the migrations have run. This list is printed so it should be visible in the deployment log. None of the duplicate keywords should remain, and there should be a list of new keywords on the ID:org_name form similar to the above. More detailed testing requires writing SQL tests I think.

Testing the APIs

It would be very useful if we could find out how the API is queried regarding partner types to see if there are use cases that we don't cover after the migrations. The APIs look almost the same, you can still query for Partnership.partner_type, but it is not 100% identical so there might be edge cases that we don't support any more. OTOH it's unlikely that anyone uses the API like this, but it'd be good to make sure.

The primary resources to find out how they're used are
http://rsr.akvo.org/rest/v1/partnership/
http://rsr.akvo.org/rest/v1/project/
http://rsr.akvo.org/rest/v1/organisation/
http://rsr.uat.akvo.org/api/v1/partnership/
http://rsr.uat.akvo.org/api/v1/project/
http://rsr.uat.akvo.org/api/v1/organisation/