78 - Fix checksum migration issue (#79)

- Rollback changes in immutable migration file and applied intended changes in new SQL script. Migrations files are signed with a md5 sum in the DB to avoid modifications, so the SQL file was restored to its original state.
- The changes were moved to a new a script, but because is not possible to add a new column to a materialized view, the change was applied recreating the view, and all its dependents indexes and views.
- The changes to the view `raw_contacts` that was rolled back was redundant, because the same migration to add 'contact' to the list of values for `doc->>'type'` was added in an earlier migration script.
- Add README notes and .gitignore rule.

Co-authored-by: Diana Barsan <35681649+dianabarsan@users.noreply.github.com>

.gitignore

@@ -1 +1,2 @@
 node_modules
+.idea/

README.md

@@ -6,6 +6,7 @@ The focus is specifically on CHT application data currently stored in CouchDB. I
 
 This version is built for medic/cht-core#3.0.0 and above. For replicating data from earlier versions, see the 2.0.x branch and associated tags.
 
+
 ## Installation Steps (if applicable)
 
 1. Clone repository
@@ -38,16 +39,25 @@ Run it locally with environment variables: `npm ci && node .`
 
 Run it locally in interactive mode: `npm ci && node . -i`
 
+
 ## Running tests through docker-compose
 
 Run tests with:
+
 ```
 docker-compose build --build-arg node_version=[node version] test
+docker-compose up
+```
+
+Then in another terminal:
+
+```
 docker-compose run test grunt test
 ```
 
 Run tests in interactive watch mode with: `docker-compose run test npm run watch`.
 
+
 ## Running tests against local couch and postgres databases
 
 Run tests with: `grunt test`.
@@ -59,10 +69,12 @@ Environment variables required for the integration tests to run correctly:
 
 NB: The integration tests destroy and re-create the given databases each time they are run.
 
+
 ## Required database setup
 
 We support PostgreSQL 9.4 and greater. The user passed in the postgres url needs to have full creation rights on the given database.
 
+
 ## Example usage
 
 You should probably install medic-analytics as a service and leave it to do its thing, as it should be able to run independently without any user input.
@@ -86,7 +98,8 @@ end script
 ```
  - The service is then a standard service, e.g. `service couch2pg-example-client start`
 
- ### Installing as a service using Systemd (18.04.3 LTS [Bionic Beaver])
+### Installing as a service using Systemd (18.04.3 LTS [Bionic Beaver])
+
 To setup couch2pg using systemd is also pretty simple. You will need to have sudo rights to the server and then follow the steps listed below:
 
  - Install git and clone this repo onto your server, check out the relevant tag `git checkout tag_id`, and run `npm ci`. 
@@ -126,6 +139,35 @@ WantedBy=multi-user.target
  - Start the service `sudo service couch2pg-sample-client start`
  - If all goes well the service should start smoothly.
  - You can check the service logs using `journalctl` like this `journalctl -u couch2pg-sample-client --since today`
-
-
 
+
+## Known issues
+
+### Error "Checksum failed for migration ..." when upgrading from 3.2.0 to latest
+
+An SQL migration file was changed in version 3.2.0. This made upgrades from 3.1.x impossible, with the process crashing upon startup after the upgrade. See more [details about the error](https://github.com/medic/medic-couch2pg/issues/78).
+
+This was fixed in version 3.2.1, by reverting the changes made to the migration file. 
+Fresh installations of 3.2.0 should execute this SQL before upgrading: 
+
+```sql
+UPDATE xmlforms_migrations
+  SET md5 = 'e0535c9fe3faef6e66a31691deebf1a8'
+  WHERE version = '201606200952' AND
+        md5 = '40187aa5ee95eda0e154ecefd7512cda';
+```
+
+See more details about the error in [#78](https://github.com/medic/medic-couch2pg/issues/78).
+
+### Error installing deps `ERR! ... node-pre-gyp install --fallback-to-build`
+
+When installing Node.js dependencies locally or building the docker image, you might get an error like:
+
+```
+...
+npm ERR! code ELIFECYCLE
+npm ERR! errno 1
+npm ERR! node-libcurl@1.3.3 install: `node-pre-gyp install --fallback-to-build`
+```
+
+It is probably related to a gcc library that is failing with some versions of Node and npm, try with Node 10 without updating the `npm` version that comes with it.

libs/analytics/migrations/201606200952.do.2318-prepareContacts.sql

@@ -1,10 +1,10 @@
 -- filter contact docs into one place
-CREATE OR REPLACE VIEW raw_contacts AS SELECT * FROM couchdb WHERE doc->>'type' IN ('clinic', 'district_hospital', 'health_center', 'person', 'contact');
+CREATE OR REPLACE VIEW raw_contacts AS SELECT * FROM couchdb WHERE doc->>'type' IN ('clinic', 'district_hospital', 'health_center', 'person');
 
 -- extract JSON data from contact docs and cache it
 DROP MATERIALIZED VIEW IF EXISTS contactview_metadata CASCADE;
 CREATE MATERIALIZED VIEW contactview_metadata AS
-SELECT doc->>'_id' AS uuid, doc->>'name' AS name, doc->>'type' AS type, doc->>'contact_type' AS contact_type, doc#>>'{contact,_id}' AS contact_uuid, doc#>>'{parent,_id}' AS parent_uuid, doc->>'notes' AS notes,
+SELECT doc->>'_id' AS uuid, doc->>'name' AS name, doc->>'type' AS type, doc#>>'{contact,_id}' AS contact_uuid, doc#>>'{parent,_id}' AS parent_uuid, doc->>'notes' AS notes,
 TIMESTAMP WITH TIME ZONE 'epoch' + (doc->>'reported_date')::numeric / 1000 * interval '1 second' AS reported
 FROM raw_contacts;
 

libs/analytics/migrations/202104281912.do.0078-contactType.sql

@@ -0,0 +1,61 @@
+-- Recreates the view only to add the new column contact_type
+-- Replaces -> https://github.com/medic/medic-couch2pg/blob/53c8b27aa6e33c8e2a62dc7d8f697cad20b891c7/libs/analytics/migrations/201606200952.do.2318-prepareContacts.sql#L5-L25
+DROP MATERIALIZED VIEW IF EXISTS contactview_metadata CASCADE;
+CREATE MATERIALIZED VIEW contactview_metadata AS
+  SELECT doc->>'_id' AS uuid,
+    doc->>'name' AS name,
+    doc->>'type' AS type,
+    doc->>'contact_type' AS contact_type,       --> only this is new
+    doc#>>'{contact,_id}' AS contact_uuid,
+    doc#>>'{parent,_id}' AS parent_uuid,
+    doc->>'notes' AS notes,
+    TIMESTAMP WITH TIME ZONE 'epoch' + (doc->>'reported_date')::numeric / 1000 * interval '1 second' AS reported
+  FROM raw_contacts;
+
+CREATE UNIQUE INDEX contactview_metadata_uuid ON contactview_metadata (uuid);
+CREATE INDEX contactview_metadata_contact_uuid ON contactview_metadata (contact_uuid);
+CREATE INDEX contactview_metadata_parent_uuid ON contactview_metadata (parent_uuid);
+CREATE INDEX contactview_metadata_type ON contactview_metadata (type);
+
+-- NOTE: The recreation of the view above caused 4 other views to be dropped in cascade,
+--       here are the scripts to recreate them:
+
+-- Replace -> https://github.com/medic/medic-couch2pg/blob/53c8b27aa6e33c8e2a62dc7d8f697cad20b891c7/libs/analytics/migrations/201606200952.do.2318-prepareContacts.sql#L31
+CREATE VIEW contactview_hospital AS
+  SELECT cmd.uuid, cmd.name
+    FROM contactview_metadata AS cmd
+    WHERE cmd.type = 'district_hospital';
+
+-- Replace -> https://github.com/medic/medic-couch2pg/blob/53c8b27aa6e33c8e2a62dc7d8f697cad20b891c7/libs/analytics/migrations/201606200952.do.2318-prepareContacts.sql#L48
+CREATE VIEW contactview_chw AS
+  SELECT chw.name, pplfields.*, chwarea.uuid AS area_uuid,
+         chwarea.parent_uuid AS branch_uuid
+    FROM contactview_person_fields AS pplfields
+         INNER JOIN contactview_metadata AS chw ON (chw.uuid = pplfields.uuid)
+         INNER JOIN contactview_metadata AS chwarea ON (chw.parent_uuid = chwarea.uuid)
+    WHERE pplfields.parent_type = 'health_center';
+
+-- Replace -> https://github.com/medic/medic-couch2pg/blob/53c8b27aa6e33c8e2a62dc7d8f697cad20b891c7/libs/analytics/migrations/201606200952.do.2318-prepareContacts.sql#L57
+CREATE VIEW contactview_clinic AS
+  SELECT cmd.uuid, cmd.name, chw.uuid AS chw_uuid, cmd.reported AS created
+    FROM contactview_metadata AS cmd
+         INNER JOIN contactview_chw AS chw ON (cmd.parent_uuid = chw.area_uuid)
+    WHERE type = 'clinic';
+
+-- Replace -> https://github.com/medic/medic-couch2pg/blob/53c8b27aa6e33c8e2a62dc7d8f697cad20b891c7/libs/analytics/migrations/201711071603.do.2635-removeNestedContactReferences.sql#L4
+CREATE VIEW contactview_clinic_person AS
+  SELECT
+      raw_contacts.doc ->> '_id' AS uuid,
+      raw_contacts.doc ->> 'name' AS name,
+      raw_contacts.doc ->> 'type' AS type,
+      raw_contacts.doc #>> '{parent,_id}' AS family_uuid,
+      raw_contacts.doc ->> 'phone' AS phone,
+      raw_contacts.doc ->> 'alternative_phone' AS phone2,
+      raw_contacts.doc ->> 'date_of_birth' AS date_of_birth,
+      cmeta.type AS parent_type
+    FROM
+      raw_contacts
+      LEFT JOIN contactview_metadata AS cmeta ON (doc #>> '{parent,_id}' = cmeta.uuid)
+    WHERE
+      raw_contacts.doc->>'type' = 'person' AND
+      raw_contacts.doc->>'_id' IN (SELECT contact_uuid FROM contactview_metadata WHERE type = 'clinic');