Bulk upload implementation guideline

[peterMuriuki]

Introduction

We have previously had a discussion on how to do bulk uploads on the web. This was done and finalized on this thread. The decision made was that we would integrate the importer script into fhir-web’s backend. This document attempts to further that discussion into how we can technically achieve this. We will explore the 2 top alternatives and make a recommendation. The options we currently have are:

1. Handle workflow orchestration on FHIR-web's backend using a library like Bull MQ and redis
2. Integrate the temporal workflow engine directly into the importer script.

Definition

Acronym	Definition
API	Application Programming Interface
FHIR	Fast Healthcare Interoperability Resources

Keywords

1. Workflow - A definition of steps/tasks/activities that are executed in a specific sequence to achieve a predefined outcome. Will also refer to a workflow as a job in the context of this document.
2. Temporal - A workflow engine, its both a framework for writing workflows and a provides a utilities that help with workflows orchestration.
3. Bull Mq - A workflow orchestration library for the node js ecosystem built on top of redis.
4. Importer script - A python shell-based script ONA created to automate the process of batch-updating resources on a HAPI FHIR server.

Feature overview

FHIR Web Frontend

• Bulk upload view
• Include a unified resources upload view that shows
  • All jobs and some meta information
  • Instructions and Option to download templates
  • Form to upload templates

FHIR Web Backend

• Define jobs and a Tasking module for job management. A job represents an instance of child-process that will run the importer script as a shell command.
• Provide endpoints for:
  • Upload of csv files
  • Track progress of jobs.
• Includes the importer script via git sub tree merging.

git remote add -f fhir-tooling https://github.com/onaio/fhir-tooling
git merge -s ours --no-commit --allow-unrelated-histories fhir-tooling/main
git read-tree --prefix=submodules/fhir-tooling/ -u fhir-tooling/main:importer
git commit -m "Merge importer subfolder into fhir-tooling"
git push origin main

Importer Script

• Contains the actual logic that reads CSV files and uploads the data to the FHIR server.

Hapi FHIR server

• Acts as the data repository
• Persist the upload attempts as tasks using the tasks and DocumentReference Resources.

{
	"resourceType": "Task",
	"id": "<uuid>",
	"status": "ready|in-progress|failed|completed",
	"intent": "order",
	"description": "Job Metrics for Job 1",
	"code": {
		"coding": [{
			"system": "http://smartregister.org/codes",
			"code": "importer-job",
			"display": "Importer Job"
		}]
	},
	"authoredOn": "",
	"executionPeriod": {
		"start": "",
		"end": ""
	},
	"requester": {
		"reference": "Practitioner/<user-id>"
	},
	"lastModified": "",
	"input": [{
		"type": {
			"coding": [{
				"system": "http://smartregister.org/codes",
				"code": "importer-job-csv",
				"display": "Import Job Csv"
			}]
		},
		"valueReference": {
			"reference": "DocumentReference/<doc-id>"
		}
	}]
}

{
	"resourceType": "DocumentReference",
	"id": "<job-uuid>",
	"status": "current",
	"subject": {
		"reference": "Task/<task-id>"
	},
	"content": [{
		"attachment": {
			"contentType": "text/csv",
			"data": "ZGF0YTEsZGF0YTIsZGF0YTMKMTIzLDQ1Niw3ODkKNDU2LDc4OSwwMTI=",
			"title": "job-data-1.csv"
		}
	}],
	"securityLabel": [{
		"coding": [{
			"system": "http://terminology.hl7.org/CodeSystem/v3-ConfidentialityClassification",
			"code": "R",
			"display": "Restricted"
		}]
	}]
}

Architecture and Design

Frontend mockups

Backend implementation Arch.

1. Temporal approach

Prepping environment

Development

1. Setup nodejs and python runtimes
2. Install temporal and run in dev mode tempral server start-dev
3. Add configurations:
   1. temporal service uri
   2. [optional] postgres persistence layer

Production

1. Using a Docker file that setups:
   1. node and python runtimes.
   2. install temporal binary and run a single instance at port 3000
   3. Begin process to run importer script
   4. Express service still runs at exposed port 3000.

Persistence

The persistence layer for this approach can be done either by use of a postgres database or the lightweight but ephemeral alternative Sqlite.

My recommendation is to go with the Sqlite option for now and provision the db later once scalability, and reliability of the workflows becomes a concern.

For both storage approaches; The application will save the workflow information to the FHIR server as tasks.

Security

The import feature will require an active logged in user session. The Logged-in user should also have the required roles that allow for updating the respective resources i.e. For a user to upload resources they should have the POST_LOCATION role. Furthermore the express API will serve as the only proxy to the embedded temporal service.

We could consider the option of adding a proxy to the temporal UI for more detailed view of the running workflows but I can consider this out of scope for now.

In addition to the roles that correspond to the upload resource types, there are 2 additional roles that a user should have so that requests to save the workflows as tasks to the fhir server are successful i.e. PUT_TASK, POST_TASK, POST_DOCUMENTREFERENCE, PUT_DOCUMENTREFERENCE.

Areas of new development

1. FHIR web UI
2. FHiR Web express backend, Add endpoints that allow a user to make requests that interact with the temporal service
3. Importer script, Integrate temporal python sdks to allow the importer to interact with the temporal service.
4. DockerFile setup

Tentative Timelines

Web ui development: - 1 week
Web express backend update - 3 days
Importer script update - TBD

Configuration and Customization

TroubleShooting

Perfomance

Monitoring and logging

Documentation updates.

2. Bull MQ & Redis

Prepping environment

Development

1. Setup nodejs
2. Setup python
3. Setup and run redis

Production

1. Using a Docker file that setups:
   1. node and python runtimes.
   2. Express service still runs at exposed port 3000.

Persistence

This approach makes use of redis to store workflow data as well as metadata regarding the job queue. Redis is required in this option whether working in dev or production.

In this approach as well; The application will save the workflows to the FHIR server as tasks.

Security

The import feature will require an active logged in user session. The Logged-in user should also have the required roles that allow for updating the respective resources i.e. For a user to upload resources they should have the POST_LOCATION role. Furthermore the express API will serve as the only proxy to the embedded temporal service.

In addition to the roles that correspond to the upload resource types, there are 2 additional roles that a user should have so that requests to save the workflows as tasks to the fhir server are successful i.e. PUT_TASK, POST_TASK, POST_DOCUMENTREFERENCE, PUT_DOCUMENTREFERENCE.

Areas of new development

1. FHIR web UI
2. FHiR Web express backend,
   1. Add apis to receive and respond to bulk upload requests.
   2. Implement job orchestration functionality: How jobs are added, scheduled, and their status resported
   3. Implement functionality to run the importer script as child processes.
3. Importer script
   1. Possibility to pass both access token and refresh token to the importer script, this should hopefully be enough for the script to request a new token once the current auth token expires.
   2. Refactor the importer script to be able to report errors per csv row where possible e.g. Users are processed one line at a time, or when editing resources. It can then combine this per row errors with any other general errors that arise say when pushing the transaction resource to the hapi fhir server.
   3. Harden the script so that its able to handle a bigger range of exceptions instead of erroring out.
4. DockerFile setup

Tentative Timelines

Web ui development: - 1 week
Web express backend update - 2 week
Importer script update - TBD but less than in the temporal option

Configuration and Customization

TroubleShooting

Perfomance

Monitoring and logging

Documentation updates.

Summary

I think so far temporal is the winner, since it provides a feature rich set of functionalities that we would have to think about and possibly implement by ourselves in the approach 2. Temporal's rich feature set however can be a disadvantage since we will include functionality that we do not need and thus bloat our environments with a possibly over-engineered solution. In addition, the importer script core team feels strongly that temporal would fit into the importer offering quite nicely especially in the scenario where someone might want to use the tool directly without having to setup a fhir-web instance.

#TODO:
Add data flow diagrams that flesh out api interactions for both approaches. we can do a sequence diagram or several flow diagrams. one for each scenario.

[peterMuriuki]

TODO:

• Define bespoke roles for the data upload.
• Remove keycloak roles & groups section from bulk upload.
• Move Storage/persistence aspect to next iteration scope

[peterMuriuki]

Update to the Non-goals for this initial scope

1. No persistence layer
2. No retries

Update:

Lessons learnt from an attempt at using the proposed solution

1. Setup challenge:

One of the goals of this feature was to eventually have this shipped as a single service, possibly use a single image to house it alongside existing web features.
This became quite challenging with Temporal:

• To embed temporal in the single image, one needs to run it in dev mode. I attempted to use supervisord to achieve this. However I found shipping a service running in dev mode within an image meant for production to be ill-advised.
• Running temporal in production mode requires setting up external services that would include a postgres layer in addition to the cluster of services running temporal itself. I found this to violate the constraint mentioned above, in addition it would require allocation of resources that would be difficult to justify to the bizdev and sre teams (I think/assume).

I know think the scope of the feature is not huge enough to warrant using temporal despite its very impressive out of the box capabilities.
As such I pivoted to use bull Queue and redis to manage the job orchestration functionality. We already optionally use redis to store user sessions, and we can disable this feature for deployements that do not a redis instance.