Skip to content

Commit

Permalink
Merge branch 'master' into isolate_fhir_sync
Browse files Browse the repository at this point in the history
  • Loading branch information
mozzy11 authored Jan 8, 2025
2 parents 0ee09d6 + a11b712 commit 1f0acf3
Show file tree
Hide file tree
Showing 3 changed files with 12 additions and 12 deletions.
18 changes: 9 additions & 9 deletions doc/docs/concepts/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ FHIR Resources are transformed into a "Parquet on FHIR" format:
* Configurable support for FHIR profiles and extensions
* (Optional) In-pipeline 'flattening' of FHIR data
using [ViewDefinition](https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/StructureDefinition-ViewDefinition.html)
resources - [read more](views#viewdefinition-resource)
resources - [read more](../views#viewdefinition-resource)

### Loading

Expand All @@ -78,7 +78,7 @@ full", "incremental", and "merger" pipelines together.
* The Pipelines Controller is built on top of pipelines and shares many of the
same settings
* Using the controller module you can schedule periodic incremental updates or
use the [Web Control Panel](../additional#web-control-panel) to start the
use the [Web Control Panel](../../additional/#web-control-panel) to start the
pipeline
manually

Expand All @@ -90,13 +90,13 @@ Choosing the right approach, comes down to a number of factors including the
specific use-case, infrastructure constraints of the project, future scalability
requirements, and expertise of the team.

| Scenario | Approach | Considerations |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
| Simple relational database to power dashboards or reporting | Custom schema defined as ViewDefinition Resources - [see tutorial](tutorial_lossy_example/) | By design, this will provide a subset of FHIR data in flat tables |
| Horizontally scalable query architecture with lossless FHIR data | Parquet based DWH and distributed query engine (e.g. SparkSQL) - [see tutorial](tutorial_add_dashboard/) | This requires a distributed file system for Parquet files and a distributed query engine. |
| Lossless FHIR DWH with a single process query engine | Parquet based DWH with non-distributed query engine set-up (e.g. single-node Spark or DuckDB) | Scalability and performance (e.g. SparkSQL >> DuckDB) |
| Exploratory data science or ML use cases | Use the generated Parquet files which as _"near lossless"_ for enhanced data science workflows | Can either use the Parquet or custom schema to power dashboards or reports |
| Push FHIR data to a central FHIR-store (e.g., for a Shared Health Record system) | Use the Pipelines Controller to push from a FHIR source to a FHIR sink | Management of the intermediate Parquet files created as part of the pipelines |
| Scenario | Approach | Considerations |
|----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
| Simple relational database to power dashboards or reporting | Custom schema defined as ViewDefinition Resources - [see Views](../views/#query-simplification) | By design, this will provide a subset of FHIR data in flat tables |
| Horizontally scalable query architecture with lossless FHIR data | Parquet based DWH and distributed query engine (e.g. SparkSQL) - [see tutorial](../../tutorials/add_dashboard/) | This requires a distributed file system for Parquet files and a distributed query engine. |
| Lossless FHIR DWH with a single process query engine | Parquet based DWH with non-distributed query engine set-up (e.g. single-node Spark or DuckDB) | Scalability and performance (e.g. SparkSQL >> DuckDB) |
| Exploratory data science or ML use cases | Use the generated Parquet files which as _"near lossless"_ for enhanced data science workflows | Can either use the Parquet or custom schema to power dashboards or reports |
| Push FHIR data to a central FHIR-store (e.g., for a Shared Health Record system) | Use the Pipelines Controller to push from a FHIR source to a FHIR sink | Management of the intermediate Parquet files created as part of the pipelines |

## Query Simplification with Flat Views

Expand Down
2 changes: 1 addition & 1 deletion doc/docs/tutorials/add_dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ that as a bar chart over time. The query is provided below.
database to "Single Machine Test Data" or the name you chose.

Enter the following as the query as either a raw SQL-on-FHIR query or using
the [pre-defined](../../concepts/#sql-virtual-views) observation_flat views:
the [pre-defined](../../concepts/views) observation_flat views:

=== "Using SQL-on-FHIR Query"

Expand Down
4 changes: 2 additions & 2 deletions pipelines/common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@

<properties>
<c3p0.version>0.10.1</c3p0.version>
<guava.version>33.3.1-jre</guava.version>
<guava.version>33.4.0-jre</guava.version>
</properties>

<dependencies>
Expand All @@ -50,7 +50,7 @@
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client-gson</artifactId>
<version>2.7.0</version>
<version>2.7.1</version>
</dependency>

<dependency>
Expand Down

0 comments on commit 1f0acf3

Please sign in to comment.