AvroTypeException for some edge cases.

[bashir2]

We don't know the root cause of this issue yet but @kimaina has reported the following exception in some special Observation resource cases:

org.apache.avro.AvroTypeException: Cannot encode decimal with scale 16 as scale 4 without rounding in field value in field quantity in field value

The full stack trace is here. This is coming from this line and as a temporary work-around, an exception handling is added around this line in PR #155. This will log the resource ID that causes this so it should be easy to find the exact resource that triggers this.

[bashir2]

I did not mean to close this (GitHub's misinterpretation of the PR comment).

[bashir2]

The root cause of this bug seems to be here in Bunsen where the schema for a decimal is set to have precision 12 and scale 4. However, this limits the domain of values that can be represented (e.g., large numbers cannot be represented). Although the schema is set like this, but the actual value returned here is not guaranteed to have this precision/scale. So for example, for large numbers we don't get any errors when turning them into BigDecimal (with parse()) but if we try to serialize these, it will throw the exceptions in this function because a number with more than 12 significant digits cannot be represented by precision 12.

It seems to me that the choice of using Avro decimal type to represent FHIR decimal type needs to be changed.

[bashir2]

Changing this to P0 since it is blocking #168 too.

[bashir2]

Update: I made this tentative fix on the Bunsen side. With that change incorporated locally, I verified that this bug will be fixed in this commit.

@kimaina, is it possible for you to patch these two commits in your local environment and test the batch pipeline with your data?

[kimaina]

@bashir2 thank you for working on this! Let me test this out then get back to you!

[kimaina]

Update: This is to confirm that this tentative fix on the Bunsen side and this commit.
fixes this issue! Thanks @bashir2 for fixing this. Let's finalize and close this ticket!