datetime and decimal should be logical types

[damccorm]

Currently datetime and decimal are implemented as primitive types in the Java SDK. They should be logical types as documented in https://s.apache.org/beam-schemas

Imported from Jira BEAM-7554. Original Jira may contain additional context.
Reported by: bhulette.

[Abacn]

.take-issue

[Abacn]

This issue was raised again in May 22, per this comment in Jira

ilango-mediaagility:
Due to this issue, cross language transforms are affected. For example, when using ReadFromJdbc transform in python, we get the decimal and datetime as logical types, but they are encoded differently than the underlying representation types, resulting in incorrect values.

This is blocked by #21433

[Abacn]

There is a problem of replace all DATETIME fieldtype to logical type, per @TheNeuralBit:

One thing that I realized as I wrote up my PR: If we make DATETIME a logical type backed by INT64, then it will have to use VarLongCoder, rather than InstantCoder as it currently does.

It seems like that would be a problem since InstantCoder is designed to make lexicographic order correspond to chronological order.
Note that InstantCoder is not only used in processing schemas but also in coding windows:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java

Line 618 in bf39489

Instant timestamp = InstantCoder.of().decode(inStream);

Tried to replace the InstantCoder implementation using either a VarInt or a ByteArray it breaks window decoding here. Seems like there is some predefined stream not using InstantCoder.encode to pack a timestamp.

Thus I decided not to change InstantCoder at this moment, instead extend the support MicrosInstant logical type in Java sdk and DateTime logical type in python sdk, seems to be the simplest way to make xlang read/write records containing timestamp type work.

[TheNeuralBit]

Yeah I agree that it's better to just prefer logical types for multi-language transforms, rather than trying to shunt the DATETIME primitive type this way. We may want to add types for additional precisions as needed (MillInstant, NanosInstant, etc)

[Abacn]

update: Python support is complete. Leave this open for Go support