Add support for text type

[sirex]

In manifest there is a possibility to separate natural language text properties from any other string types.

Specification: https://atviriduomenys.readthedocs.io/dsa/duomenu-tipai.html#tekstiniai-duomenys

text type should be implemented only on postgresql backend.

On PostgreSQL text type properties should be saves as JSONB objects, like this:

{
  "lt": "Pavyzdys",
  "en": "Example",
  "": "Example",
}

Here "" key will be used, when language code for property is not specified.

• Add support for text type.
• Parse name@lang string and store language code as type parameter.
• Implement write operation on text types.
• Implement read operation on text types.
  • Select language by Accept-Language.
  • Fall back to default language specified in configuration.
• Test if patching and changelog works properly.
• Test if reading from external sources works properly.
• Test if pushing works properly.
• Add support for cases, when language code is not given, in this case it should save under empty language code and level must be less than or equal to 3. Level 2 if multiple languages are mixed.
• Test if select(prop@lang) works.
• Test if prop@lang=value works.
• Test if sort(prop@lang) works.

Links

• https://w3c.github.io/json-ld-syntax/#string-internationalization

[sirex]

A suggestion. Maybe we should separate text from string types? Like here:

d | r | b | m | property | type   | ref | source  | uri
example                  |        |     |         |
                         | prefix | dct |         | http://purl.org/dc/terms/
  |   |   | City         |        |     | CITIES  |
  |   |   |   | name     | text   |     |         | dct:title
  |   |   |   | name@en  | string |     | NAME_EN |
  |   |   |   | name@lt  | string |     | NAME_LT |

Here we have a name of text type and we can specify uri for name, which also includes all language tags. We have to specify uri only for text type, because RDF has built-in support for languages.

name@en and name@lt are defined as string, but since property name has language tag, we know, that these strings are part of name.

Also text type could be implied, without explicitly declaring it, like this:

d | r | b | m | property | type   | ref | source
example                  |        |     |
  |   |   | City         |        |     | CITIES
  |   |   |   | name@en  | string |     | NAME_EN
  |   |   |   | name@lt  | string |     | NAME_LT

Here, we know, that name is a text, because property names have language tags.

Regarding source, if we would have following manifest table:

d | r | b | m | property | type   | ref | source | level
example                  |        |     |        |
  |   |   | City         |        |     | CITIES |
  |   |   |   | name     | text   |     | NAME   | 2

If we don't know the language or data has mixed languages, then we can read data directly into text type, this would result in:

{"name": {"": "Vilnius"}}

Empty language tag does not mean a default language, it means, that language is unknown.

When reading or writing, text type should accept two forms:

Implicit form:

{"name": "Vilnius"}

Explicit form:

{"name": {"lt": "Vilnius"}}

When querying data, for example here:

/example/City?select(name)&name.startswith("V")&sort(name)

we use implicit form, since language tag is not specified. When language tag is not specified, we always detect language from:

• Accept-Language header, to pick client prefered language.
• If client does not prived Accept-Language or if prefered languaged is not available (is not declared in manfiest), then fall back to default language specified in Spinta languages configuration and pick first language declared in manifest.

For example if we write data like this:

{"name": "Vilnius"}

Then we use language detection described above and set client prefered or system default language. If client prefered en and if en is declared in manifest, then we convert above value into:

{"name": {"en": "Vilnius"}}

But if explicif form is given, then we do not do any detection, and used specified language. If specified language is not declared, then we raise an error.

If we want to specify an unknown language, we use C as language code in @ expressions, for example:

/example/City?select(name@C)&name@C.startswith("V")&sort(name@C)

Here, we explicily refer to unknown language, which internaly is represented as:

{"name": {"": "Vilnius"}}