Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for text type #304

Merged
merged 43 commits into from
Nov 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
1fb9dc3
text type initial support
sirex Oct 2, 2022
4e7f760
204 text test from external source
Feb 9, 2023
b87a67d
204 text type push tests
Feb 9, 2023
1727cb8
fix
Feb 9, 2023
39bc315
204 test changelog
Feb 10, 2023
ae04840
204 test by prop select
Feb 13, 2023
a6f50aa
Testing notes for text type
sirex Feb 17, 2023
421a381
204 gramma fix, text property saving
Mar 1, 2023
1a7cebb
204 read, export, text prop
Apr 7, 2023
5231ca6
merge fix
Apr 19, 2023
0157746
Notes fixes, TODO tests
May 8, 2023
28c4ef7
text patching, sort, tests
May 15, 2023
88d7988
files naming rollback
May 15, 2023
799f1fb
merge fix
May 15, 2023
fe362fb
merge fix
May 15, 2023
30583c2
response fix
May 16, 2023
df1d895
file names rollback, and few fixes
May 16, 2023
50b4de8
tests fixes
May 17, 2023
ffb9ccb
test chunks TODO
May 17, 2023
acf5542
test with access level
May 17, 2023
e61315c
clean up
May 17, 2023
d7fc7cd
merge fix
Aug 2, 2023
04f6e3e
rollback as it is master
Aug 2, 2023
d2063dd
Merge branch 'master' into 204-add-support-for-text-type
adp-atea Oct 27, 2023
58587f5
204 transferred test
adp-atea Nov 7, 2023
3296241
Merge branch 'master' into 204-add-support-for-text-type
adp-atea Nov 7, 2023
52fe32a
204 refactored text tabular, get, write
adp-atea Nov 9, 2023
939aecb
204 added check if parent exists
adp-atea Nov 9, 2023
e8d612f
204 added kwargs to getall
adp-atea Nov 9, 2023
db1bd6a
204 refactored BaseQueryBuilder metadata transfer
adp-atea Nov 10, 2023
99b9a3c
204 added Text to sql, bug fixes
adp-atea Nov 10, 2023
0ae5fb7
204 added better support for external Sql
adp-atea Nov 13, 2023
994b8d8
204 Text html support, fixed tests
adp-atea Nov 13, 2023
ce25c9b
204 added explicitly_given False to dict manifest array type
adp-atea Nov 13, 2023
1ce75f5
204 fixed some Text env requests
adp-atea Nov 13, 2023
8924aa4
204 added more tests
adp-atea Nov 13, 2023
0d80e81
204 added exceptions
adp-atea Nov 14, 2023
78232af
204 added select validation, external push with text
adp-atea Nov 14, 2023
d49f5e7
Merge branch 'master' into 204-add-support-for-text-type
adp-atea Nov 14, 2023
d3e9b2e
204 added none checks
adp-atea Nov 14, 2023
6b5e6f6
204 added array support for text and string type
adp-atea Nov 14, 2023
ad8c2a2
Testing notes
sirex Nov 21, 2023
9a622c1
Merge branch 'master' into 204-add-support-for-text-type
sirex Nov 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
244 changes: 244 additions & 0 deletions notes/types/text.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,244 @@
# 2023-02-17 16:10

git log -1 --oneline
#| 0ebfd06 (HEAD -> geometry-z, origin/master, origin/HEAD, master) Check if public data was written

# notes/docker.sh Start docker compose
# notes/postgres.sh Reset database

INSTANCE=types/text
DATASET=$INSTANCE
# notes/spinta/server.sh Configure server



cat > $BASEDIR/manifest.txt <<EOF
d | r | b | m | property | type | ref | source | access
$DATASET | | | |
| data | sql | db | |
| | | Country | | id | countries |
| | | | id | integer | | id | open
| | | | name | text | | | open
| | | | name@lt | string | | name_lt | open
| | | | name@en | string | | name_en | open
EOF
poetry run spinta copy $BASEDIR/manifest.txt -o $BASEDIR/manifest.csv
cat $BASEDIR/manifest.csv
poetry run spinta show $BASEDIR/manifest.csv

ed -vs $BASEDIR/config.yml <<EOF
/backends:/a
db:
type: sql
dsn: sqlite:///$PWD/$BASEDIR/db.sqlite
.
wq
EOF
cat $BASEDIR/config.yml

test -f $BASEDIR/db.sqlite && rm $BASEDIR/db.sqlite
sqlite3 $BASEDIR/db.sqlite <<'EOF'
CREATE TABLE countries (
id INTEGER PRIMARY KEY,
name_lt TEXT,
name_en TEXT
);
INSERT INTO countries VALUES (1, 'Lietuva', 'Lithuania');
INSERT INTO countries VALUES (2, 'Latvija', 'Latvia');
EOF
sqlite3 $BASEDIR/db.sqlite "SELECT * FROM countries;"
#| +----+---------+-----------+
#| | id | name_lt | name_en |
#| +----+---------+-----------+
#| | 1 | Lietuva | Lithuania |
#| | 2 | Latvija | Latvia |
#| +----+---------+-----------+

# notes/spinta/server.sh Run migrations

psql -h localhost -p 54321 -U admin spinta -c '\dt public.*'
psql -h localhost -p 54321 -U admin spinta -c '\d "'$DATASET'/Country"'
#| Table "public.types/text/Country"
#| Column | Type | Collation | Nullable | Default
#| -----------+-----------------------------+-----------+----------+---------
#| _txn | uuid | | |
#| _created | timestamp without time zone | | |
#| _updated | timestamp without time zone | | |
#| _id | uuid | | not null |
#| _revision | text | | |
#| id | integer | | |
#| name | jsonb | | |
#| Indexes:
#| "types/text/Country_pkey" PRIMARY KEY, btree (_id)
#| "ix_types/text/Country__txn" btree (_txn)
#| "types/text/Country_id_key" UNIQUE CONSTRAINT, btree (id)

# notes/spinta/server.sh Run server

http DELETE "$SERVER/$DATASET/:wipe" $AUTH
test -f $BASEDIR/push/localhost.db && rm $BASEDIR/push/localhost.db

poetry run spinta push $BASEDIR/manifest.csv -o test@localhost

# notes/spinta/client.sh Configure client

http GET "$SERVER/$DATASET/Country"
#| HTTP/1.1 200 OK
#|
#| {
#| "_data": [
#| {
#| "_id": "e4a346e9-5848-4799-af2a-18b4e2acb146",
#| "_page": "WzEsICJlNGEzNDZlOS01ODQ4LTQ3OTktYWYyYS0xOGI0ZTJhY2IxNDYiXQ==",
#| "_revision": "33e41832-115d-4f07-a514-e5cc02708370",
#| "_type": "types/text/Country",
#| "id": 1,
#| "name": "Lietuva"
#| },
#| {
#| "_id": "f077eee9-d827-4451-bc0a-7ff1da18a36e",
#| "_page": "WzIsICJmMDc3ZWVlOS1kODI3LTQ0NTEtYmMwYS03ZmYxZGExOGEzNmUiXQ==",
#| "_revision": "2791ea9c-1c80-4854-a017-c8aed9e82301",
#| "_type": "types/text/Country",
#| "id": 2,
#| "name": "Latvija"
#| }
#| ]
#| }

http GET "$SERVER/$DATASET/Country" Accept-Language:en
#| HTTP/1.1 200 OK
#|
#| {
#| "_data": [
#| {
#| "_id": "e4a346e9-5848-4799-af2a-18b4e2acb146",
#| "_page": "WzEsICJlNGEzNDZlOS01ODQ4LTQ3OTktYWYyYS0xOGI0ZTJhY2IxNDYiXQ==",
#| "_revision": "33e41832-115d-4f07-a514-e5cc02708370",
#| "_type": "types/text/Country",
#| "id": 1,
#| "name": "Lithuania"
#| },
#| {
#| "_id": "f077eee9-d827-4451-bc0a-7ff1da18a36e",
#| "_page": "WzIsICJmMDc3ZWVlOS1kODI3LTQ0NTEtYmMwYS03ZmYxZGExOGEzNmUiXQ==",
#| "_revision": "2791ea9c-1c80-4854-a017-c8aed9e82301",
#| "_type": "types/text/Country",
#| "id": 2,
#| "name": "Latvia"
#| }
#| ]
#| }

http GET "$SERVER/$DATASET/Country/e4a346e9-5848-4799-af2a-18b4e2acb146"
#| {
#| "_id": "e4a346e9-5848-4799-af2a-18b4e2acb146",
#| "_revision": "33e41832-115d-4f07-a514-e5cc02708370",
#| "_type": "types/text/Country",
#| "id": 1,
#| "name": {
#| "en": "Lithuania",
#| "lt": "Lietuva"
#| }
#| }


http GET "$SERVER/$DATASET/Country?select(name)&format(ascii)"
#| name
#| -------
#| Lietuva
#| Latvija

http GET "$SERVER/$DATASET/Country?select(name@lt)&format(ascii)"
#| name.lt
#| -------
#| Lietuva
#| Latvija

http GET "$SERVER/$DATASET/Country?select(name@lt, name@en)&format(ascii)"
#| name.lt name.en
#| ------- ---------
#| Lietuva Lithuania
#| Latvija Latvia

http GET "$SERVER/$DATASET/Country?name='Lietuva'&select(name)&format(ascii)" Accept-Language:lt
#| name
#| -------
#| Lietuva

http GET "$SERVER/$DATASET/Country?name@en='Lithuania'&select(name)&format(ascii)" Accept-Language:lt
#| name
#| -------
#| Lietuva

http GET "$SERVER/$DATASET/Country?select(name)&sort(name)&format(ascii)" Accept-Language:lt
#| name
#| -------
#| Latvija
#| Lietuva

http GET "$SERVER/$DATASET/Country?select(name)&sort(-name@en)&format(ascii)" Accept-Language:lt
#| name
#| -------
#| Lietuva
#| Latvija

http GET "$SERVER/$DATASET/Country?format(rdf)"
#| <?xml version="1.0" encoding="UTF-8"?>
#| <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:pav="http://purl.org/pav/" xmlns="http://localhost:8000/">
#| <rdf:Description rdf:about="/types/text/Country/e4a346e9-5848-4799-af2a-18b4e2acb146" rdf:type="types/text/Country" pav:version="33e41832-115d-4f07-a514-e5cc02708370">
#| <_page>WzEsICJlNGEzNDZlOS01ODQ4LTQ3OTktYWYyYS0xOGI0ZTJhY2IxNDYiXQ==</_page>
#| <id>1</id>
#| <name>Lietuva</name>
# TODO: Should be
# <name xml:lang="lt">Lietuva</name>
# https://github.com/atviriduomenys/spinta/issues/549
#| </rdf:Description>
#| <rdf:Description rdf:about="/types/text/Country/f077eee9-d827-4451-bc0a-7ff1da18a36e" rdf:type="types/text/Country" pav:version="2791ea9c-1c80-4854-a017-c8aed9e82301">
#| <_page>WzIsICJmMDc3ZWVlOS1kODI3LTQ0NTEtYmMwYS03ZmYxZGExOGEzNmUiXQ==</_page>
#| <id>2</id>
#| <name>Latvija</name>
#| </rdf:Description>
#| </rdf:RDF>

xdg-open http://localhost:8000/$DATASET/Country
xdg-open http://localhost:8000/$DATASET/Country/:changes/-10

http GET "$SERVER/$DATASET/Country/:changes/-10"
#| {
#| "_data": [
#| {
#| "_cid": 1,
#| "_created": "2023-11-21T07:14:59.642472",
#| "_id": "e4a346e9-5848-4799-af2a-18b4e2acb146",
#| "_op": "insert",
#| "_revision": "33e41832-115d-4f07-a514-e5cc02708370",
#| "_txn": "0770f48f-be39-4593-9341-deda7cef0a85",
#| "id": 1,
#| "name": {
#| "en": "Lithuania",
#| "lt": "Lietuva"
#| }
#| },
#| {
#| "_cid": 2,
#| "_created": "2023-11-21T07:14:59.642472",
#| "_id": "f077eee9-d827-4451-bc0a-7ff1da18a36e",
#| "_op": "insert",
#| "_revision": "2791ea9c-1c80-4854-a017-c8aed9e82301",
#| "_txn": "0770f48f-be39-4593-9341-deda7cef0a85",
#| "id": 2,
#| "name": {
#| "en": "Latvia",
#| "lt": "Latvija"
#| }
#| }
#| ]
#| }
http GET "$SERVER/$DATASET/Country/:changes/-10?format(ascii)"
#| _cid _created _op _id _txn _revision id name
#| ---- -------------------------- ------ ------------------------------------ ------------------------------------ ------------------------------------ -- ----
#| 1 2023-11-21T07:14:59.642472 insert e4a346e9-5848-4799-af2a-18b4e2acb146 0770f48f-be39-4593-9341-deda7cef0a85 33e41832-115d-4f07-a514-e5cc02708370 1 ∅
#| 2 2023-11-21T07:14:59.642472 insert f077eee9-d827-4451-bc0a-7ff1da18a36e 0770f48f-be39-4593-9341-deda7cef0a85 2791ea9c-1c80-4854-a017-c8aed9e82301 2 ∅
# TODO: `name` should be expanded into `name.en` and `name.lt`.
# https://github.com/atviriduomenys/spinta/issues/550
Loading