- Backwards incompatible:
- conversion of XSD schemas to DSA manifests in an improved way. (#842)
- support for language tag for properties. (#582)
New features:
- Added support for Object type with external Sql backend (#973).
Backwards incompatible:
SqlAlchemyKeyMap synchronization no longer uses individual transactions for each synchronization action. Now it batches the actions under multiple transactions. By default it batches 10000 rows. In order to change that value, set sync_transaction_size in config under your keymaps configuration (#1011).
Like so:
keymaps: default: type: sqlalchemy dsn: ... sync_transaction_size: 20000
Changed postgresql naming convention. This will result in old tables having incorrect constraint and index names. spinta migrate should be able to find most of them (P#153).
AccessLog no longer stores scope field on every request. Instead it will store token field (token JTI value). In order to track what scopes token uses, now we log auth requests (/auth/token), which will store list of scopes. This change should reduce the spam in logging and reduce log file size.
In order track unique token identifiers, JTI field has been added to all new tokens (meaning old tokens, that still do not have the field, will not be properly logged) (#1003).
Improvements:
- SqlAlchemyKeyMap now uses batch transactions to synchronize data, which greatly improves performance (#1011).
- added enum level support, allowing to indicate a level for enum. (#982)
- Standardized postgresql naming convention, now all new constraints and indexes should follow same naming scheme (P#153).
- spinta migrate now tries to rename constraints and indexes (if the name only changed) instead of dropping them and adding them with correct name (P#153).
- JWT tokens now also store JTI claim (#1003).
- AccessLog now has auth logging (#1003).
Bug fix:
- Postgresql summary now properly handles tables with long names (P#160).
- Fixed various cases where migrate command would not take into account truncated names (P#153).
Backwards incompatible:
- Keymap synchronization now uses sync_page_size config argument to limit amount of data being fetched with a single request. This will result in more actions being called to remote server. If keymap synchronization takes too long to start the process, reduce sync_page_size value. Keep in mind, that lower values reduce performance and increase server load (#985).
- push command now has explicit timeouts set for requests. Previously, there were no timeouts set for requests, which meant that execution time was unlimited. After the changes the default values are 300 seconds (5min) for read and 5 seconds for connect timeouts. The timeout values can be adjusted using --read-timeout and --connect-timeout push command options (#662).
New features:
- Add -d --datasets option to migrate command (#935).
- Add export cli command, that will export data to specified format (#960).
- Add keymap sync command (#666).
- Add --read-timeout, --connect-timeout options to spinta push command (#662).
Improvements:
- Keymap synchronization now uses pagination to fetch data (#985).
New features:
- Added support for Denorm type migrations (#932).
Improvements:
- Added better support for migrations with nested data types (#722).
- Added a check for reading client data files, to provide better error messages (#933).
- Added scope information to access logs (#903).
- Improved summary query memory usage (#955).
Bug fix:
- Resolved ambiguity warning messages (#895).
- Fixed Denorm properties being mapped to Ref foreign key migrations (#722).
- Fixed memory leak caused by resource_filename function (#954).
Bug fix:
- Removed pymssql library from requirements (was added in previous version by accident).
Backwards incompatible changes:
- wait command no longer raises exceptions, when it fails to connect to backend (PostgresSql and Sql). This means that you will only know if backend failed to connect, when you try to call transaction or begin methods, which should be called on every request (#730).
- Changed minimum starlette version requirement to 0.40>= (fixes vulnerability issue). More about it: https://github.com/encode/starlette/security/advisories/GHSA-f96h-pmfr-66vw
New features:
- Added support for literal values in property prepare expression (#670).
- Added uuid data type (#660).
Improvements:
- Added backend``transaction and begin method validations (PostgresSql and Sql backends). When launching spinta server, wait command no longer raises exceptions if it failed to connect to backend (#730).
- Added the ability for 'Backref' to have nested properties; improved 'Backref' and 'ArrayBackref' handling (#664).
Backwards incompatible changes:
- You can no longer directly set Ref foreign key values to None. Meaning you cannot set "ref": {"_id": None}. Now, if you want to unassign Ref value, you have to set it to None ("ref": None), it will also now set all nested values (Denorm) to None as well, this new feature now ensures, that there cannot be floating Denorm values when trying to remove references (#846).
Improvements:
- Added removal of duplicate models when converting XSD to DSA even when source is different (#787).
- Improved invalid scope error messaging for token auth (#537).
- Added ability to remove all nested property values for Ref type, when assigning None to the value itself (#846).
Bug fixes:
- Fixed a bug in XSD->DSA conversion, where properties need to become arrays in a choice which has maxOccurs="unbounded" (#837).
- Fixed checksum() function bug, where it tried to calculate checksums before converting data from backend specific to python types (#832).
- Fixed an oversight where geoalchemy2 values were propagated to prepare_dtype_for_response instead of being converted to backend indifferent type (shapely.geometry.base.BaseGeometry) (#832).
- Fixed errors when Ref changelog values were incorrect. Now, if changelog ref _id, or ref itself is "", it assumes that it is supposed to be None (#556).
- Fixed Ref value unassignment not updating the values in database (#846).
Improvements:
- Reverted github actions postgresql version to 11, until production server is updated to 16, so we don't get similar issues again (#827).
Bug fixes:
- Fixed summary for Geometry not working with older than 16 postgresql version (#827).
Bug fixes:
- Fixed api inspect clean_up function failing when there are exceptions while reading manifest files (#813).
- Fixed client add not finding config_path when using config.yml instead of setting it with -p (#818).
Backwards incompatible changes:
- Changed pymongo version requirement from "*" to "<=4.8.0". Version 4.9.0 changed import paths, that broke spinta (#806).
Improvements:
- Added support for negative float values in starlette float routing (use spinta_float instead of float type) (#781).
- Changed manifests.default.backend config value from '' to 'default'. Now if nothing is set, default backend will be MemoryBackend instead of nothing (#798).
- Added removal of duplicate models when converting XSD to DSA (#752).
Bug fixes:
- Fixed _srid routing error, when using negative float values as coordinates (#781).
- Fixed Geometry boundary check not respecting SRID latitude and longitude order (used to always assume, that x = longitude, y = latitude, now it will try to switch based on SRID) (#737).
- Fixed some errors when trying to access api endpoints, while server is running with default config settings (#798).
- Fixed a problem in PropertyReader and EnumReader where enums were always added to the top level property (#540).
Backwards incompatible:
- Spinta no longer automatically migrates clients structure (#122). Now you have to manually use spinta upgrade command to migrate files. Meaning if there are issues with clients file structure you will going to get errors, suggesting to fix the problem, or run spinta upgrade command (#764).
Improvements:
- Changed postgresql github actions and docker compose version to 16-3.4 (P#129).
- Changed report bug link to atviriduomenys@vssa.lt email (#758).
New features:
- Added spinta upgrade command, that will migrate backwards incompatible changes between versions (#764).
- Use spinta upgrade to run all scripts.
- spinta upgrade -m <script_name> to run specific script.
- spinta upgrade -f to skip all checks and forcefully run scripts.
- spinta upgrade -d to run destructive mode, which, depending on script, will override existing changes. Only use destructive mode, if you know what will be changed, and you have made backups.
- Added clients migrate script to spinta upgrade command (#764).
Main goal is to migrate client files from old structure to newly introduced one in #122 task.
- You can specify it with spinta upgrade -r clients command.
- Use spinta upgrade -r clients -f if you want to make sure that all files are migrated correctly. It will skip already migrated files and update keymap.yml.
- spinta upgrade -r clients -f -d will override any new files that match old ones. This is destructive and there are no rollbacks for it, so only use it if you have backups and understand what will be changed.
Bug fixes:
- Added missing cluster limit to :summary for Geometry type properties. Now it's set to 25 clusters (P#130).
Improvements:
- Improved performance of PostgreSQL and SQL backend getall functions (#746).
Improvements:
- Nested properties for XSD. (#622).
Bug fixes:
- Removed from mypy.dmypy.client import request import from spinta/components.py.
Backwards incompatible:
- Renamed push_page_size config field to default_page_size (#735).
Improvements:
- Changed default config sync_page_size and default_page_size parameters to be 100000 instead of 1000 (#735).
New features:
- Added enable_pagination config field, which will enable or disable default pagination behaviour. Request and schema specifications take priority, meaning even if enable_pagination is set to False, you can still specify page(disable:false) to enable it for specific requests (#735).
Backwards incompatible:
- Changed spinta_sqlite driver name to spinta. Old naming was unnecessary since you needed to use sqlite+spinta_sqlite:///..., now you can just use sqlite+spinta:///... (#723).
- spinta push state database now will always going to append sqlite+spinta:/// prefix, instead of sqlite:///. This ensures, that sqlite version is now dependant on sqlean library, instead of taking default python sqlite version (makes it easier to ensure, that users are using correct version of sqlite) (#723).
- Changed sqlalchemy default sqlite driver to SQLiteDialect_spinta (instead of SQLiteDialect_pysqlite). Meaning every time you use sqlite:///... it will default to spinta driver, instead of pysqlite (default sqlalchemy) (#723).
Improvements:
- Writing InternalSQLManifest now is done using transaction, meaning if there are errors, it will rollback any changes (This is useful when doing copy on already existing structure, since it clears all old data before writing new) (#715).
- Changed state db, to always use spinta sqlite driver (#723).
Bug fixes:
- Fixed InternalSQLManifest structure being fetched without index order (#715).
New features:
- Added support for eq, & and | operators to Dask backend (#702).
Bug fixes:
- Fixed formula being ignored when using inspect (#685).
- Fixed errors with different formats when returning empty data (#684).
- Fixed keymap.yml not updating mapping when changing client_name (#688).
- Fixed error when opening changes in html format, when there is no select and you have only one language given to Text property (#693).
- Fixed assertion error when only selecting not expanded array (#696).
- Fixed issue, where sometimes json blank nodes gets discarded and return empty dict (#699).
- Fixed error when trying to use Dask backend & and | operators (#705).
Backwards incompatible changes:
- Changed starlette version requirement from "*" to ">=0.22". From version 0.22.0 starlette added better compatibility support for AnyIO.
Bug fixes:
- Fixed getone with jsonl format (#679)
- Rolled back Templates warning fixes (caused errors with older starlette versions) (#679)
Bug fixes:
Changed json Geometry type converter import to BaseGeometry (#673)
Backwards incompatible changes:
- When migrating from version of spinta, where push pagination was not supported, to a version, where it is, the old push state database structure is outdated and it can result in getting InfiniteLoopWithPagination or TooShortPageSize errors (new push state database structure now stores pagination values, while old one does not). With the addition of (P#98) change, you now are able to run push --sync command to synchronize push state database. It is important to note that it will also update pagination values, which could fix some of the infinite loop errors.
- With (P#98) change, internal will no longer disable pagination when page key types are not supported. Before this change, when model's page went through link process, if there was any page keys, that were not supported, pagination was disabled, no matter what type of backend is used. Since all internal backends support _id property, which is always present and unique, if we find page keys that are not supported, we can always force pagination using _id. This results in that all of the requests will now by default going to be sorted by _id property. Important to note, if we use sort with unsupported keys, pagination is still going to be disabled.
New features:
- Mermaid format support for ability to create class diagrams (#634).
- Parametrization support for XML and JSON external backends (#217, #256).
- Added new manifest backend for XSD schemas (#160).
- Added distinct() function to model.prepare (#579).
- Added push state database synchronization. (P#98)
- Added checksum() select function to PostgreSQL backend. (P#98)
Improvements:
- Added ResultBuilder support to PostgreSQL backend, also changed it's QueryBuilder to work like external SQL. (P#98)
- Changed internal backend page assignment logic to default to _id property, if any of the page keys are not supported. (P#98)
- Added proper support for functions in select() expressions (P#100).
Bug fixes:
- Migrate internal backend changed types (#580).
- Added support for language tags in RDF strings (#549).
- Show values of text type in tabular output (#550, #581).
- Added support for PostgreSQL OID type (#568).
- Fixed sorting issue with MySQL and MSSQL external backends (P#90).
- Fixed issue with open transactions when writing data (P#92).
- Fixed issue with outdated page key in push state tables (P#95).
- Words in dataset names separated by underscores. (#626).
- Added support for getone for sql backend (#513).
- Fixed Ref id mapping with non-primary keys when primary keys were not initialized (#653).
- Fixed issue with Geometry type conversion when pushing data (#652).
- Fixed issue with Geometry bounding box check not applying CRS projection (#654).
New features:
- Add possibility to update manifest via HTTP API, without restarting server (#479).
Bug fixes:
- Fixed error with index names exceeding 63 character limit on PostgreSQL (#566).
- Set WGS84 SRID for geometry tupe if SRID is not given as specified in documentation (#562).
Backwards incompatible changes:
- Check geometry boundaries (#454). Previously you could publish spatial data, with geometries out of CRS bounds, now if your geometry is out of CRS bound, you will get error. To fix that, you need to check if you specify correct SRID and if you pass geometries according to specified SRID specifikation.
New features:
- New type of manifest read from database, this enables live schema updates (#113).
- Automatic migrations with spinta migrate command, this command compares manifest and database schema and migrates database schema, to match given manifest table (#372).
- HTTP API for inspect (#477). Now it is possible to inspect data source not only from CLI, but also via HTTP API.
Improvements:
- Generate next page only for last object (#529).
Bug fixes:
- Fixing denormalized properties (#379, #380).
- Fix join with base model (#437).
- Fix WIPE timeout with large amounts of related data (#432). This is fixed by adding indexes on related columns.
- Fix changed dictionaly size error (#554).
- Fix pagination infinite loop error (#542).
New features:
- Add new text type (#204).
Bug fixes:
- Fix client files migration issue (#544).
- Fix pagination infinite loop error (#542).
- Do not sync keymap on models not required for push operation (#541).
- Fix /:all on RDF format (#543).
Backwards incompatible changes:
With addition of new API for client management, structure how client files are stored, was changed.
Previously clients were stored in SPINTA_CONFIG_PATH like this:
clients/ └── myclient.yml
Where myclient was usually a client name if given, if not given it was an UUID.
Client file content looked like this:
client_id: myclient client_secret: secret client_secret_hash: pbkdf2$sha256$346842$yLpG_ganZxGDuwzIsED4_Q$PBAqfikg6rvXzg2_s74zIPlGGilA5MZpyCyTjlEuzfI scopes: - spinta_getall - spinta_getone
Now clients/ folder structure looks like this:
├── helpers/ │ └── keymap.yml └── id/ └── 7e/ └── 1c/ └── 0625-fd42-4215-bd86-f0ddff04fda1.yml
In the new structure, all clients are stored under id/ folder and client files are named after client_id uuid form.
In the example above 7e1c0625-fd42-4215-bd86-f0ddff04fda1 is a client_id.
client_id now a clear meaning ant now it is just a client id in UUID form. Client name is stored in client_name. If client name is not given, then client_name is the same as client_id.
There is another file called helpers/keymap.yml, that looks like this:
myclient: 7e1c0625-fd42-4215-bd86-f0ddff04fda1
This file, stores a mapping of client names as an index to help locating clients by name faster.
Client names can change, but id can't.
Structure of client file mostly stays the same, except client_id is not only id in UUID form and a new option client_name was added to store client name. For example content of id/7e/1c/0625-fd42-4215-bd86-f0ddff04fda1.yml now looks like this:
client_id: 7e1c0625-fd42-4215-bd86-f0ddff04fda1 client_name: myclient client_secret: secret client_secret_hash: pbkdf2$sha256$346842$yLpG_ganZxGDuwzIsED4_Q$PBAqfikg6rvXzg2_s74zIPlGGilA5MZpyCyTjlEuzfI scopes: - spinta_getall - spinta_getone
New features:
- Add possibility to manage clients via API (#122).
Improvements:
- Add better support for denormalized properties (#397).
Bug fixes:
- Fix error on object counting when running spinta push (#535).
- Restore recognition of views in spinta inspect (#476).
- Fix single object change list rendering in HTML format (#459).
Bug fixes:
- Fix error in CSV containing NULL data (#528).
- Fix swap() containing quotes (#508).
- Fix UnauthorizedKeymapSync error on spinta push command (#532).
New features:
- Add support for array type (#161).
- Add support for backref type (#96).
- Add support for XML resources (#217).
- Add support for JSON resources (#256).
- Add support for CSV resources (#268).
Improvements:
- Add support for custom subject URI in RDF/XML format (#512).
Bug fixes:
- Fixed pagination error with date types (#516).
- Fix issue with old SQLite versions used for keymaps (#518).
- Fix summary bbox function with negative values (#523).
New features:
- Pagination, this should enable possibility to push large amounts of data (#366).
- Push models using bases (#346, #391).
- Sync push state from push target (#289).
- Add support for non-primary key refs in push (#345).
- Push models with external dependencies (#394).
- swap() function (#508).
New features:
Bug fixes:
New features:
Improvements:
- Delete push target objects in correct order (#458).
Bug fixes:
- Add support for Oracle RAW type (#493).
Improvements:
- Recognize Oracle ROWID data type.
New features:
- Add support for param dimension (#210).
- Spinta inspect now supports JSON data as schema source (#98).
Improvements:
Bug fixes:
- Fix spint push with ref type set to level 3 or below (#460).
- Automatically add unique constraints for all primary keys specified in model.ref (#371).
New features:
- Add support for reading data from models with base (#273).
- Add support for unique constraints in tabular manifests (#148).
Improvements:
- Much better implementation for updating manifest files from SQL as data source (#364).
- Show better error messages on foreign key constraint errors (#363).
- Return a non-zero error code if spinta push command fails with an error (#423).
- Add support for older SQLite versions (#411).
Bug fixes:
- Correctly handle level 3 references, when referenced model does not have a primary key or property references a non-primary key (#400).
- WIPE command now works on tables with long names (#431).
Bug fixes:
- Fix issue with order of axes in geometry properties (#410).
- Fix write operations models containing geometry properties (#417, #418).
Bug fixes:
- Fix issue with dask/pandas version incompatibility (dask#10164).
Improvements:
- Add support for point(x,y) and cast() functions for sql backend (#407).
Bug fixes:
- Error when loading manifest from XLSX file, where level is read as integer (#405).
Bug fixes:
- Correctly handle cases, when a weak referece, references a model, that does not have primary key specified, in that case _id is used as primary key (#399).
Improvements:
- Multiple improvements in spinta push command (#311):
- New --no-progress-bar option to disable progress bar, this also skips counting of rows, which can be slow in some cases, for example when reading data from views (#332).
- New --retry-count option, to repeat push operation only with objects that ended up in an error on previous push. By default 5 times are retried.
- New --max-error-count option, to stop push operation after specified number of errors, by default 50 errors is set.
- Now instead of sending upsert, push became more sofisticated and sends insert, patch or delete.
- If objects were deleted from source, they are also deleted from target server.
- Errors are automatically retried after each push.
- Now it is possible to reference external models, this is done by specifying 3 or lower data maturity level. When property.level is set to 3 or lower for ref type properties, local values are accepted, testing notes notes/types/ref/external (#208).
- Now it is possible to specify required properties in property.type (#259).
- Specifying SRID for geometry type data on writes is no longer required (#330).
- Now it is pssible to specify geometry(geometry) and geometry(geometryz) types.
- base dimension is now supported in tabular manifest files (#325), but reading and writing to models with base is still not fully implemented.
- Support for new RDF format was added (#308).
Bug fixes:
- New ascii table formater, that should fix memory issues, when large amounts of data are downloaded (#359).
- Fix order logitude and latidude when creatling links to OSM maps (#334).
- Add possibility to explicitly select _revision (#339).
Bug fixes:
- Convert a non-WGS coordinates into WGS, before giving link to OSM if SRID is not given, then link to OSM is not added too. Also long WKT expressions like polygons now are shortened in HTML output (#298).
Improvements:
- Add pid (process id) to request messages in access log.
Bug fixes:
- Fix recursion error on getone (#255).
Improvements:
- Add support for comments in resources..
Improvements:
- Add support for HTML format in manifest files, without actual backend implementing it. (#318).
Improvements:
- Add memory usage logging in order to find memory leaks (#171).
Bug fixes:
- Changes loads indefinitely (#291). Cleaned empty patches, fixed :/changes/<offset> API call, now it actually works. Also empty patches now are not saved into the changelog.
- wipe action, now also resets changelog change id.
Bug fixes:
- Correctly handle invalid JSON responses on push command (#307).
- Fix freezing, when XLSX file has large number of empty rows.
Bug fixes:
- Incorrect enum type checking (#305).
New features:
- Check enum value to match property type and make sure, that level is not filled for enums.
Bug fixes:
- Correctly handle situation, when no is received from server (#301).
Improvements:
- More informative error message by showing exact failing item (#301).
- Upgrade versions of all packages. All tests pass, but this might introduce new bugs.
- Improve unit detection (#292). There was an idea to disable unit checks, but decided to give it another try.
New features:
Bug fixes:
- Error on getone request with ascii format (#52).
New features:
- Allow to use existing backend with -r option (#231).
- Add non-SI units accepted for use with SI (#214).
- Add uri type (#232).
Bug fixes:
- Allow NULL values for properties with enum constraints (#230).
But fixes:
- Fix bug with duplicate _id's (#228).
But fixes:
New features:
- Add time type support (#223).
New features:
- Add support for geometry data type in SQL data sources (#220).
Bug fixes:
- Fix KeyError issue when joining two tables (#219).
Bug fixes:
- Fix errr on select(left.right) when left has multiple references to the same model (#211).
- Fix geojson resource type (#215).
Bug fixes:
New features:
- Add support for comments in tabular manifest files.
Bug fixes:
- Fix sql backend join issue, when same table is joined multiple times.
- Fix ref html rendering with null values.
- Fix ref and file rendering on csv and ascii formats.
New features:
- Add cast() function for sql backend.
Improvements:
- Do not output resources with spinta copy --no-source.
New features:
- Add spinta token get command to receive access token using credentials from ~/.config/spinta/credentials.cfg file.
- Add support for prefixes on dataset dimension.
Improvements:
- Show a human readable error message when a property is not found on a sql backend.
Backwards incompatible changes:
- Some reserved properties were changed in :changes endpoint:
- _id -> _cid
- _rid -> _id
- _id -> name was renamed in :ns endpoint.
- In :ns endpoint title is no longer populated with name and is empty if not explicitly specified.
New features:
- Add geometry type support with PostGIS. Now it is possible to store spatial data.
- Add --dry-run option to spinta push. This will run whole push process, but does not send data to the target location. Useful for testing push.
- Add --stop-on-error option to spinta push. This will stop push process when first error is encountered.
Refactoring:
- Exporting data to variuos formats and specifically HTML format was heavilly refactored. HTML format is mostly rewritten.
Bug fixes:
- Fix spinta inspect when updating existing manifest and a property with the ref type has changed.
Refactoring:
- Unify manifest loading and configuration. Now more placed uses unified configure_rc function for loading and configuring Spinta.
- Add possibility to load manifest from a file stream, without specifying file name. Currently this is not yet exposed via CLI interface.
Backwards incompatible changes:
- Refactored accesslog, now accesslog only logs information about a request, not a response content. Previously whole response content was logged, which created huge log files pretty quickly. Now logs should be a lot smaller. But information about each individual object accessed is no longer available. (#97)
New features:
- Add support for units in property.ref.
Improvements:
- spinta run no longer requires setting AUTHLIB_INSECURE_TRANSPORT=1 environment variable, it is set internally.
Bug fixes:
- Fix incorrect parsing of null, false and true.
Backwards incompatible changes:
- When returning error messages now eid became string, previously it was an integer.
New features:
- Add support for enums on datasets.
- Add support for type arguments, for example geometry(point). But currently type arguments are not interpreted in any way.
- Added geojson to list of supported backend, but actual geojson backend is not yet implemented.
Improvements:
- When reading manifest from XLSX show tab number.
Bug fixes:
- Fix copy command to render -2 as is, instead of negative(2).
- Fix enum with 0 as value, before this fix, enum items with 0 was ignored.
Backwards incompatible changes:
- Configuration reader now assumes, that a required configuration parameter is not given if it is None. Previously zeros, empty strings or lists were considered as if required value was not given. But zero or an empty list can be a valid given value. Since all tests pass I assume, this change should not create any issues.
New features:
- Allow unknown columns to be added at the end of manifest table.
Bug fixes:
- Fix a bug related with dynamic manifest construction from command line arguments. Now dynamic manifest uses inline manifest type, which now supports sync parameter. Also simplified code responsible for dynamic manifest building.
- Fix a bug on external sql backend in dynamic query construction from related models with filters. Bug appeared only of a model had more than one related models (#120).
- Fix a bug on external sql backend, when select was used with joins to related tables.
Backwards incompatible changes:
- Use different push state file for each server (#110). Previously push state was stored in {data_dir}/pushstate.db, now it is moved to {data_dir}/push/{remote}.db, where remote is section name without client name part from credentials.cfg file. When upgrading, you need to move pushstate.db manually to desired location. If not moved, you will loose you state and all data will be pushed.
- Use different location for keymap SQLite database file (#117). Previously, by default keymaps.db file, was stored in a current working directory, but now file was moved to {data_dir}/keymap.db. Please move keymaps.db file to {data_dir}/keymap.db after upgrade. By default {data_dir} is set to ~/.local/share/spinta.
New features:
- Show server error and first item from data chunk sent to server, this will help to understand what was wrong in case of an error (#111).
- Add --log-file and --log-level arguments to spinta command.
- In HTML format view, show file name and link to a file if _id is included in the query (#114).
- Add support for ASCII manifest files. This makes it easy to test examples from tests or documentation. ASCII manifests files must have .txt file extension to be recognized as ASCII manifest files.
Bug fixes:
- Fix issue with self referenced models, external SQL backend ended up with an infinite recursion on self referenced models (#110).
Bug fixes:
- Because an incorrect template was used, html format was not outputing anything at all. Added a test to actually test what is inside rendered html, this should prevent errors like this in future.
New features:
- Add /robots.txt handler. Currently it allows everything for robots and is mainly added to avoid error messages in logs about missing robots.txt.
Bug fixes:
- Allow private properties to be used ad file() arguments for file types.
- When pushing data to remote server, read data as default client, by default
- Previously data was read with admin rights, which caused issues with non-open properties being sent to remote, which was refused by remote as unknown properties.
- When copying data with spinta copy --no-source, also clean ref, source and prepare values of resource rows.
New features:
- spinta inspect now can read an existing manifest file and update it with new schema changes made in data source, preserving all manual edits made in manifest file. This is not yet fully tested, but does work in simple cases. This feature is not yet ready for use in production, because not all manual edits in manifest file can be preserved. For example composite foreign keys are not yet implemented.
- Add API endpoint /:check for checking if manifest table is correct.
- Add file() function for reading file data from external datasets. Currently this is only implemented for SQL backend.
Bug fixes:
- Now root namespace is always added to manifest event if manifest is empty. This fixes 404 error when accessing root namespace on an empty manifest.
- Create default auth client automatically if it does not exists. Currently this was enabled only for spinta inspect command.
Actually this version was released, but because of human error, it was the same as 0.1.14 version.
Backwards incompatible changes:
- spinta push command is now unified with other commands and works like this spinta push manifest1 manifest2 target. Target configuration is moved to XDG compatible credentials configuration, state is now saved in a XDG directory too, by default. -r was replaced with --credentials, but by default credentials are looked in ~/.config/spinta/credentials so there is no need to specify it. -c flag is also no longer exists, you can add client to target like this client@target, if client is not specified it will be read from credentials file.
- Now configuration and data files are stored in a XDG Base Directory Specification compatible directories, by default, but can be overridden via main config file, environment variables or command line arguments.
Performance improvements:
- Migrated from Earley to LALR(1) parser algorithm and this made formula parser 10 times faster, doing write operations involving _where, things should be about 3-5 times faster. Whole test suite after this because 20% faster.
- Moved select list handling out of rows loops and this made lists of objects about 5 times faster.
- Enabled server-side cursors for getall actions, now memory consumption is always constant even when downloading large amounts of data.
- Fix few bugs in access logging, because of these bugs whole result set was consumed at once and stored in memory. This cause delays, when starting to download data and also used a lot of memory.
New features:
- Add support for XLSX format for manifest tables (#79).
- Add lang support in manifest files, now it is possible to describe data structures in multiple languages (#85).
- Add spinta pii detect --limit which is set to 1000 by default.
- Now it is possible to pass AST query form to _where for upsert, update and patch operations. This improves performance of data sync.
Bug fixes:
- Do a proper content-type header parsing to recognize if request is a streaming request.
- Fix bug with incorrect type conversion before calculating patch, which resulted in incorrect patch, for example with date types (#94).
Bug fixes:
- Fix a bug in spinta push. It failed when resource was defined on a dataset.
New features:
- Add implicit filters for external sql backend. With implicit filters, now you can specify filter on models once and they will be used automatically on related models (#74).
Bug fixes:
- Fix ref data type in HTML export.
Backwards incompatible changes:
- choice type was changed to enum.
New features:
- Add root config option, to set namespaces, which will be shown on /. Also this option restricts access only to specified namespace.
- Change ufunc schema(name) to connect(self, schema: name).
- Possibility to provide title and description metadata for namespaces (#56).
- Fix duplicate items in /:ns/:all query results (#23).
- Add spinta copy --format-name option, to reformat names on copy (#53).
- Add spinta copy --output --columns flags. Now by default spinta copy writes to stdout instead of a file (#76). --columns is only available when writing to stdout.
- Add spinta copy --order-by access flag (#53).
- Add enum type dimension for properties. This allows to list possible values of a property (#72).
- Filter data automatically by enum.access (#73).
- Add spinta --version.
- Add spinta init command, to initialize empty manifest table.
- Add spinta show command, to print manifest table to stdout.
- Backend now became optional and by default manifest is configured without manifest.
- spinta inspect no longer overwrites existing manifest. By default, manifest is printed to stdout. Only if -o flag is given, then manifest is written into a csv file.
- Fix incorrectly built python packages (poetry#3610).
- Fix URL link formatting in HTML output.
- external.prepare for Model and Property became Expr instead of dict.
- Expr now has it's own unparse and preserves exact expression representation.
- Sql backend now supports formulas in select(). This was only added to
- support composition keys, but also all kinds of formulas in select() are supported, but not yet implemented.
- count() now must be inside select(), but only for Sql backend.
- Property.external no longer can be a list, if you need more than one value, use prepare. That means, listing multiple items in source column is no longer supported.
Backwards incompatible features:
spinta migrate command was renamed to spinta bootstrap. spinta migrate command still exists, but now it does real migrations.
All environment variables now must use __ to separate configuration name nested parts. You can list all configuration options using this command:
> spinta config Origin Name Value ----------------- -------------------- ------------- app.config:CONFIG backends.default.dsn postgresql://
By using -f env command line argument you can turn configuration option names into environment variable names:
> spinta config -f env Origin Name Value ----------------- ------------------------------ ---------- app.config:CONFIG SPINTA_BACKENDS__DEFAULT__TYPE postgresql
Previously SPINTA_BACKENDS__DEFAULT__TYPE was SPINTA_BACKENDS_DEFAULT_TYPE, bit this name is no longer recognized.
Configuration option backends.*.backend was replaced by backends.*.type. And backends.*.backend now is moved to components.backends.*. For example previoulsy it looked like this:
backends.default.backend=spinta.backends.postgresql:PostgreSQL
Now must be written like this:
components.backends.postgresql=spinta.backends.postgresql:PostgreSQL backends.default.type=postgresql
Previously Spinta had multiple manifests, now only one default manifest exists and it is specified like this:
manifest = default manifests.default.type = internal manifests.default.sync = yaml manifests.yaml.type = yaml
Here we have two manifess default and yaml, but only one manifest named default is enabled. Default manifest is specified using manifest configuration option.
Only one manifest can be used, the one specified by manifest configuration option.
But multiple manifest can be configured. In the example above, default manifest is synced from yaml manifest. That menas, when spinta sync command is run it synces default manifest from another manifest specified in manifests..sync configuration option.
From code perspective, all code liek store.manifests['default'] is now replaced with store.manifest, because now only one active manifest is available. There can be multiple backends, bet other backends must be synced to the default one.
Previously there was only one manifest type, YAML files based manifest. Now multiple manifest types were introduced and currently implemented two manifest types internal and yaml.
internal manifest is stored in manifests..backend database, in _schema and _schema/Version models.
yaml manifest is same manifest as was used previously.
Yeach manifest type can do multiple manifest specific activities, liek loading manifest into memory, running migrations, synchronizing manifest from specified sources and etc.
Now default manifest usualy should be internal, which is synchronized from a yaml manifest.
Internal transaction model was renamed to _txn.
Configuration interpretation now slighty changes. Previously in order to add new items into configuration, you had to do things like this:
backends=default,mongo backends.mongo.type=mongo backends.mongo.dsn=mongo://...
In order to make new item to be visible, you had to explicitly add it via backends=default,mongo. Now this is not needed. All parent configuration nodes are added automatically, this whould be enough:
backends.mongo.type=mongo backends.mongo.dsn=mongo://...
But possibility to explicitly specify list of keys is still supported.
Configuraiton using Python dicts now suports dotted notation:
CONFIG = { 'backends.mongo': { 'type': 'mongo', 'dsn': 'mongo://...', }, }
This also works with environments:
CONFIG = { 'environments': { 'test': { 'backends.default.dsn': 'postgresql://...', 'backends.mongo.dsn': 'mongo://...', } } }
Configuration value provided as dict is no longer merged. For example:
CONFIG = { 'backends': { 'default': { 'type': 'postgresql', }, 'mongo': { 'type': 'mongo', }, }, 'environments': { 'test': { 'backends': { 'default': { 'type': 'mongo', }, }, }, 'dev': { 'backends.default.type': 'mongo', } } }
Here, test configuration environment fully overrides backends and removes mongo backend defined in default configuration scope.o
But dev environment overrides only backends.default.type and leaves everything else as is, mongo backend stays untouched.
Previously all configuration parameters were always merged.
Context variable config.raw was renamed to rc.
Test fixture config was renamed to rc.
cli test fixture, now overrides CliRunner.invoke and adds RawConfig as first argument. This gives possibility to execute commands under different configuration. Each command invocation creates new context using given configuration object, so now there is no issues related with using same context for multiple commands.
Removed get_referenced_model command. Now Ref objects are linked with referenced model in link command.
Renamed object to model on ref properties.
New features:
New commands:
spinta bootstrap - this command does same thing as previously did spinta migrate it simply creates all missing tables from scratch and upates all migration versions as applied. With internal manifest bootstrap does nothing if it finds that _schema/Version table is created. But with yaml manifest bootstrap always tries to create all missing tables.
spinta sync - this command updates default manifest from list of other manifests specified in manifests.<manifest>.sync. It is also possible to add other kinds of manifests, for example we can add Qvarn YAML files directly.
spinta migrate - this command automatically runs spinta bootstrap, then spinta sync and then executes migration actions for all versions that are not yet migrated.
All these three commands helps to control schema and data migrations.
Introduced access log. Access log can be configured using accesslog configuration option. Corrently two accesslog backends are implemented, file and python. python backend is used only for tests and it logs into memory. file backend can log to stdout, stderr, /dev/null and to a file. When /dev/null is specified as accesslog.file, then nothing is logged, internally logs are not even written to real /dev/null file, log messages are simply ignored.
spinta config command now does not tries to load manifest, it just reads configuration and prints it. Previously spinta config tried to load manifest and if something is misconfigured it failed without showing configuration which could help solve the issue.
spinta config command now accepts queries liek backends..type it prints all backends.*.type backends. I did not use *, because * is reserved symbol in command line.
spinta config now has -f env argument to show config option names as environment variables.
Error response now includes component context var with pyton path of component class.
Added new command spinta decode-token, this command decoded token from stdin and prints its content to stdout in JSON format.
Added support for Json Web Key Sets.
Added new token_validation_key configuration parameter.
Internal changes:
Changed internal file structure, not code is organized into packages and each package has following structure:
backends/ backend/ constants.py components.py helpers.py commands/ load.py link.py check.py wait.py init.py freeze.py bootstrap.py migrate.py encode.py validate.py verify.py write.py read.py query.py changes.py wipe.py types/ array/ init.py write.py wipe.py manifest/ load.py sync.py types/ array/ components.py commands/ load.py link.py check.py backends/ postgresql/ init.py write.py read.py wipe.py manifests/ yaml/ components.py commands/ load.py link.py sync.pyInternal structure now is organized same way as Spinta extensions should be organized. There are two types of structures, one is backend focused and another is type focused. Essentially everything is composed of components and commands, both types and backends are components and there are number of commands responsible for various actions performed on components.
Actions are organized into these categories:
- Loading components from manifest:
- load - do initial component loading.
- link - when everythin is loaded link dependent components.
- check - when all components are loaded and linked, check components.
- wait - wait while backends are up and accepts connections.
- init - initialized backends.
- Schema and data migration commands:
- freeze - save all changes to manifest files as new migration versions.
- bootstrap - bootstrap empty databases, just creates all missing tables.
- sync - synchronizes two manifests.
- migrate - run migrations
- Data convertion between external and internam forms:
- encode - convert values from internal to external form.
- decode - convert values from external to internal form.
- Data validation:
- validate - simple data validation.
- verify - complex data validation involving access to stored data.
- Writing data to dabases (high level):
- insert - insert new data to database.
- upsert - insert or modify existing data in database.
- update - overwrite existing data in database.
- modify - modify or patch existing data in database.
- delete - delete exisint data in database.
- Writing data to database (low level):
- insert - insert new objects into database.
- update - updated existing data.
- delete - delete existin data from database.
- Reading data from database:
- getone - read one object from database.
- getall - read multiple objects from database.
- Query functions:
- Functions used in query.
- Changelog:
- commit - save changes to changelog.
- changes - read changes from changelog.
- Wipe all data in fastest way possible:
- wipe - wipes all data of a given model.
- Loading components from manifest:
RawConfig was moved from spinta.config to spinta.core.config. spinta.config now contains only configration dict CONFIG, nothing else.
RawConfig was fulle refactored. Previously RawConfig supported only hardcoded list with hardcoded ordering of configuration sources. Now that was changed to a list of sources. And each configuration sources was refactored to separate components. So now there is a possibility to add other configuration sources if needed.
Now RawConfig can be initialized like this:
rc = RawConfig() rc.read(sources, after='name')
This gives possibility to provide configuraiton sources in any order and even inject sources at specified position via after argument.
In tests RawConfig fixture is initialized into session scope, but a new modified instance can be crated using RawConfig.fork method.
RawConfig now uses configuration schema defined in spinta/config.yml file. Now, this schema is only used to identify if given environment variable should go to environments and used to recognize if a configuration option is leaft or not.
But in future, configuration schema can be used to fully validate all configuration paramters.
Switched to declarative app init style, that means there is no longer global app instances created, app configuration is fully declarative and app is always initialized dynamicaly insing spinta.api.init.
spinta.api.init accepts Context argument, that means, we can confure app in any way we want, before initializing it.
Same thing is done to comman line commans initialization. Commands can receive context via command scopes, this means, that command can be configured before running it.
All these changes gives more control in tests and now it is possible to do things like these:
from spinta.testing.utils import create_manifest_files, read_manifest_files from spinta.testing.client import create_test_client from spinta.testing.context import create_test_context def test(rc, cli, tmp_path, request): create_manifest_files(tmp_path, { 'country.yml': { 'type': 'model', 'name': 'country', 'properties': { 'name': {'type': 'string'}, }, }, }) rc = rc.fork().add('test', {'manifests.yaml.path': str(tmp_path)}) cli.invoke(rc, freeze) cli.invoke(rc, migrate) context = create_test_context(rc) request.addfinalizer(context.wipe_all) client = create_test_client(context) client.authmodel('_version', ['getall', 'search']) data = client.get('/_schema/Version').json()
There is no longer separate internal manifest. Since now there is only one manifest, internal manifest does not exist as a separate manifest, but it is injected into the default manifest.
When default manifest is loaded, in addition, internal manifest is always loaded from YAML files and injected into default manifest.
Now internal manifest is always exists as part of default manifest.
Manfest loading was abstracted using manifest components and all places reading YAML files directly was replaced with abstract manifest components. this way it does not matter were manifest is defined.
PostgreSQL backends no longer uses tables[manifest][table], this was replaced with tables[table], since now there is only one manifest.
In PostgreSQL backends, references to _txn model is no longer used, in order to remove interdependence between two separate manifests.
Also, _txn might be saved on another backend.
RawConfig now can take default values from spinta/config.yml.
prop.backend was moved to dtype.backend.