Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres 12 support #5796

Open
Zharktas opened this issue Dec 16, 2020 · 9 comments
Open

Postgres 12 support #5796

Zharktas opened this issue Dec 16, 2020 · 9 comments
Assignees

Comments

@Zharktas
Copy link
Member

These are general notes about what needs to be changed to support Postgres 12 which comes with Ubuntu 20.04, these probably come up in questions once users start upgrading their OS's unless they are using docker.

Master branch / 2.9
Upgrading psycopg2 should be enough as 2.8.4 has support for Postgres 12.

2.8
CKAN 2.8 requires upgrading sqlalchemy to at least 1.2.17 as it has a bugfix for Postgres 12. If updated to 1.3, #4853 needs to be backported too. The PR includes upgrading vdm and changes to codebase.

ThrawnCA added a commit to qld-gov-au/ckan that referenced this issue Dec 17, 2020
ThrawnCA added a commit to qld-gov-au/ckan that referenced this issue Dec 17, 2020
[QOL-7455] update SQLAlchemy to support PostgreSQL 12, ckan#5796
@ThrawnCA
Copy link
Contributor

SQLAlchemy 1.2 requires updating to vdm 0.15

@amercader
Copy link
Member

Add a note on the CHANGELOG for the next 2.8 patch release

ThrawnCA added a commit to qld-gov-au/ckan that referenced this issue Dec 17, 2020
@avdata99
Copy link
Member

Old SQLAlchemy has medium severity vulnerability: https://snyk.io/vuln/SNYK-PYTHON-SQLALCHEMY-590109.
Update to 1.3.19 is important.
Is possible to update SQLAlchemy for CKAN 2.8?

@ThrawnCA
Copy link
Contributor

Is possible to update SQLAlchemy for CKAN 2.8?

I encountered errors upon trying it, and reverted. I don't think it's a drop-in replacement. Someone who knows the code in more detail might be able to resolve it.

@avdata99
Copy link
Member

I run some manual test and just making this change, it works.
Not sure this commit is a good idea.
Probably CKAN is not affected by this specific vulnerability.

@Zharktas
Copy link
Member Author

I had errors with active extras mappings in sqlalchemy was upgraded to 1.3, did not have time enough to look into it. In the end I just upgraded to 1.2, vdm to 0.15 and that code change as it is required with 1.2 too.

@managume
Copy link

Please, someone could add a clarification in CKAN 2.8 documentation about this issue? CHANGELOG note at 2.8.7 patch release is not totally correct.

If only update sqlalchemy and vdm packages, the db init bug is fixed but when you try to create and organisation or group, CKAN returns a SERVER ERROR page. So is mandatory patch meta.py with True and False instead of '1'and '0' values.

For me CKAN 2.8.9 works with PostgreSQL 12 with the following steps:

  • Upgrade pip version: pip install --upgrade pip after installs setuptools
  • Install or upgrade CKAN from github source to last patch release
  • Install requirements and update relevant packages: pip install --upgrade sqlalchemy==1.2.17 vdm==0.15
  • Edit meta.py as @avdata99 proposal

@avdata99
Copy link
Member

@managume, CKAN 2.9 and master no longer include this code
I assume you need to apply a manual patch on your CKAN 2.8.x code.
Not sure if this is something to apply (sending a PR) to 2.8 branch and include on the next CKAN 2.8 tag release.

@ThrawnCA
Copy link
Contributor

ThrawnCA commented Nov 10, 2021

In other circumstances we'd have been happy to test this in qld-gov-au/ckan, but we're biting the bullet and upgrading to CKAN 2.9 in the near future, and PostgreSQL 11 still has years of support left, so it's unlikely to be prioritised.

ThrawnCA added a commit to qld-gov-au/ckan that referenced this issue Sep 2, 2022
amercader added a commit that referenced this issue Sep 6, 2022
[2.9] Update psycopg2 to support PostgreSQL 12, #5796
ThrawnCA added a commit to qld-gov-au/ckan that referenced this issue Sep 14, 2022
ThrawnCA added a commit to qld-gov-au/ckan that referenced this issue Sep 14, 2022
[QOL-9344] update psycopg2 to support PostgreSQL 12, ckan#5796
duttonw pushed a commit to qld-gov-au/ckan that referenced this issue Nov 24, 2022
update psycopg2 to support PostgreSQL 12, ckan#5796

[QOL-9275] add config flag to control private dataset behaviour

- If set to True, then return Forbidden or redirect to login page, instead of hiding the dataset's existence.

Bump moment from 2.26.0 to 2.29.4

Bumps [moment](https://github.com/moment/moment) from 2.26.0 to 2.29.4.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md)
- [Commits](moment/moment@2.26.0...2.29.4)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

fix logic import

[DC-32816] update sqlparse to handle CSV contents more robustly, GitHub ckan#5822

- This is already fixed on master but not backported to 2.9

[DC-31965] handle missing resources in activity stream

- Copy cleanup from upstream

Revert "Add test and changelog"

This reverts commit 3745031.

Revert "Check if locale exists on i18n JS API"

This reverts commit c906ee1.

Revert "Fixing tests."

This reverts commit 233be7e.

Add test and changelog

Fixing tests.

Check if locale exists on i18n JS API

fix errno2

check for resource

remove extra comma

(cherry picked from commit fb27f2a)

[QOL-8785] only import Pylons response when relevant

[QOL-8785] fix missing imports and make Flake8 happy

[QOL-8785] reuse uploader mimetype if present

[QOL-8785] store resource URL in uploader so we can retrieve it for Pylons downloads

[QOL-8785] parse config flag as a boolean

[QOL-8785] skip test for now

[QOL-8785] drop test environment for post

[QOL-8785] make mailer test syntax match other tests

[QOL-8785] add blank password fields to pass validation

[QOL-8785] restore 'save' flag to test form

[QOL-8785] add debugging statement for test failure

[QOL-8785] add username to test environment

[QOL-8785] adjust test assertion for better failure message

[QOL-8785] fix test assertion syntax

[QOL-8785] adjust test username change parameters

[QOL-8785] fix test assertion syntax

[QOL-8785] fix test assertion syntax

[QOL-8785] oops only add version to emails if flag is not set

[QOL-8785] use string not bool for test config value

[QOL-8785] add mail server parameter to test

[QOL-8785] make post-reset landing page configurable

[QOL-8785] use 'download' function to simplify controller

[QOL-8785] drop 'REMOTE_USER' value in test as it's confusing the dashboard

[QOL-8785] update assertion syntax for pytest

[QOL-8515] get test app from fixtures instead of directly retrieving it

[QOL-8515] make pep8 happy

[QOL-8515] skip updating package modified timestamp if only updating resources

- copied from 2.8.8 QGOV branch

[QOL-8515] allow sysadmins to update usernames

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally redirect to a different page after password resets

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally hide CKAN version in site status and emails

- copied from 2.8.8 QGOV branch

[QOL-8515] make pep8 happy

[QOL-8515] make pep8 happy

[QOL-8515] fix import spelling

[QOL-8515] make pep8 happy

[QOL-8515] add functionality to delete uploaded files from storage on resource deletion

[QOL-8515] adjust mocking of 'open' function to handle PY2 and PY3

[QOL-8515] oops provide necessary test import

[QOL-8515] add testing of extra IUploader functions

add metadata to uploader interface, next step, surface it via package

* add metadata to IUploader interface's
* Add new resource_file_metadata_show get api to surface metadata checks, usage case is archiver or similar verifying files have not change or disappeared
* missed storage_path from file delete/download

[QOL-8576] simplify imports

[QOL-8576] fix isoformat syntax so lint is happy

- raw string literals aren't allowed, but Python 2 'isoformat' can't handle unicode,
so use the 'str' function, which is documented to have the same effect

Fix isoformat argument for Python 2

[QOL-7544] handle storage directory already existing, ckan#6521

remove white space

lint

check if dir already exists

[QOL-7544] apply QGOV patches to 2.9.5 base

- render full name as 'Displayed Name' since that better describes its purpose
- handle race condition in creation of site user
- ignore authentication when loading user profile during password reset
duttonw pushed a commit to qld-gov-au/ckan that referenced this issue Nov 24, 2022
duttonw pushed a commit to qld-gov-au/ckan that referenced this issue Nov 24, 2022
update psycopg2 to support PostgreSQL 12, ckan#5796

[QOL-9275] add config flag to control private dataset behaviour

- If set to True, then return Forbidden or redirect to login page, instead of hiding the dataset's existence.

Bump moment from 2.26.0 to 2.29.4

Bumps [moment](https://github.com/moment/moment) from 2.26.0 to 2.29.4.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md)
- [Commits](moment/moment@2.26.0...2.29.4)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

fix logic import

[DC-32816] update sqlparse to handle CSV contents more robustly, GitHub ckan#5822

- This is already fixed on master but not backported to 2.9

[DC-31965] handle missing resources in activity stream

- Copy cleanup from upstream

Revert "Add test and changelog"

This reverts commit 3745031.

Revert "Check if locale exists on i18n JS API"

This reverts commit c906ee1.

Revert "Fixing tests."

This reverts commit 233be7e.

Add test and changelog

Fixing tests.

Check if locale exists on i18n JS API

fix errno2

check for resource

remove extra comma

(cherry picked from commit fb27f2a)

[QOL-8785] only import Pylons response when relevant

[QOL-8785] fix missing imports and make Flake8 happy

[QOL-8785] reuse uploader mimetype if present

[QOL-8785] store resource URL in uploader so we can retrieve it for Pylons downloads

[QOL-8785] parse config flag as a boolean

[QOL-8785] skip test for now

[QOL-8785] drop test environment for post

[QOL-8785] make mailer test syntax match other tests

[QOL-8785] add blank password fields to pass validation

[QOL-8785] restore 'save' flag to test form

[QOL-8785] add debugging statement for test failure

[QOL-8785] add username to test environment

[QOL-8785] adjust test assertion for better failure message

[QOL-8785] fix test assertion syntax

[QOL-8785] adjust test username change parameters

[QOL-8785] fix test assertion syntax

[QOL-8785] fix test assertion syntax

[QOL-8785] oops only add version to emails if flag is not set

[QOL-8785] use string not bool for test config value

[QOL-8785] add mail server parameter to test

[QOL-8785] make post-reset landing page configurable

[QOL-8785] use 'download' function to simplify controller

[QOL-8785] drop 'REMOTE_USER' value in test as it's confusing the dashboard

[QOL-8785] update assertion syntax for pytest

[QOL-8515] get test app from fixtures instead of directly retrieving it

[QOL-8515] make pep8 happy

[QOL-8515] skip updating package modified timestamp if only updating resources

- copied from 2.8.8 QGOV branch

[QOL-8515] allow sysadmins to update usernames

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally redirect to a different page after password resets

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally hide CKAN version in site status and emails

- copied from 2.8.8 QGOV branch

[QOL-8515] make pep8 happy

[QOL-8515] make pep8 happy

[QOL-8515] fix import spelling

[QOL-8515] make pep8 happy

[QOL-8515] add functionality to delete uploaded files from storage on resource deletion

[QOL-8515] adjust mocking of 'open' function to handle PY2 and PY3

[QOL-8515] oops provide necessary test import

[QOL-8515] add testing of extra IUploader functions

add metadata to uploader interface, next step, surface it via package

* add metadata to IUploader interface's
* Add new resource_file_metadata_show get api to surface metadata checks, usage case is archiver or similar verifying files have not change or disappeared
* missed storage_path from file delete/download

[QOL-8576] simplify imports

[QOL-8576] fix isoformat syntax so lint is happy

- raw string literals aren't allowed, but Python 2 'isoformat' can't handle unicode,
so use the 'str' function, which is documented to have the same effect

Fix isoformat argument for Python 2

[QOL-7544] handle storage directory already existing, ckan#6521

remove white space

lint

check if dir already exists

[QOL-7544] apply QGOV patches to 2.9.5 base

- render full name as 'Displayed Name' since that better describes its purpose
- handle race condition in creation of site user
- ignore authentication when loading user profile during password reset
duttonw pushed a commit to qld-gov-au/ckan that referenced this issue Nov 24, 2022
[QOL-7544] apply QGOV patches to 2.9.5 base

update psycopg2 to support PostgreSQL 12, ckan#5796

[QOL-9275] add config flag to control private dataset behaviour

- If set to True, then return Forbidden or redirect to login page, instead of hiding the dataset's existence.

Bump moment from 2.26.0 to 2.29.4

Bumps [moment](https://github.com/moment/moment) from 2.26.0 to 2.29.4.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md)
- [Commits](moment/moment@2.26.0...2.29.4)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

fix logic import

[DC-32816] update sqlparse to handle CSV contents more robustly, GitHub ckan#5822

- This is already fixed on master but not backported to 2.9

[DC-31965] handle missing resources in activity stream

- Copy cleanup from upstream

Revert "Add test and changelog"

This reverts commit 3745031.

Revert "Check if locale exists on i18n JS API"

This reverts commit c906ee1.

Revert "Fixing tests."

This reverts commit 233be7e.

Add test and changelog

Fixing tests.

Check if locale exists on i18n JS API

fix errno2

check for resource

remove extra comma

(cherry picked from commit fb27f2a)

[QOL-8785] only import Pylons response when relevant

[QOL-8785] fix missing imports and make Flake8 happy

[QOL-8785] reuse uploader mimetype if present

[QOL-8785] store resource URL in uploader so we can retrieve it for Pylons downloads

[QOL-8785] parse config flag as a boolean

[QOL-8785] skip test for now

[QOL-8785] drop test environment for post

[QOL-8785] make mailer test syntax match other tests

[QOL-8785] add blank password fields to pass validation

[QOL-8785] restore 'save' flag to test form

[QOL-8785] add debugging statement for test failure

[QOL-8785] add username to test environment

[QOL-8785] adjust test assertion for better failure message

[QOL-8785] fix test assertion syntax

[QOL-8785] adjust test username change parameters

[QOL-8785] fix test assertion syntax

[QOL-8785] fix test assertion syntax

[QOL-8785] oops only add version to emails if flag is not set

[QOL-8785] use string not bool for test config value

[QOL-8785] add mail server parameter to test

[QOL-8785] make post-reset landing page configurable

[QOL-8785] use 'download' function to simplify controller

[QOL-8785] drop 'REMOTE_USER' value in test as it's confusing the dashboard

[QOL-8785] update assertion syntax for pytest

[QOL-8515] get test app from fixtures instead of directly retrieving it

[QOL-8515] make pep8 happy

[QOL-8515] skip updating package modified timestamp if only updating resources

- copied from 2.8.8 QGOV branch

[QOL-8515] allow sysadmins to update usernames

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally redirect to a different page after password resets

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally hide CKAN version in site status and emails

- copied from 2.8.8 QGOV branch

[QOL-8515] make pep8 happy

[QOL-8515] make pep8 happy

[QOL-8515] fix import spelling

[QOL-8515] make pep8 happy

[QOL-8515] add functionality to delete uploaded files from storage on resource deletion

[QOL-8515] adjust mocking of 'open' function to handle PY2 and PY3

[QOL-8515] oops provide necessary test import

[QOL-8515] add testing of extra IUploader functions

add metadata to uploader interface, next step, surface it via package

* add metadata to IUploader interface's
* Add new resource_file_metadata_show get api to surface metadata checks, usage case is archiver or similar verifying files have not change or disappeared
* missed storage_path from file delete/download

[QOL-8576] simplify imports

[QOL-8576] fix isoformat syntax so lint is happy

- raw string literals aren't allowed, but Python 2 'isoformat' can't handle unicode,
so use the 'str' function, which is documented to have the same effect

Fix isoformat argument for Python 2

[QOL-7544] handle storage directory already existing, ckan#6521

remove white space

lint

check if dir already exists

[QOL-7544] apply QGOV patches to 2.9.5 base

- render full name as 'Displayed Name' since that better describes its purpose
- handle race condition in creation of site user
- ignore authentication when loading user profile during password reset
duttonw pushed a commit to qld-gov-au/ckan that referenced this issue Nov 24, 2022
[QOL-7544] apply QGOV patches to 2.9.5 base

update psycopg2 to support PostgreSQL 12, ckan#5796

[QOL-9275] add config flag to control private dataset behaviour

- If set to True, then return Forbidden or redirect to login page, instead of hiding the dataset's existence.

Bump moment from 2.26.0 to 2.29.4

Bumps [moment](https://github.com/moment/moment) from 2.26.0 to 2.29.4.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md)
- [Commits](moment/moment@2.26.0...2.29.4)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

fix logic import

[DC-32816] update sqlparse to handle CSV contents more robustly, GitHub ckan#5822

- This is already fixed on master but not backported to 2.9

[DC-31965] handle missing resources in activity stream

- Copy cleanup from upstream

Revert "Add test and changelog"

This reverts commit 3745031.

Revert "Check if locale exists on i18n JS API"

This reverts commit c906ee1.

Revert "Fixing tests."

This reverts commit 233be7e.

Add test and changelog

Fixing tests.

Check if locale exists on i18n JS API

fix errno2

check for resource

remove extra comma

(cherry picked from commit fb27f2a)

[QOL-8785] only import Pylons response when relevant

[QOL-8785] fix missing imports and make Flake8 happy

[QOL-8785] reuse uploader mimetype if present

[QOL-8785] store resource URL in uploader so we can retrieve it for Pylons downloads

[QOL-8785] parse config flag as a boolean

[QOL-8785] skip test for now

[QOL-8785] drop test environment for post

[QOL-8785] make mailer test syntax match other tests

[QOL-8785] add blank password fields to pass validation

[QOL-8785] restore 'save' flag to test form

[QOL-8785] add debugging statement for test failure

[QOL-8785] add username to test environment

[QOL-8785] adjust test assertion for better failure message

[QOL-8785] fix test assertion syntax

[QOL-8785] adjust test username change parameters

[QOL-8785] fix test assertion syntax

[QOL-8785] fix test assertion syntax

[QOL-8785] oops only add version to emails if flag is not set

[QOL-8785] use string not bool for test config value

[QOL-8785] add mail server parameter to test

[QOL-8785] make post-reset landing page configurable

[QOL-8785] use 'download' function to simplify controller

[QOL-8785] drop 'REMOTE_USER' value in test as it's confusing the dashboard

[QOL-8785] update assertion syntax for pytest

[QOL-8515] get test app from fixtures instead of directly retrieving it

[QOL-8515] make pep8 happy

[QOL-8515] skip updating package modified timestamp if only updating resources

- copied from 2.8.8 QGOV branch

[QOL-8515] allow sysadmins to update usernames

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally redirect to a different page after password resets

- copied from 2.8.8 QGOV branch

[QOL-8515] optionally hide CKAN version in site status and emails

- copied from 2.8.8 QGOV branch

[QOL-8515] make pep8 happy

[QOL-8515] make pep8 happy

[QOL-8515] fix import spelling

[QOL-8515] make pep8 happy

[QOL-8515] add functionality to delete uploaded files from storage on resource deletion

[QOL-8515] adjust mocking of 'open' function to handle PY2 and PY3

[QOL-8515] oops provide necessary test import

[QOL-8515] add testing of extra IUploader functions

add metadata to uploader interface, next step, surface it via package

* add metadata to IUploader interface's
* Add new resource_file_metadata_show get api to surface metadata checks, usage case is archiver or similar verifying files have not change or disappeared
* missed storage_path from file delete/download

[QOL-8576] simplify imports

[QOL-8576] fix isoformat syntax so lint is happy

- raw string literals aren't allowed, but Python 2 'isoformat' can't handle unicode,
so use the 'str' function, which is documented to have the same effect

Fix isoformat argument for Python 2

[QOL-7544] handle storage directory already existing, ckan#6521

remove white space

lint

check if dir already exists

[QOL-7544] apply QGOV patches to 2.9.5 base

- render full name as 'Displayed Name' since that better describes its purpose
- handle race condition in creation of site user
- ignore authentication when loading user profile during password reset
fostermh added a commit to cioos-siooc/ckan that referenced this issue Dec 8, 2022
* Rename lib/io.py module which was giving problems

* Rename __init__.py file in extension

* Show job title on job start/finish log messages

To make it easier to debug background job calls.

Before:

```
INFO  [ckan.lib.jobs] Worker rq:worker:f0792c8bd67344f288b5704d39c43124 starts job 2baa42e5-4582-4103-92e5-b4a384d0b1da from queue "default"
```

After:

```
INFO  [ckan.lib.jobs] Worker rq:worker:f0792c8bd67344f288b5704d39c43124 starts job 2baa42e5-4582-4103-92e5-b4a384d0b1da (Process data fields) from queue "default"
```

* Add missing __init__.py file

* String literals

* snippet names rendered in non-debug mode

* Update changelog for 2.9.4

* Build frontend

* [i18n] Pull po files from Transifex

* [i18n] Compile mo files

* Upgrade version for 2.9.4

* Update version for 2.9.5b

* Consistent cli behavior

* pep8

* Py2 compatible fix for ckan#6135

* [ckan#6390] fix user create/edit email validators

* Allow strict types for user/group uploads

CKAN 2.9 specific changes when cherry-picking:

* Replace f-strings with .format()
* Don't use faker / Pillow for tests, as there is no faker fixture in
the Python 2 version

* Add changelog entry for group image types

* Move type verification into upload method

* Fix APIToken CLI test

* Update docs

* Link to config options from changelog

* Allow children for select2

* Fix children type

* [ckan#6531] Py2/py3 compatible version of open

* Add select2 features

* Undo change

* Replace f-string

* Fix standards

* [ckan#6530] Add Solr 8 support

* Set logging level to error in error mail handler

* Add RootPathMiddleware to flask stack to support non-root installs running on python 3

* Add previously removed RootPathMiddleware back to common middleware as it is still needed

* Added utility functions for common CKAN admin commands.

* Use correct auth function when editing organizations

* [ckan#5820] fix invite user with existing email error

* Fix regression when validating resource subfields (by @TomeCirun)

* [ckan#6408] Add timeout param to request get calls (by @EricSoroos)

* [ckan#6408] Document new options

* Accept empty string in one of validator

* Negate empty string check

* Fix pep8

* [i18n] Pull translation from Transifex

* [i18n] Compile mo files

* Compile frontend

* Small fix adding virtual env path to ckan command.

* Update changelog before 2.9.5

* Include the Solr 8 schema file in the 2.9 branch

* Update version for 2.9.5

* Update version for 2.9.6b

* Unpin pytz (ckan#6665)

* Pytz is a stable package, and should always be at the most recent version

* Pin zope.interface to a more recent version (ckan#6665)

* Supports py3 > 3.5
* Allows for modern setuptools > 44.1

* fix errno2

* Add Dockerfile.py3 based on d9a49a8

* Check if locale exists on i18n JS API

* Add test and changelog

* Updates to ckan_utils.sh.

* move spatial harvester into ckanext-cioos_harvest extension and allow POST requests to the spatial search api endpoint in the spatial api

* document spatil harvester config

* update submodules

* create dev branch in submodules and update

* remove extra comma

(cherry picked from commit fb27f2a)

* Fixing tests.

* --passthrough-errors overrides conflicting options

* Describe --passthrough-errors

* Add --passthrough-errors example inside docker-compose.yml

* Disable reloader when passthrough_errors is set

* Add --host 0.0.0.0 to pdb example

* add try/except block when creating test data

* add compile css command

* check for data before attempting to create it again

* upgrade solr to 8.11.1

* update

* update all submodules to latest dev version

* update submodules again

* update schema

* Updated submodule contrib/docker/src/ckanext-cioos_theme

* Updated submodule contrib/docker/src/ckanext-spatial

* fix a few integration bugs

* update

* Updated submodule contrib/docker/src/ckanext-cioos_theme

* update translations

* fix bugs, update logos

* Updated submodule contrib/docker/src/ckanext-scheming

* add atlantic eov icons

* Updated submodule contrib/docker/src/ckanext-cioos_theme

* add collapsed option to indicate how truncated fields initially load

* add wasRevisionOf to schema.org profile output

* release all submodules -  merge dev into main

* remove geoview pip file from dockerfile

* Merge branch 'cioos' into cioos_dev
Removed submodule contrib/docker/src/ckanext-cioos_theme

* Fixes a two errors when dealing with a encoded url.

* url in question /%EF%AC%81?foo=bar&bz=%AC%81
* This is a unicode character, which can't be decoded from
ascii. Jinja templates will handle this if it's unicode, or if it's
hex encoded ascii, but can't take a non-unicode string in python 2 and
put this in a template.
* The querystring was being quoted, which is incorrect, as:
  1) the special characters in the query string mean something
  2) The rest of the querystring is already quoted. This makes it
  double quoted, as seen in the datastore file
* We don't want to unquote urls before putting them in the template
anyway.
* There was s further error passing this unicode path to the template
resolution, where in posix path, it fails:
```
File '/usr/lib/ckan/default/lib/python2.7/posixpath.py', line 73 in join
  path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 1: ordinal not in range(128)
```
The solution here is to make sure it's unicode passed into the
function.

* check for resource

* Fix urlparse function call

* [ckan#6948] Avoid storing the session on each request

Override the `is_null_session()` method on our own custom
`BeakerSessionInterface` class to take into account that Beaker always
adds two keys to the session.

* [ckan#6948] Add tests to ensure we don't create unneeed sessions

Move the BeakerSessionInterface class to make testing easier

* [ckan#6948] pep8

* [ckan#6948] prefix literal

* [ckan#6948] Add more keys for Beaker==1.11.0 (py3)

* [ckan#6948] Prefixes

* [ckan#6948] Cross-py version compatible fix

* [ckan#6948] line too long

* [ckan#6948] [ckan#6984] More elegant check, thanks @ThrawnCA

* [ckan#6948] pep8

* update

* update

* Fix additional casting error (str(unicode)->ascii decode error)

* add helper csrf_input

* replaced package_read with package_show

* [ckan#5727] Fix datapusher trigger in case of resource_update without changing URL but using new file via API

The notify method in the IResourceUrlChange will be triggered only
when the URL is changed or if we do the resource update via the API
_submit_to_datapusher will not be triggered and cause the old preview to be displayed.

* [ckan#5727] Add the changelog

* add organization facets

* Expose check_ckan_version to templates

* add try/except block when creating test data

* check for data before attempting to create it again

* Return zero results instead of raising NotFound when vocabulary does not exist

* [ckan#5822] Update sqlparse version

* reorder resource view button: allow translation

* check if dir already exists

* lint

* remove white space

* Exclude site_user from user_list

* Remove typing from cherry-pick

* [QOL-8368] fix race condition in creating the default site user

- creating the user is idempotent so just ignore the error

* [ckan#6649] gettext not for metadata fields

* [ckan#6743] Include root_path in activity email notifications

* [ckan#5857] Extract translations from emails

* Improve error when downloading resource

* Views return 403 for NotAuthorized

* [ckan#6838] Use the headers Reply-to value if its set in the extensions

* Fix broken URL in migration docs

* ckan_config test mark works with request context

* Fix caching logic on logged in users

* [ckan#6892] Fix member delete

* Fix relative import

* Handle missing resources in activity stream

* [ckan#6439] Concurrent-safe resource updates

* Fix tests after ckan#6820

* lint

* Fix tests after ckan#6618

* prefixes

* Remove duplicated class

* add auth functions for 17 actions that didn't have them before

* add bilingual support to resource names

* fix formatting to satisfy linter

* remove new auth functions from blacklist

* add auth function for recently_changed_packages_activity_list

* remove recently_changed_packages_activity_list from blacklist

* document sitemap generation

* add sitemap url to docs

* [2.9] Bump markdown requirement to support Python 3.9

* update psycopg2 to support PostgreSQL 12, ckan#5796

* [ckan#6789] Fix error when listing tokens in the CLI in py2

* [ckan#6519] Use get_action in patch actions to allow custom logic

* [ckan#6658] Fix not_empty validator to allow falsy values

* [ckan#6956] Prevent non-sysadmin users to change their own state

* [i18n] Pull translations from Transifex for 2.9.6

* [i18n] Compile mo files

* Add user_patch action

Needed for the ckan#6956 fix

* Fix patch for ckan#6956

* Frontend build

* lint

* Fix resource file size not updating with resource_patch

* Added changelog fragment

* [ckan#6817] Fix theme settings

* Replace characters in url

* Fix url check location

* Use user id in auth cookie rather than name

* [ckan#6815] Allow get_translated helper to fall back to base version of a language

* lint

* lint2

* Updated submodule contrib/docker/src/ckanext-cioos_theme

* Update CHANGELOG before 2.9.6

* Update version for 2.9.6

* Update version for 2.9.7b

* Updated submodule contrib/docker/src/ckanext-spatial

* Revert deletion portions of f9084f9

* Restores main_css as an app global
* Restores helpsrs.get_rtl_theme

* Reset the form after downloading

* Perform checks on provided id when creating user

* [ckan#7149] Fix organization delete form (via @Zharktas)

* Update changelog

* Update version for 2.9.7

* fix install docs

* add support for multilingual resource description and name

* make uri's more visible in the interface

* make uri handling more robust and clean up org about page

* fix translations

* merge in subrepo changes

* update sub repos

* match eov labels to munged  keywords

* add missing organization uri display in media grid view

* add fq to organization_list api endpoint

example query ```/api/3/action/organization_list?q=hakai&all_fields=true&include_extras=true&fq=-organization-uri:code"_ "",&fq=organization-uri:__```

* upgrade postgis to 3.3

* document organization_list fq addition

* add matching on org uri during harvest.
consolidate uri fields into code field when possible.

* update submodules

* Updates to functions in ckan_utils.sh.

* add harvest object delete chunk instructions

* Remove duplicate function, minor edits on ckan_utils.sh.

* fix organization matching on UID during ckan harvest

* Update the production.ini as variable, and change ec dump/load to generic functions.

* Minor fix to ckan_utils.sh.

* Updated submodule contrib/docker/src/ckanext-cioos_harvest

* update delete harvest objects by chunks code

* upgrade ckanext-harvest tp 1.4.1

* adjust get_fully_qualified_package_uri

* update submodules

* better populate resources during a cioos ckan harvest

* allow round brackets in keywords

* Updated submodule contrib/docker/src/ckanext-cioos_theme

* add tips to instructions doc

* [#175] Add or relocate volumes for solr and redis data in docker-compose files (#176)

* update pacific css

* add WSP logo

Co-authored-by: amercader <amercadero@gmail.com>
Co-authored-by: Sergey Motornyuk <sergey.motornyuk@linkdigital.com.au>
Co-authored-by: calexandr <cherniavskyi.alexandr@linkdigital.com.au>
Co-authored-by: Andres Vazquez <andres@data99.com.ar>
Co-authored-by: Francesco Frassinelli <francesco.frassinelli@nina.no>
Co-authored-by: Jari Voutilainen <jari.voutilainen@iki.fi>
Co-authored-by: Teemu Erkkola <teemu.erkkola@gofore.com>
Co-authored-by: Jeff Cullis <jcullis@dal.ca>
Co-authored-by: Eric Soroos <eric@derilinx.com>
Co-authored-by: cirun <bitola@Bitolas-MacBook-Pro.local>
Co-authored-by: Tome Cirun <cirun@live.com>
Co-authored-by: Tomasz Sabała <tomek@fjelltopp.org>
Co-authored-by: Sergey <tonnydexter@gmail.com>
Co-authored-by: hq-ods <hao.qin@ontario.ca>
Co-authored-by: Shubham Mahajan <mr.shubhammahajan@gmail.com>
Co-authored-by: Sunny-NEC <t_sunny.malik@india.nec.com>
Co-authored-by: Ian Ward <ian@excess.org>
Co-authored-by: Tome Cirun <cirun@pop-os.localdomain>
Co-authored-by: ThrawnCA <shell_layer-github@yahoo.com.au>
Co-authored-by: pdelboca <patriciodelboca@gmail.com>
Co-authored-by: Knud Möller <dunken69@gmail.com>
Co-authored-by: antuarc <carl.antuar@smartservice.qld.gov.au>
Co-authored-by: Konstantin Sivakov <konstantin.sivakov@gmail.com>
Co-authored-by: Jari Voutilainen <jari-pekka.voutilainen@gofore.com>
Co-authored-by: I G Borrelli <44072946+ItaloBorrelli@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants