GNIP-79: GeoNode REST APIs (v2) #6685

afabiani · 2020-12-01T14:23:37Z

GNIP-79: GeoNode REST APIs (v2)

Overview

Create a real REST API for GeoNode resources, which is:

Pluggable: new external dependencies and modules should be easily allowed to add their own API endpoints without modifying the GeoNode code base.
Modular: each app has their own API module if needed. No more monolithic modules with a ton of classes
REST compliant: the new API must adhere to the REST paradigm and protocol
Live together with the standard Tastypie based GeoNode APIs

Proposed By

@afabiani
@giohappy

Assigned to Release

This proposal is for GeoNode 3.2.

State

Motivation

We would like to transition out from old Tastypie based GeoNode APIs, or, at least, provide a more lightweight, REST compliant and secure set of API endpoints.

The proposal is to collect all those endpoints under the /api/v2 URI prefix.

Proposal

Define a common REST router under `geonode.api`

router = routers.DynamicRouter()

schema_view = get_schema_view(
    openapi.Info(
        title="GeoNode REST API",
        default_version='v2',
        description="Application for serving and sharing geospatial data",
        terms_of_service="https://github.com/GeoNode/geonode/wiki/Community-Bylaws",
        contact=openapi.Contact(email="dev@geonode.org"),
        license=openapi.License(name="GPL License"),
    ),
    public=True,
    permission_classes=(permissions.AllowAny,),
)

Define APIs for each GeoNode module

As an instance for geonode.base.api we can easily plug ResourceBase and Users routes as follows:

from geonode.api.urls import router

from . import views

router.register(r'users', views.UserViewSet)
router.register(r'groups', views.GroupViewSet)
router.register(r'base_resources', views.ResourceBaseViewSet)

Backwards Compatibility

NO Backwards Compatible.

Future evolution

Get rid of Tastypie
Allow GeoNode to be shipped in two modules, pure models/APIs and frontend

Feedback

Update this section with relevant feedbacks, if any.

Voting

Project Steering Committee:

Alessio Fabiani: 👍
Francesco Bartoli:
Giovanni Allegri:👍
Simone Dalmasso:
Toni Schoenbuchner: 👍
Florian Hoedt: 👍

Links

The text was updated successfully, but these errors were encountered:

t-book · 2020-12-01T14:39:21Z

absolutely +1!

gannebamm · 2020-12-01T20:11:27Z

I like the idea and current dev implementation. Nonetheless we should discuss how and if this will connect and/or interfere with pygeoapi.

francbartoli · 2020-12-01T22:05:32Z

I like the general idea to get rid of the old API system. However, I have some concern about how this has been tackled. In particular as far as I can see:

The data model is driven by the Django models rather than being OpenAPI centric with a document defining, at the least, the minimum viable product for a new GeoNode Core API system in terms of endpoints, resources, parameters, responses, each of them with their own schema to enforce the validation eventually. I understood this has been decided to preserve the retro-compatibility. In other words, the OpenAPI document (formerly swagger) is just the output of the ORM at the moment. It would be great to have (if possible) a baseline for a GeoNode openapi.json definition in the codebase.
The specification used for OpenAPI is based on OAS2 that is not maintained anymore and doesn't have some nice features available in OAS3. Also, the new generation of OGC APIs doesn't support OAS2. Is it possible to switch from OAS2 to OAS3? This would allow integrating pygeoapi eventually that supports only OAS3

I'd like to discuss the topic during the summit because I think it is strategic for the evolution of GeoNode in the long run.

tomkralidis · 2020-12-02T02:03:40Z

Valuable and timely discussion here for the Summit next week. Comments based on pygeoapi viewpoint.

definitely agree with OAS3, which is what we are implementing in support of the OGC API standards
pygeoapi is built as a Python API which is capable of handling 1..n web frameworks (we support Flask and Django out of the box). For example, see our Flask support which is a very simple web routing layer atop our pygeoapi.api machinery. In similar "multiple API" projects that I am working on, we simply blueprint pygeoapi to, say /oapi and delegate accordingly. Here in GeoNode, perhaps /api/ogc could be a suitable option
pygeoapi builds the OAS3 document "by hand", i.e. (Python REST framework agnostic), as a build/deployment step. If we want both GeoNode and pygeoapi routes in the same OAS3 document, then we'll need a merge of the two of sorts when building/deploying
what are the auth considerations (pygeoapi does not do auth yet and would likely delegate to the downstream application)?

Having said this, whichever way the evolving GeoNode API is developed, it should be easy enough to build routes on top of pygeoapi for anything OGC API.

t-book · 2020-12-02T07:43:03Z

Thanks @francbartoli @tomkralidis

I like the general idea to get rid of the old API system.

From my point of view we shouldn't just disable old api versions for the sake of backward compatibility (but to offer /v1 /v2 /v3 depending on how the api develops.)
Furthermore I do not see the present proposal as a complete redesign but as a technical improvement of version 1. But of course this is open to discussion. I welcome the replacement of Tastypie with Django Rest Framework as it is lightweight, has a great documentation and is productively proven but I understand the question if pygeoapi can be an alternative.

Valuable and timely discussion here for the Summit next week. Comments based on pygeoapi viewpoint.

👍

I would like to know more about how far the development has progressed and what are the possibility with the auth layer.

Nobody would prevent us from considering v2 as a technically improved version of v1 (tastypie -> DRF) and offering a later v3 which is more OpenAPI centric and based on geopyapi.

kalxas · 2020-12-02T07:48:04Z

First of all, thanks @afabiani for the proof of concept branch! It looks really good.
I also agree with @francbartoli and @tomkralidis that the new API should be based on OAS3 if possible.
@gannebamm yes, we should discuss how to interact with pygeoapi during the summit next week.

On the catalogue perspective I see some possible solution keeping the data model while working with the new OGC APIs at the same time: For resource base, we can keep/extend the model to keep compatibility with the CSW endpoint (which should not go away in my opinion). We can then re-use the resource-base model as a pygeoapi backend to support OGC-API Records, in a similar way we do with pycsw. For the rest of OGC APIs I echo @tomkralidis opinion.

afabiani · 2020-12-02T08:34:55Z

Dear all, thanks for the feedbacks. For the time being the REST APIs are just an improvement which does not impact the model and GeoNode core structure at all, neither touches the old apis. The idea is to transition from an api to another ones, but nothing prevents the two apis to live together. About the pygeoapi, as I already said before, my main concern is about the GeoNode model structure. IMHO the backward compatibility with older versions of GeoNode should (more or less) always taken into account. At least there should be a way to move, without struggling too much, from a model to another one. The second concern is the amount of work to be done if we decide somehow to try to get rid of the current GeoNode structure... I know we now have much more experience, but rewriting GeoNode from scratch is not an option, at least for us. That would mean that we won't have commercial support anymore from entities like GeoSolutions. Il giorno mer 2 dic 2020 alle ore 08:48 Angelos Tzotsos < notifications@github.com> ha scritto:

…

First of all, thanks @afabiani <https://github.com/afabiani> for the proof of concept branch! It looks really good. I also agree with @francbartoli <https://github.com/francbartoli> and @tomkralidis <https://github.com/tomkralidis> that the new API should be based on OAS3 if possible. @gannebamm <https://github.com/gannebamm> yes, we should discuss how to interact with pygeoapi during the summit next week. On the catalogue perspective I see some possible solution keeping the data model while working with the new OGC APIs at the same time: For resource base, we can keep/extend the model to keep compatibility with the CSW endpoint (which should not go away in my opinion). We can then re-use the resource-base model as a pygeoapi backend to support OGC-API Records, in a similar way we do with pycsw. For the rest of OGC APIs I echo @tomkralidis <https://github.com/tomkralidis> opinion. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6685 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYARJNYTBLVLHOWGBOGALSSXWMFANCNFSM4UJDCTKQ> .

-- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Alessio Fabiani @alfa7691 Founder/Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A - 55054 Massarosa (LU) - Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 331 6233686 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

afabiani · 2020-12-02T10:28:52Z

@francesco Bartoli <francesco.bartoli@geobeyond.it> @angelos Tzotsos <gcpp.kalxas@gmail.com> I committed here <399b420> an update to generate the Swagger OAS3 schema Il giorno mer 2 dic 2020 alle ore 09:35 Alessio Fabiani < notifications@github.com> ha scritto:

…

Dear all, thanks for the feedbacks. For the time being the REST APIs are just an improvement which does not impact the model and GeoNode core structure at all, neither touches the old apis. The idea is to transition from an api to another ones, but nothing prevents the two apis to live together. About the pygeoapi, as I already said before, my main concern is about the GeoNode model structure. IMHO the backward compatibility with older versions of GeoNode should (more or less) always taken into account. At least there should be a way to move, without struggling too much, from a model to another one. The second concern is the amount of work to be done if we decide somehow to try to get rid of the current GeoNode structure... I know we now have much more experience, but rewriting GeoNode from scratch is not an option, at least for us. That would mean that we won't have commercial support anymore from entities like GeoSolutions. Il giorno mer 2 dic 2020 alle ore 08:48 Angelos Tzotsos < ***@***.***> ha scritto: > First of all, thanks @afabiani <https://github.com/afabiani> for the > proof of concept branch! It looks really good. > I also agree with @francbartoli <https://github.com/francbartoli> and > @tomkralidis <https://github.com/tomkralidis> that the new API should be > based on OAS3 if possible. > @gannebamm <https://github.com/gannebamm> yes, we should discuss how to > interact with pygeoapi during the summit next week. > > On the catalogue perspective I see some possible solution keeping the data > model while working with the new OGC APIs at the same time: For resource > base, we can keep/extend the model to keep compatibility with the CSW > endpoint (which should not go away in my opinion). We can then re-use the > resource-base model as a pygeoapi backend to support OGC-API Records, in a > similar way we do with pycsw. For the rest of OGC APIs I echo @tomkralidis > <https://github.com/tomkralidis> opinion. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#6685 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAJYARJNYTBLVLHOWGBOGALSSXWMFANCNFSM4UJDCTKQ > > . > -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Alessio Fabiani @alfa7691 Founder/Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A - 55054 Massarosa (LU) - Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 331 6233686 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6685 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYARJTQP6HF6UR3ZEZVPTSSX337ANCNFSM4UJDCTKQ> .

-- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Alessio Fabiani @alfa7691 Founder/Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A - 55054 Massarosa (LU) - Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 331 6233686 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

giohappy · 2020-12-02T22:16:10Z

As already said by @afabiani this is neither a proposal for the API of GeoNode 4, neither for a GeoNode "core". We wanted to make a step beyond the current one, experiment with new use cases as required by our clients which more and more ask for an extende REST interface (including for permissions, users management, etc.). Of course this API reflects somehow the underlying model (including the parts of it that we don't like much).

I don't see any conflict with pygeoapi here. Nothing prevents, now and in the future, haning both the interfaces. We already have pycsw for CSW, we can havd pygeoapi for OGC API. What are your concerns here @francbartoli?

gannebamm · 2020-12-03T11:53:14Z

I see that the proposal does not interfere with other APIs and therefore voted my +1. Nonetheless we should give developers and users a clear way to interact with GeoNode API-wise. Having many different APIs alongside could cause confusion and convolute the codebase. This is something we should keep in mind.

giohappy · 2020-12-03T12:09:59Z

I totally agree @gannebamm. Sometime users ask questions or requirements about the GeoNode "API", but then I discover they were talking about the OGC APis. Probably a dedciated section inside the docs might help to clarify the different exposed APIs.

* [WIP] GeoNode API v2 - Prototype * [ref. #6388] Error when installing core GeoNode into virtual environment * - GeoNode REST APIs v2 - Swagger schema * [ref. #6388] Error when installing core GeoNode into virtual environment * - GeoNode REST API v2 : Map and MapLayers endpoints * - GeoNode REST API v2 : Adding workflow ResourceBase metadata fields * [Dependencies] Removing conflicts * - Minor fixes and improvements * - Improving ResourceBase REST api permissions * - Improve Dynamic REST Sorting/Filtering * - Expose more metadata fields * - Expose more polymorphic_ctype_id field for filtering * - Improve Metadata Fields serialization in order to make it searchable/filterable * - Updating style sheets * - Introducing "DynamicSearchFilter" exposing search_fields * Merge branch 'master' of https://github.com/GeoNode/geonode into rest_api_v2_proof_of_concept # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit. * - Allow gs proxy to parse workspace prefix dynamically * - Downgrade pyproj to 2.6.1 * Revert " - Downgrade pyproj to 2.6.1" This reverts commit a702740. * - Improve Dockerfile * [Issue #6414] Use Django storage API in delete_orphaned_* functions * [Issue #6414] Use Django storage API in geonode.qgis_server.tests.test_views * [Issue #6414] Use Django storage API when generating document thumbnails * [Issue #6414] Thumbnail generation fix for local storage * Add thumbnail convenience functions * Cleanup Django storage API changes * [Hardening] Minor improvements and error checks * [Minor] Headers and formatting files * - Adding "OAuth2Authentication" to the default REST APIs auth_classes * - Fix "get_perms" api issue with non existant keys * - Fix "get_perms" api issue with non existant keys * [APIs] superusers have always perms to change resources * [Dependencies] bump djangorestframework to >=3.1.0,<3.12.0 * - Improve IsOwnerOrReadOnly REST permission class * - Improve IsOwnerOrReadOnly REST permission class * - Improve IsOwnerOrReadOnly REST permission class * - Fix Recenet Activity List for Documents * [Hardening] - Recenet Activity List for Documents error when actor is None * [Frontend] Monitoring: Bump "node-sass" to version 4.14.1 * [Frontend] Bump jquery to version 3.5.1 * [Fixes: #6519] Bump jquery to 3.5.1 (#6526) (cherry picked from commit e532813) # Conflicts: # geonode/static/lib/css/assets.min.css # geonode/static/lib/css/bootstrap-select.css # geonode/static/lib/css/bootstrap-table.css # geonode/static/lib/js/assets.min.js # geonode/static/lib/js/bootstrap-select.js # geonode/static/lib/js/bootstrap-table.js # geonode/static/lib/js/leaflet-plugins.min.js # geonode/static/lib/js/leaflet.js # geonode/static/lib/js/moment-timezone-with-data.js # geonode/static/lib/js/underscore.js * Merge branch 'master' of https://github.com/GeoNode/geonode into rest_api_v2_proof_of_concept # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit. * [Hardening] Re-create the map thumbnail only if it is missing * Fixes error with GDAL 3.0.4 due to a breaking change on GDAL (https://code.djangoproject.com/ticket/30645) * Fixes error with GDAL 3.0.4 due to a breaking change on GDAL (https://code.djangoproject.com/ticket/30645) * - Introducing REST APIs for "Docuemnts" resources too * - Bump django_mapstore_adapter to 2.0.6 / Bump django-geonode-mapstore-client to 2.0.9 * - Travis and LGTM fixes * - Swagger OAS3 * - Adding REST API v2 Test Suite * - Implementing API v2 "extent" filter * - Travis and LGTM fixes Co-authored-by: Matthew Northcott <matthewnorthcott@catalyst.net.nz> Co-authored-by: Toni <toni.schoenbuchner@csgis.de>

afabiani added the gnip A GeoNodeImprovementProcess Issue label Dec 1, 2020

afabiani self-assigned this Dec 1, 2020

afabiani added this to the 3.2 milestone Dec 1, 2020

afabiani mentioned this issue Dec 1, 2020

[Fixes #6685] GNIP-79: GeoNode REST APIs (v2) #6686

Merged

11 tasks

afabiani closed this as completed in #6686 Dec 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GNIP-79: GeoNode REST APIs (v2) #6685

GNIP-79: GeoNode REST APIs (v2) #6685

afabiani commented Dec 1, 2020 •

edited

Loading

t-book commented Dec 1, 2020

gannebamm commented Dec 1, 2020

francbartoli commented Dec 1, 2020

tomkralidis commented Dec 2, 2020

t-book commented Dec 2, 2020 •

edited

Loading

kalxas commented Dec 2, 2020

afabiani commented Dec 2, 2020 via email

afabiani commented Dec 2, 2020 via email

giohappy commented Dec 2, 2020

gannebamm commented Dec 3, 2020

giohappy commented Dec 3, 2020

GNIP-79: GeoNode REST APIs (v2) #6685

GNIP-79: GeoNode REST APIs (v2) #6685

Comments

afabiani commented Dec 1, 2020 • edited Loading

GNIP-79: GeoNode REST APIs (v2)

Overview

Proposed By

Assigned to Release

State

Motivation

Proposal

Define a common REST router under geonode.api

Define APIs for each GeoNode module

Backwards Compatibility

Future evolution

Feedback

Voting

Links

t-book commented Dec 1, 2020

gannebamm commented Dec 1, 2020

francbartoli commented Dec 1, 2020

tomkralidis commented Dec 2, 2020

t-book commented Dec 2, 2020 • edited Loading

kalxas commented Dec 2, 2020

afabiani commented Dec 2, 2020 via email

afabiani commented Dec 2, 2020 via email

giohappy commented Dec 2, 2020

gannebamm commented Dec 3, 2020

giohappy commented Dec 3, 2020

afabiani commented Dec 1, 2020 •

edited

Loading

Define a common REST router under `geonode.api`

t-book commented Dec 2, 2020 •

edited

Loading