Contents of ckanext ngds directory

Most of the code in ckanext-ngds lives in the NGDS directory. All of the code is part of one big, gigantic extension into the CKAN framework. In the NGDS directory is where we keep our plugins. Most of the folders in that directory are CKAN plugins that we've built. Here's an attempt to explain what it all does:

base: CKAN is a database-driven system; the base code contains logic for extending the vanilla CKAN database structure to support NGDS. This is also where NGDS custom paster commands live.

contentmodel: Almost all of this code is deprecated; the only function out of this entire plugin that we're still using is this action definition, which checks whether a given csv file follows a specified content model. We're now relying on the USGIN Models API for our USGIN content model dependencies.

csw: All of this code is deprecated since we switched over to CKAN's new CSW server. An effort was made here to build a custom NGDS CSW server; we should see what parts we can scrap and extend into the new CSW server.

geoserver: Contains logic for extending Geoserver support into CKAN. This includes publishing both csv files and zipped shapefiles as WMS/WFS, unpublishing those resources, creating and destroying workspaces and datastores, making database connections, making new CKAN resources out of new services, etc. We also keep our code for handling external OGC services and for doing OGC previews here. Work has begun on taking this plugin and abstracting it into it's own extension. Any new effort on developing this code should be done in the ckanext-geoserver codebase.

harvest: Almost all of this code is not in use, but not necessarily deprecated. There are two ways to extend the CKAN harvester: through a full plugin (with actions, helpers, templates, etc.) or just by extending the CKAN harvester object. Right now we're doing the latter, which means that this object is the only portion of this plugin which is necessary/in use.

histogram: An experiment which somehow made it to the master branch of our project. This code should be ignored.

importer: Should be renamed bulkuploader; this is where all of the code for the now deprecated bulk upload functionality lives here. Bulk upload functionality has been completely removed from ckanext-ngds, which means that this all deprecated code. It will be replaced in future releases of ckanext-ngds with support for UGSIN metatdata compilation datasets, which will allow users to bulk upload data.

indexer: Logic to build full-text indexes of resources in SOLR. It's unclear whether or not this logic actually does anything. Visiting this link will redirect the user to a webpage with the message: Full text indexer is executed successfully. Have fun with searching through documents..... And this command will supposedly initiate the full-text indexing process.

lib: Some paster commands for doing things like compiling client-side scripts, creating an NGDS organization, full-text indexing and bulk uploading data. None of this code looks like it is still in use.

logic: Code for extending CKAN resource and package validation. Functions here expect to receive resource and package information which has already been extended to accommodate NGDS data standards. There are a bunch of file-processor Python scripts contained here which should all be ignored. They were part of a short-lived effort to extract data from uploaded files and display them in the UI.

metadata: Logic for constructing NGDS-compliant ISO19139 documents. Some of the code here is deprecated, like metadata logic for bulk uploads.

ngdsui: Probably the extension with the largest amount of code specific to NGDS which actually functions. Here is where all of the NGDS Jinja HTML templates, JavaScript, LESS and CSS files are organized. A lot of backend logic for communicating with the front-end via REST is kept and exposed here as well.

tests: Unit tests for code in the ckanext-ngds codebase. A lot of the code in this extension has no unit test associated with it and the unit tests which do exist are likely outdated.

If we're to follow Ryan's ideas for modularization:

We'll end up with three CKAN extensions:

• ckanext-geoserver
• ckanext-ogcpreview
• ckanext-ngds

Ryan also suggested that we build ckanext-csw, but OKFN has already done that.

• ckanext-geoserver: Work has already begun on this; an effort to take all of the Geoserver code in NGDS and abstract it so that it has no NGDS dependencies.

• ckanext-ogcpreview: Work has already begun on this, too. Right now it's just being developed as a plugin to ckanext-geoserver, but we can (and probably should) take it and make it it's own extension. It might be an even better idea to just make this extension ckanext-ogc. That way we could support the OGC preview functionality as well as abstract logic and processors for dealing with OGC services (assuming that this isn't already part of ckanext-spatial).

• ckanext-ngds: Considering the amount of dead code living in the current ckanext-ngds build, this really shouldn't be too daunting of a task. We really only need to refactor these parts of the current build:

  • ngdsui
  • logic
  • harvest
  • base
  • metadata

  The brunt of this task lies with refactoring ngdsui. We'll probably end up having to completely rewrite substantial portions of it (contribute form, user management, combine two websites into a single website). Keep in mind that the ngdsui implements a significant amount of functionality that supports operations in the user interface and that a bunch of this will need to be refactored as well. Before any refactoring of the UI happens, we must decide on a single framework to use. The current implementation uses two different front-end web frameworks which is illogical and unmaintainable.