[Proposal] Project system

Overview

The goal of this improvement is to simplify the current MapStore project creation and update system.

Proposed by

Stefano Bovio (author)
Lorenzo Natali (reviewer and writer of the proposal)

Based on a previous analysis by Mauro Bartolomeoli

Assigned to Release

The proposal is for 2021.02.00 or next

State

Motivation

The current project system makes hard to update custom projects because of the complexity and the quantity of files that have to be updated on every MapStore version upgrade.

Actually createProject.js creates a directory structure like the following :

.
// BACK END
├── pom.xml                      <-- main pom (to build back-end)
├── backend                      <-- custom folder for optional back-end custmization
│   ├── pom.xml                  
│   └── src                       
│       ├── main
│       └── test
├── web <-- Various backend configuration files
│   ├── client
│   │   └── libs
│   ├── pom.xml             <-- This builds the project real war file
│   ├── src
│       ├── main
│       └── printing 

// DOCKER
├── docker                        <-- dockerfile and configurations (TODO: define purpose)
│   ├── geostore-datasource-ovr-h2.properties
│   ├── geostore-datasource-ovr-postgres.properties
│   ├── geostore-datasource-ovr.properties
│   └── wait-for-postgres.sh
├── Dockerfile

// FRONT END
// -- configuration files for maps
// -- optionally localConfig.json (that anyway is frequently needed)
├── new.json                      <-- config for new maps, or new context base map
├── config.json                   <-- map configuration for default map (used in georchestra, nothing more                      
├── assets                       <-- optional forlder for 
│   └── placeholder
// -- html copies
├── index.html
├── indexTemplate.html             
├── embedded.html                  <-- copy of various html files
├── embeddedTemplate.html
├── api.html
├── apiTemplate.html
├── unsupportedBrowser.html        <-- unsupported browsers page
├── js                             <-- MAIN DIRECTORY WITH THE CODE
│   └── app.jsx                    <-- SOME CUSTOMIZATIONS

// TEST FILES
├── karma.conf.continuous-test.js  <-- some files for test running, different modes
├── karma.conf.single-run.js  
├── tests.webpack.js        <-- test build file

// BUILD FILES (excluded pom.xml files)
├── build.sh                <-- main build script (runs many mvn, test, npm, docs... in case of the main project)
├── package.json            <-- current project dependencies and scrips
├── package-lock.json       
├── webpack.config.js       <-- dev running build configuration
├── prod-webpack.config.js  <-- production build configuration
├── updateDevDependencies.js <-- utility script

// OTHER FILES
├── MapStore2   <-- MapStore git sub-module
├── version.txt <-- version of the project (retrieved via ajax)
├── LICENSE.txt          
└── README.md

The structure contains many files that are simply a copy of the main project. Moreover maintenance of the createProject script is hard because every change on the product structure, if needed in projects, have to be created also in the template files for project generation.

When we update a project we need to follow the migration guidelines following the general update checklist + the updates required for every increment of version, listed below (Migration from xxxx.xx.xx to yyyy.yy.yy) for breaking changes.

The updates mainly consists in updating any file can be changed. Sometimes the updates can be hard to a new comer or a new user, because there is a lot of complexity that require and advanced knowledge of webpack, mapstore, etc...

Projects may differ a lot in terms of:

front end
- configurations (new.json, localConfig.json ...)
- translations
- basic JS configuration (e.g. address of main configuration files)
- Custom Plugins
- themes
- entry points (webpack build entries)
- html files that includes entry ponits (e.g. embed pages for custom pages, custom independend pages)
back-end
- authentication / authorization system ( LDAP/SPID ...future oauth )
- modules (printing)
- custom entry points

The target of the proposal is to use modern instruments to reduce the number of files and simplify as much as possible the final structure of the project, (making the structure ideally to contain only customizations). In order to improve maintenance of the project system, we want to make MapStore product itself a project, so every effort of upgrading the main product will need to upgrade the project system itself.

Proposal

The Github repository mapstore-project contains the new system in development.

The main idea of this new project is similar to create-react-app.

Host a script for creation of the project
Host scripts for maintainance
create a set of devDependencies that can be included all at once.

Project creation script

npx @mapstore/project create standard

npx is a utility that allow to run node scripts downloading directly from npm. So once you installed npx with npm install -g npx, you can run this command.
@mapstore/porject is the npm lib. create is the command.
standard is the type of project to create. Now standard and geonode exist. We should add extension TODO

note: TODO we should separate the creation script in a different project to avoid to download mapstore dependencies before the time**

New directory structure

The new directory structure is this

├── assets  <-- Ideally contains all assets of the application that have to be copied in the final app
│   └── img
│       └── favicon.ico
├── configs <-- Contains all configurations of the application 
│   ├── newgeostory.json
│   └── new.json
├── js      <-- Contains the main front-end 
│   └── apps  <-- main entry points for build, they import many parts from mapstore framework, and contain the customizations 
│       ├── embedded.jsx
│       ├── mapstore.jsx
│       └── ms2-api.js
├── package.json  <-- main project file
├── pom.xml <-- pom for back-end
├── themes  <-- theme main entry point, allows customization
│   └── default 
│       ├── theme.less
│       └── variables.less
├── translations <-- custom translation files
│   ├── data.de-DE.json
│   ├── data.en-US.json
│   ├── data.es-ES.json
│   ├── data.fr-FR.json
│   └── data.it-IT.json
└── version.txt <-- version of the project

Front end

The package.json is very minimal

{
  "name": "test",
  "version": "1.0.0",
  "description": "mapstore project",
  "eslintConfig": { <-- configuration of eslint externalized (already in mapstore)
    "extends": [
      "@mapstore/eslint-config-mapstore"
    ],
    "parserOptions": { // babel configuration from node modules
      "babelOptions": {
        "configFile": "./node_modules/mapstore/build/babel.config.js"
      }
    },
    "globals": {  // some customizations you can apply to the project
      "__MAPSTORE_PROJECT_CONFIG__": false
    }
  },
  "scripts": { // all these scripts are pre-defined, for each type of project
    "compile": "mapstore-project compile standard",
    "lint": "eslint js --ext .jsx,.js",
    "start": "mapstore-project start standard",
    "test": "mapstore-project test standard",
    "test:watch": "mapstore-project test:watch standard"
  },
  "devDependencies": { // dev dependenies are grouped in one single dependency
    "@mapstore/project": "1.0.4"
  },
  "dependencies": {  // dependencies is only mapstore (this is already in MapStore for new project, with file dependency)
    "mapstore": "git+https://github.com/geosolutions-it/MapStore2.git#mapstore-project" // this should be published on npm
  },
  "mapstore": {} // configuration specific to MapStore
}

The main idea in this project is the standardization of the front-end scripts, with a system of convention over configuration.

Every file in js/app directory will be compiled as an entry point by webpack
Customizations to build system can be done using variables in package.json (globals, mapstore).
HTML files are now ejs templates from mapstore project, and variables are injected inside them (avoiding 2 files for each page)

Back End

The new pom.xml MapStore.war includes only

mapstore.war as dependency, the maven-war-plugin
maven-resources-plugin to copy the content of the dist directory inside mapstore.war.
tomcat7-maven-plugin to run the back-end locally

The maven-war-plugin should already allow to override files in the main war. To make this work we need CI of mapstore.war on our maven repository.

We need define standard procedure to:

add back-end modules like printing
overrides / config (e.g. confgure database etc...)
typical customization (e.g. LDAP integration, SPID ...)
Add custom entry points

Main issues

This system doesn't requires that all the parts of the project are retrieved by version. Not having a mono-repo in this case may make things hard, and need a CI system and a versioning system well defined. Anyway npm allows to use git links or file links to work with "snapshots" versions

TODO to complete list

List of pending tasks

Here a list of pending tasks to complete (draft)

main

Nice to have / suggestions

we should separate the creation script in a different project to avoid to download mapstore dependencies before the time.
by @offtherailz add one or more optional files to insert common configurations for app files. (e.g. position of local config, translation files...). See how to make them optional (file presence/absence or empty file)
by @offtherailz add one or more files to customize some aspects of the build and or dev tools. In this case I refer mainly to the devtool and devServer --> proxy (so mainly for dev environment) that should be easy to customize in a project.

Optional/future

A mono repo that includes these project could be better for maintenance purposes, in the near future, when this is stable. Need to see lerna or yarn-workspaces to simplify the update process. (maybe in a second phase)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly