Skip to content
This repository has been archived by the owner on Dec 16, 2024. It is now read-only.

Latest commit

 

History

History
1635 lines (1326 loc) · 110 KB

NotesStandardsPractices.md

File metadata and controls

1635 lines (1326 loc) · 110 KB

M4OPS2 Notes Standards and Practices

Table of contents generated with markdown-toc

Overview

This is an eclectic, pragmatic working document to help me remember how to do things, and evolving as I get more experience (or change technologies!). It has lots of links for useful reference. Be aware that although the intention is there, it is likely that at any time the code will not meet the standards by a long way. There is also lots of stuff here that I do not understand yet!

Whilst the codebase represented here is usually a working system, there is currently quite a bit of the code that is non-functional or not yet working, because it is:

  • redundant
  • in process of being adopted from v1 of M4OPS
  • experimental, or
  • under development

These files are excluded from linting by virtue of

  • being in a folder called "unused" (per src/.eslintignore)
  • being in a folder with the word OLD in the name (per src/.eslintignore, or
  • starting with the line /* eslint-disable */

Although fairly complex, this is a simplified approach to aid understanding, development and maintenance, and has been adapted in the light of experience. See other possibilities in M4OPS2 Other Technical Possibilities, although as we go on any that are actually used will be moved from there into this document.

Notes on this document

  • To see diagrams you need mermaid installed
  • Do not use , ( ) / in headings else links from TOC will not function without manual change

Some useful Terminology

  • Semantic versioning (semver) - Change major number when it may break existing usages

  • a software stack is a set of software subsystems or components needed to create a complete platform such that no additional software is needed to support applications

  • polyfill - a snippet of code that patches a piece of functionality that's missing in some browsers.

    • ponyfill provides that functionality as a standalone module you can use – see this discussion

  • Function currying is the process of successive partial applications, until the last argument is given at which point the result of the function is returned

  • functional programming is a style of building computer programs—that treats computation as the evaluation of mathematical functions and avoids changing-state and mutable data. In contrast, imperative programming changes state with commands in the source code, the simplest example being assignment. (Ignore as too complex?)

  • A closure is the combination of a function and the scope object in which it was created. Closures let you save state — as such, they can often be used in place of objects. (Not used here)

  • CRUD - create, read, update, and delete

  • REST stands for Representational State Transfer -The fundamental concept within REST is that of a resource - the object we are operating on. For each resource we define what methods can operate on that resource. Methods such as GET, POST, UPDATE, DELETE.

  • HOC - higher-order component - a function that takes a component and returns a new component (used in React), compare in Vue.js ..

  • In a mixin you can put any component’s methods and they will be merged with the ones of the component that uses it.

  • a variadic function is one of indefinite arity, i.e., one which accepts a variable number of arguments

  • Parentheses, Braces, or Brackets?

    • Round: Parentheses () for function calls, conditional statements, or enforcing Order of Operations.
    • Curly: Braces {} for the declaration of Object Literals, or to enclose blocks of code
    • Square: Brackets []for accessing the properties of an Object (or the elements of an Array)

  • ORM - Object-relational mapping for converting data between object-oriented and relational forms

  • Duck Typing is better than using instanceof

  • memoization (memoisation) is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.

  • a Promise is an object that is used as a placeholder for the eventual results of a deferred (and possibly asynchronous) computation.

  • PWA (Progressive Web Apps)

    • a hybrid of regular web pages (or websites) and a mobile application
    • see 4 important points to know about Progressive Web Apps (PWA)
    • the Service Worker is a script that the browser runs in the background separate from web page is the backbone of every PWA
    • also Accelerated Mobile Pages (AMP) provides reliably fast web components for first load

  • BEM - "Block, Element, Modifier” naming convention - see Introducing BEM: The popular CSS naming convention

  • VNode - a “virtual node” containing information (returned by createElement) describing to Vue what kind of node it should render on the page, including descriptions of any child nodes. The entire tree of VNodes is called the “Virtual DOM”.

The stack

The main stack is

Folder and File set-up

**client** Folder Structure
├── dist - result of the build process
├── public
├── src
└── tests (TODO)

**public** Folder Structure
├── index.html (main entry point, includes  \<div id="app"></div> into which the whole app is injected by main.js)
├──  ...
├── OPS (one folder for each OPS)
│   ├── HcN - all the images, html files etc used for HcN in M4OPS
│   ├── HNB
│   └── ...
└── img (used by the system)
    └── ...

**src** Folder Structure: based on [How to Structure a Vue.js Project](https://itnext.io/how-to-structure-a-vue-js-project-29e4ddc1aeeb)
├── main.js - the first script loaded
├── App.vue - the first Vue component - contains all the others router.js
├── router.js - 'routes' URLs to views
├── assets - any assets that are imported into components
│   └── ...
├── components - All the components that are not routed (ie not views)
│   └── ...
├── global
│   ├── components - a few that are generic, and potentially used anywhere
│   │   └── ...
│   ├── constants ... system-wide
│   ├── utils ... functions used in various places
│   ├── plugins - TODO
│   │   └── ...
│   └── styles - any not included in SFCs
│       └── ...
├── initialising (used on first load)
│   └── ...
├── modules - for routines by application area (will change) - each potentially has index.js, components, submodules, utilities
│   ├── demo - to do
│   ├── framework - header, sidebar
│   ├── geography - ??
│   ├── mapping
│   └── params - parameters to the URL
├── store
│   ├── index.js - where we assemble modules and export the store
│   ├── mutation-types.js - Vuex constants
│   └── modules - each potentially has index.js, components, submodules, utilities
│       ├── ...
│       └── vuexApi - getters etc but no index.js
├── translations - Locales files (for eg using Vue-i18n) TODO
│   └── index.js
├── views - the components that are routed in router.js eg  xxx/dashboard
│   └── ...
└── other folders ... such as filters,or constants, API.

**server** src Folder Structure
├── controllers - the various 'find' functions to get data from the database
├── middleware TODO
├── models - defining the Mongoose/MongoDB schemas
└── routes - the routes from the URL to the controllers

Notes: Each folder

  • can have a README.md file describing it
  • can have .gitignore for files to be excluded from git
  • can have .eslintignore for files to be excluded from eslint
  • can have .eslintrc.json to direct eslint and give it parameters

Our OPS images etc go in the public folder - see this

Naming conventions

See under Standards and styles

Visual Studio Code

  • Our main development environment is VSCode - intelligent code editing
  • Documentation and Chrome debugging in VSCode
  • a workspace (defined by an editable workspace file) is defined to be one or more folders and VSCode settings - we will not use them, unless it becomes necessary
  • Note that almost all VSCode settings (including cSpell words) are done at the folder level (eg in C:\projects\m4ops.vscode\settings.json), rather than at the User (C:\Users\Peter2\AppData\Roaming\Code\User) or workspace level
  • Similarly for ESLint
  • as the VSCode shell defaults to PowerShell on Windows 10 we set the terminal.integrated.shell.windows in user settings to "C:\Windows\System32\cmd.exe"
  • Use Ctrl+Shift+P for the list of commands (Ctrl+S to save - but we have set it to save automatically), and use Ctrl+space for context-sensitive snippets (or start typing)
  • Ctrl+/ toggles lines as comments
  • For documentation and markdown see separate Documenting section below
  • Useful about Node.js and Express in VSCode
  • VS Code can do that?!

VSCode extensions used

GIT for source code version control

Eslint for proofing code

Configuring eslint

  • See guide to configuring
  • List of rules
  • We do use some Configuration (in-file) comments)
    • /* eslint-disable */ or // eslint-disable-next-line or // eslint-disable-line
      • for rules eg no-console, max-len
      • can just right click in VSCode to implement this
  • We do not use the eslintConfig field, but do use .eslintrc.json files
    • /m4ops/.eslintrc.json has the basic configuration that applies to both client and server code
      • do not include any rules (even common ones) as the whole rules object is overwritten by that in either the client or server file
    • /m4ops/client/.eslintrc.json has any specific configuration needed for the client (Vue) system, including rules
      • in particular it has the kebab-case names of imported components, for which the PascalCase rule is ignored
    • /m4ops/server/.eslintrc.json is similar for the server (Express) system, including rules
  • We also use an /m4ops/.eslintignore file in any folder we run eslint in (note that only one such file is used)

Documenting

HTML5 standards

JS and ES6 standards

Special functions and routines

Regular Expressions

with a global regular expression, .exec is meant to be used in a loop, as it will retrieve all matched subexpressions. String.match does this for you and discards the captured groups. Without the /g both exec and match return the first match ([0]) and any captured groups ([1] etc)

Node

  • Node.js® is a JavaScript runtime built on Chrome's V8 JavaScript engine.
  • npm - the package manager for javascript (documentation)
  • npm globals are in C:\Users\Peter2\AppData\Roaming\npm
    • claudia, jest, nodemon, sass, vue

Making an npm package

Looking after vfg-display-fields

Set up
  • (using "building your first node module" from above)
  • choose the name (vfg-display-fields)
  • npm view vfg-display-fields => confirms it does not exist yet
  • create repository in github, and clone to c:\projects
  • npm init => initialises package
  • edit, commit, merge into master, synchronise with repo
  • see guide to test package
  • npm login (p6 lower case, )
  • npm publish

Maintenance

Client Overview

The Starting point

client/public/index.html is the file a browser will open - this provides a title and a root div (app), and runs client/src/main.js which takes the output of App.vue and puts it into the root div. (registerServiceWorker.js is used just in production)

The client is the system that the user interacts with. The server is the system that the client system can use, eg if it wants to store or retrieve data). We keep these systems separate, and develop them separately.

When the client needs to communicate with the server it sends an asynchronous request (with a promise attached) and when the server is done it notifies the client the promise has been completed. The client can then do any more processing dependent on the promise being completed. [More on this later]

(Almost) all of the state of the client is held in the Vuex store, and access to it is very structured [see later].

graph LR;
  M[Vuex store] -->|getters| C[Client]
  C --> A[Actions]
  A --> |mutations| M
  C -->|POST or GET| S[Server]
  S -->|Promise| C
Loading

Vue.js

Scoped Slots

Complex components

Vue Router

M4OPS2 URL structure

path parameters

(specific order separated by /) - see client\src\router.js "path: '/maps/"

  • :ops: code for the OPS (optional - default HcN)
  • :layers: zero or more layer titles (starting at layer 0) separated by / (if none then default is Bing%20Aerial/OSM)
    • layer titles can be alphanumeric, including -_ and spaces
  • :opacities: zero or more opacity numbers (% starting at layer 1, as the base is always 100% opaque) separated by / (if none then default is 50 for rasters, 100 for feature layers)
  • :ZoomOrFitTo: zoom level preceded by Z eg 18, or FitTo preceded by F eg 1 (default is the HomeView)
    • (FitTo was Extent and is the number of the layer to initially fit to)
  • :Lon/Lat: Longitude/Latitude decimal degrees, eg -0.0318640/52.3304020 for central Needingworth
    • (This is in 'EPSG: 4326' or Spherical Mercator, and is irrelevant if a FitTo is specified)
Examples

(for now the final / is important) Basic Pubs Protected Layer 2 Spinney Way HNB is Protected

query parameters

  • (key/value pairs, or just the key - meaning true)

  • NoCHNG means you cannot change the OPS

  • Showlevel= (default 9999) for demonstrations starts at 0 then goes up at cutoffpoints

  • Splash= html text for splash screen when M4OPS first opens, can include abbreviations (#..#)

    • one word abbreviations will, if necessary, have the word Splash appended, and be surrounded by #..#
    • (thus Splash=Spyglass becomes #SpyglassSplash#)
    • (and Splash=25inch becomes #25inchSplash#)

  • Tab= the initial advanced option tab to show:

    • Actions - PNG, Demo, icons (the default)
    • MFL - Modifiable Feature Layers
    • Upload - Upload, compile
    • Time - Time sliders

  • Displaystyle= (initial view)

    • onemapOpacity - One map with Opacity slider, or
    • sidebyside - Side by Side maps, or
    • onemapSpy - One Map with Spyglass

  • Colours= initial colour scheme for features

  • Click= the initial drop-down value - one of:

    • no - No lat/lon click
    • M4OPScsv - M4OPS lon;lat csv
    • M4OPSparam - M4OPS parameters
    • csv - lat,lon csv
    • geojson - {lon,lat} GeoJSON
    • EPSG3857 - EPSG:3857 (x/y)
    • HDMS - DegMinSec N/E
    • GeoHack - GeoHack links
    • Featureid - Feature id

  • Green if you want the background in development to be the normal Green

  • NoShift if you want the layers NOT shifted east or north

  • LoadwA if you want tiles loaded during animations (may improve the user experience, but can also make things stutter on devices with slow memory)

  • LoadwI if you want tiles loaded while interacting with the map (ditto)

  • (Mouse if you want the next feature to be Georeferenced to appear by the mouse pointer)

Other M4OPS parameters
  • from forum
  • File= filename of json to use, default M4OPS (.json) NA

Events

Vuex

  • Vuex - State management
  • the store is injected into all child components of the root and will be available on them as this.$store
    • other routines can access the store and its getters etc by simply importing it
    • "getters" in the store are computed properties that can be referenced in components via store.getters.xxx (can be functions)
    • a mutation is a function eg increment-mutation (state) { state.count++ } - always synchronous
      • mutation types are UPPERCASE and defined in mutation-types.js
    • an action commits a mutation eg increment-action ({ commit }) { commit('increment-mutation') }
    • we dispatch actions in components with this.$store.dispatch('xxx')
    • in components use the helper functions
      • mapState to generate computed getter functions
      • mapGetters to map store getters to local computed properties
      • mapActions maps component methods to store.dispatch
    • inside module actions and getters, the root state will be exposed as as well as the module's state
    • We do not Namespace vuex Modules (except alert)
  • Various Vuex Utilities
  • Not using vuex-pathify although it simplifies the Vuex wiring
  • Vuex getters are great but use mapState for simple ones
  • Vuex Instance Properties etc - available to all components as this.$store.xxx

Vuex plugins

Typescript in Vue

HTTP Requests

Authorisation

  • We use Jason Watmore's approach, and have absorbed his routines into ours (users modules), adapted to our standards
  • The source of truth is in the server's (MongoDB) database

Our structure

  • The key constraints are on:

    • being able to see an OPS at all
    • being able to see a layer
    • being able to upload a file
    • being able to compile
    • being able to set up, alter, delete users

  • OPS are one of:

    • Unprotected - anyone can see it (default)
    • Protected - users need a specific right for that OPS to see it (and then can see all layers)

  • Each Layer (settled and modifiable/MFL) is one of [see protectionStatusEnum]:

    • Unprotected - anyone can see it (default), if they can see the OPS
    • Protected - logged-in users can see it (eg with Census images), if they can see the OPS
    • Personal - only the one user (and admin) can see it
    • Test - only an admin (globalAdmin, or opsAdmin for that OPS) can see it, and then only if the Test switch is On

  • Everyone (including [not logged-in] Guests) can see all Unprotected layers in all Unprotected OPS

    • and login as a User (thereby acquiring any specific rights they have)
    • and register as a User (but without any specific rights)
    • but not Upload, compile, create or change an MFL

  • Logged-in Users (without needing any specific rights) can see all Protected (and Unprotected) layers in all Unprotected OPS

    • and their own Personal layers
    • and create an MFL, or change an MFL they can see

  • In addition, Users can have none, one or more of the following specific rights [see userRightsEnum]:

    • opsViewer - as well as everything a Logged-in User can do
      • can see all Protected layers in the specific Protected OPS (not necessary if the OPS is Unprotected)
      • but not Upload, compile, create or change an MFL
    • opsTeamMember - can do everything the opsViewer can do
      • and Upload, compile, create or change an MFL in the specific OPS
    • opsAdmin - can do anything in the specific OPS
      • including everything an opsTeamMember can do
      • and see all Personal layers in that OPS (from everyone)
      • and Register/edit Users as opsTeamMembers and opsViewers
    • globalAdmin - can do anything

See client\src\global\constants.js for protectionStatusEnum {UN,PD,PL,TT}, userRightsEnum = {9_NO,6_OV,4_OT,2_OA,0_GA}

Client system

  • See Jason Watmore's approach - client
  • clone of Jason Watmore's system is in C:projects/vue-vuex-registration-login-example
  • routines in \client\src
    • \store\modules\users - defines the user aspects of the Vuex store (state, actions, mutations)
      • account.module.js - the current user and their loggedin status
      • alert.module.js - alert state: type and message
      • users.module.js - all users
    • \views
      • LoginPage.vue - includes logout
      • RegisterPage.vue
      • ManagePage - (originally HomePage) sb only accessible by administrator
    • \modules\users
      • _helpers
      • auth-header.js - generates a header
      • fake-backend.ts - intercepts certain api requests and mimics the behaviour of a real api
      • user.service.js - all backend api calls, and logging in/out (handleResponse handles if the JWT token is no longer valid for any reason)
  • the URLs that are public are in router.js (beforeEach)
  • Could use Route Meta Fields to determine navigation limits

Server system

JSON Web Tokens (JWT)
  • JSON Web Tokens are an open, industry standard method for representing claims securely between two parties - and the library jwt.io allows you to decode, verify and generate JWT
  • We use jwt-express to facilitate this on the server
  • As well as in REST APIs, JWTs can also be used in everyday web applications to store session data in a cookie - the JWT contains all the information we need
  • We ignore the issue of CSRF/Cross-site request forgery
  • JWTs can be stale (different than expired) after a period of inactivity, so sometimes need to ensure their JWT is fresh, as well as valid
General Server system
  • See Jason Watmore's approach - server/API
  • clone of Jason Watmore's system is in C:projects/node-mongo-registration-login-api (server)
  • routines in \server\src
    • \models\User.js for the Schema/model
    • \modules\users for routines:
      • config.json - has just the secret
      • error-handler.js - centralised handling of api errors (TO DO include other modules' errors?)
      • jwt.js - calls expressJwt to revoke the user token unless the route is specified as not requiring authentication
      • user.service.js - contains the core business logic, encapsulates all interaction with the mongoose user model, and exposes a simple set of methods
    • app.js includes a new major route: '/users', usersRoute
    • \routes\user.js defines the actions for each sub-route (including our normal Controller aspects)
  • the API calls that are public are in jwt.js, and to begin with we allow all calls

Management

  • A list of users is given at menu option 'Manage', and they can be deleted from there

Other Approaches

Demo

  • For demo could use Vue Tour - a quick and easy way to guide your users through your application.

Icons

CSS

Useful components

Dates and Times

  • Moment.js - Parse, validate, manipulate, and display dates and times

Buefy

  • Buefy Components include all we need except (see below):
    • Cascader (asked for)
    • Slider
  • source monitor ? for infinite scroll

Element UI components

(We are replacing Element by Buefy when possible)

Vue Slider Component

Vue Forms

Plain
VueFormGenerator
VFG documentation
  • multiple, and multi: true fields, is about fields that can have more than one value (cf see this)
  • featured means bold
  • validateDebounceTime (milliseconds) is used eg on text fields with validation
  • fields can have complex definitions, and even incorporate buttons
  • core field types are checkbox, checklist, input, label, radios, select, submit (button), textArea
  • other field types we might use include cleave, image, pikaday, switch, vueMultiSelect (trying to avoid jQuery)
  • fields can be grouped
  • Built in Validators are number, integer, double, string, array, date, regexp, email, url, creditCard, alpha, alphaNumeric - Custom Validators are possible
  • for style see client\vfg.css.txt
Other on forms

Colors/Colours

  • For the (optional) color type VFG uses Spectrum Colorpicker, but this needs JQuery and there is no recommended non-JQuery equivalent
  • We use TinyColor

Modals

Our Modals structure
  • Essentially
    • the modals are all defined at the top level (M4OPSView) but hidden until 'wanted'
    • a modal is 'wanted' when showPortal sets title (in store/forms)
    • showPortal also always sets portalName (eg ModalForForms), and (usually) actionTextsArray
  • the Modal actually appears in portal-target in App.vue, a standard component from portal-vue
    • this has a name bound to store/forms/portalName in Vuex
  • one main Modal form component is ModalForForms (for general M4OPS forms such as LogIn), which handles
    • linking to the portal-target (with name "ModalForForms")
    • whether it shows (only if store/forms/title !== NOPORTAL - and the portal-target has the right name) or not
    • the background and closing (hidePortal sets forms/title = NOPORTAL)
    • the modal-card classes for head, body, foot etc
    • the VueFormGenerator form, whose spec is defined by the 'thisFormSpec' getter, which
      • assumes the value of forms/formId has been set (by showPortal)
      • collects the relevant mode, schema, options from the Forms database
    • the text on button(s) (from store/forms/actionTextsArray)
    • the Submit action(s) (handleSubmit is hard-coded for now, with a switch on formId)
  • This main Modal component is switched on from elsewhere by the actions showPortal({}), eg in
    • ActionsPane and UserStatus
    • it is switched off internally by hidePortal()
    • it needs showPortal to have also set: formId
  • Note that ModalForForms uses the VuexApi data as spec, thus its '(vfg_)model' is changed by any form entries
    • for passwords, and any other sensitive data, the action to clear it should be included
    • eg dispatch('clearFormField', {formId: 'LogIn', fieldName: 'password'});
    • this data is imported from client\src\modules\forms\vfgData\forms.json
  • Another main Modal component is ModalForMessages, similar to ModalForForms, except
    • it needs showPortal to have set: messagesArray
    • it just displays the text from the messagesArray, without using vfg
  • Another main Modal component is ModalForOPSForms (for OPS-specific forms):
    • it needs showPortal to have set: ldid (of a vector layer)
    • it expects the specified layer to have a 'formId' property (eg 1B) corresponding to an OPSForm in the OPS' FormsArray
    • it uses the getOPSFormByLdid getter to get the schema and formOptions
    • it has a local copy of the model so nothing in the form data is overwritten
      • but the initial values and updated values therefore need to be handled separately

Other Vue aspects

Standards and styles

  • See the official style guide 1 > 2 > 3

  • Naming conventions

    • Component names are always multi-word, except for the root “App” component.
    • Each component is in its own file.
    • Filenames of single-file components (.vue) are in PascalCase.
    • Components that are only used once per page begin with the prefix “The”, to denote that there can be only one (eg TheNavbar.vue)
    • Child components include their parent name as a prefix (eg a “Photo” component used in “UserCard” is named UserCardPhoto).
    • Always use full name instead of abbreviations in the name of components, eg UserDashboardSettings.
    • Folder names are in lower case.
    • Non component filenames are in camelCase according to the job that they perform, eg userDropdown.js.
    • For components it's generally best to use PascalCase in the JavaScript, and kebab-case in the template - but Vue sees them as the same.
    • Beware the kebab/Pascal issue (OPSDetails would have become o-p-s-details, so we use opsdetails)

  • Component data must be a function, as in export default {data () {return {foo: 'bar'}}}

  • Prop definitions should be as detailed as possible, including where possble a validator function

  • Always use key with v-for

  • Never use v-if on the same element as v-for

  • Styles in components should be scoped (except in the top-level App component and in ?? view components styles may be global)

    • and use class selectors (eg .btn-close) rather than element selectors (eg button)

  • Always use the $_ prefix for custom private properties in a plugin, mixin, etc

  • Base components (pure components) that apply app-specific styling and conventions should begin with the prefix Base

  • Components with no content should be self-closing in single-file components, string templates, and JSX - but canot be in DOM templates

  • Where possible we use Single File Components (.vue) - needs Webpack, and has all in the one file:

    • <template> - HTML code template
    • <script> - JavaScript logic
    • <style> - CSS styling

  • Elements with multiple attributes should span multiple lines, with one attribute per line (similar to JS object properties)

  • Don’t use arrow functions on an options property or callback, since arrow functions are bound to the parent context, 'this' will not be the Vue instance as you’d expect

  • Don’t use arrow functions in methods and computed properties as they almost always reference this to access the component data

  • Do use arrow functions for filters (see also the vue2-filters package)

  • “Mustache” syntax (double curly braces): {{}} for interpolation (of component data into a component template)

  • Component templates should only include simple expressions, with more complex expressions refactored into computed properties or methods.

  • Complex computed properties should be split into as many simpler properties as needed.

  • Non-empty HTML attribute values should always be inside quotes (even though without spaces they don't need to be)

  • Directive shorthands (: for v-bind: and @ for v-on:) should be used always or never (we choose always)

  • Component/instance options should be ordered consistently - see this list

  • Attributes of elements (including components) should be ordered consistently - see this list

  • In .vue files we do not indent what is in the <script> or <style> blocks (even though Vetur could do it)

  • For a comparison of Methods vs Watchers vs Computed Properties see page 90 of Vue Handbook.pdf

  • Can do dependency injection (provide and inject - sort of 'long-range props'), see also, and vue-inject - but beware

Vue CLI 3

  • Vue CLI Provides Standard Tooling for Vue.js Development

  • To run a project > npm run serve

  • See Deployment

  • For amazon S3

  • (Remember that npm install -E (or --save-exact) ensures that the current version is not updated)

Sources of truth

Current OPSCode

  • state.mapping.currentOptionArray[3]
  • default initialOpsCode
  • getter: use mapState
  • setter via updateCurrentOptionArray
  • Note that store.getters.place.OPSCode holds the OPSCode for the OPS currently loaded into place

Current User

  • store.state.users.account.user
  • has .username and .status.loggedIn
  • default none (username === 'Guest')
  • localStorage.getItem('user') is used to keep user logged in between page refreshes - includes jwt token
    • so is kept updated by routines in user.service.js, and used in authHeader
    • but should not be used for any other reasons

Current Chosen Layers

  • store.state.mapping.mainmap.chosenLayers (array)
  • each has .ldid, .opacity (0-1), .displaytype (A or B)
  • getter chosenLayersMainmap
  • set by setLayer and setOpacity
  • there is also the simpler store.state.mapping.chosenRhLayer which has just ldid
  • initialised to initialStateChosenLayersByOpsCode(opsCode) - none yet
  • default initialStateChosenLayers.default [Bing Aerial, OSM]

Babel

Basics of Babel

Bundler - Webpack

Debugging

Testing

OpenLayers

Use version number eg v5.3.0 (which is the one we use) or latest (the original M4OPS used v4.6.5)

OL and Vue

Projections

Projections - Background

  • FAQ: What projection is OpenLayers using? - default is EPSG:3857

  • Coordinate Systems Worldwide

  • See the note on projections

  • this note on Coordinates in M4OPS includes something on projections

  • In M4OPS v1 we had: a Note on projections, From this discussion

    • The data in Open Street Map database is stored in EPSG: 4326 (Spherical Mercator: decimal degrees)
    • The Open Street Map tiles and the WMS webservice, are in EPSG 3857
    • So it is usual to have to transform the basic maps from 'EPSG:4326' to 'EPSG:3857'
    • When you configure a source with features, you have to provide them in the view projection (ie in EPSG:3857) as no transformation is possible.

  • Correction Features are always geojson, and always 'EPSG:4326'. Not sure what the statement above means

  • The values for HcN in the Testbed are

    • lon0 = -3970;
    • lat0 = 6860390;
    • d = 300;

  • Other than that we do everything with GeoJSON in Lon/Lat ('EPSG:4326'), and the View is 'EPSG:3857' (metres) See these insights

  • in M4OPS v1 we convert from the featureProjection: 'EPSG:3857' to the dataProjection: 'EPSG:4326', decimals: 8}) when we writeFeatures

  • See Vuelayers on Global data projection

Projections - Basics

  • EPSG:4326 is WGS 84 -- WGS84 - World Geodetic System 1984, used in GPS, or Spherical Mercator: decimal degrees
    • and HcN is [-0.0322294, 52.3305610] (lon, lat)
  • EPSG:3857 is WGS 84 / Pseudo-Mercator -- Spherical Mercator: metres
    • used by Google Maps (EPSG:900913), OpenStreetMap, Bing, ArcGIS, ESRI
    • and HcN is [-3591, 6860098] (x,y) - (metres East, metres North)
    • We do all georeferencing work based on the EPSG:3857
  • EPSG:27700 is OSGB 1936 / British National Grid -- United Kingdom Ordnance Survey
    • and HcN is [534174, 272097] or TL3417472097
  • Northings and Eastings
    • and HcN is 52° 19′ 50″ N 0° 01′ 56″ W

Projections - What we do

  • For the internal view projection we use EPSG:3857 (the default) - this is the projection with which OpenLayers components will work (ol.View, ol.Feature etc)
  • For the data-projection property on the vl-map component we use EPSG:4326 (as the demo does)
  • We assume that all vector layers are geojson, and always read (and written) as 'EPSG:4326'
    • Thus for plain coordinates, and GeoJSON encoded features or geometries there is a thin projection transform layer between Vue (EPSG:4326) and OpenLayers (EPSG:3857)
  • to add a projection use createProj, addProj from ol-ext Projection transform helpers
    • this has lots of other useful transform functions
  • TODO
    • shiftingProjections.js - we have not adapted yet (see ProjectionsArray in Place)
    • extents in layerAndSourceTile.js
    • projection in layerAndSourceWms.js
    • projection in mousePosition
    • Use the LayerDefs (string) projection member of their sourcedef

Server Overview

When the client needs to communicate with the server it sends an asynchronous GET or POST request via eg fetch(/continents). The corresponding route is defined in the server's routes folder (and referenced in its app.js). When the server is done it notifies the client, which can then do any more processing dependent on the promise being completed.

graph LR;
  C[Client] -->|POST or GET| S[Server]
  S -->|Promise| C
Loading

The main program is app.js which sets up connections to the database, (express) routes, and listens on a port.

Each route (including with parameters eg :id) is

  • in a file in the server/routes folder, which is
  • referred to in app.js
  • can be called from the client by eg fetch(/continents)

Routes

  • All on localhost:5000 (and formatted by JSONView)
    • /places - list of studies in M4OPS
      • /places/xxx - OPSDetails for XXX study, with all its bits and arrays
    • /continents - continent/country/location/study (each has M4OPS:true where it is/contains an included study)
    • /m4opsdata - the M4OPS.json file, with all its bits and arrays
    • /featurelayers/OPS_Xxxx - the given feature layer from the given OPS in FeatureLayers

Connecting to MongoDB

The MongoDB URL (including database) is set in the environment variables (qv)

For each type of data held in the database, (via Mongoose) we have

  • a data structures (or schema) defined in a file in the server/models folder
  • a model defined in the same file
  • a controller in the server/controllers folder that queries, deletes and/or updates the data using MongoDB
  • call(s) from routes to the controller to do its work
graph LR;
  C[Client] -->|POST or GET| S[Server]
  S -->|Promise| C
  S -->|model & controller| M[Mongoose/MongoDB]
  M -->|Promise| S
Loading

Notes

  • middleware
  • server needed this then npm i before running with nodemon

MongoDB

  • provides users with a NoSQL document database system (Open Source) Documentation

  • we are on v4.0

  • Install MongoDB Community Edition on Windows

    • Start windows service: Command line as Administrator “net start MongoDB” (not Powershell)

  • MongoDB is designed to be run in trusted environments (ie Localhost) - but in production we use authentication

  • URL (if we needed it) is mongodb://localhost:27017/m4opsdb

  • See Introduction to mongodb and Getting Started

  • MongoDB Compass provides a GUI, queries, CRUD – use localhost:27017 documentation, Getting started

    • Example query on Places {OPSCode:"HcN"}
    • If problem, use Resource Monitor: cmd as Admin, resmon.exe -> CPU tab -> in handles type Compass, and Kill one (will kill all) associated processes - and wait for it to happen.

  • for the MongoDB interactive shell see the manual, or this tutorial - example use:

    • mongo
    • use m4opsdb
    • show collections
    • db.M4OPSData.drop()

Imports into MongoDB

  • MongoDB Compass (but did not work)
  • mongoimport can import JSON and csv into MongoDB
  • Remember that it is one (JSON) document per line, although a single JSON document can span more than one line
  • can use jsonformatter for checking one of the json, not using –jsonArray option
  • from terminal, for a whole collection or a single document, use
    • mongoimport --db m4opsdb --collection M4OPSData --drop --file C:\Users\Peter2\Documents\Mapping\Software\M4OPS2\M4OPS.json
    • mongoimport --db m4opsdb --collection Places --drop --file C:\Users\Peter2\Documents\Mapping\Software\M4OPS2\Places.json
    • mongoimport --db m4opsdb --collection Continents --drop --file C:\Users\Peter2\Documents\Mapping\Software\M4OPS2\Continents.json
    • mongoimport --db m4opsdb --collection Forms --drop --file C:\projects\m4ops\client\src\modules\forms\vfgData\Forms.json
    • (Note that for now we do not import Users - we set them up manually via /regiter)
  • For Feature Layers
    • can use: ...m4ops\utils\importAllFeatureLayers.bat
    • Remember the url of the source in Places (eg Testing VFG.geojson) when prefixed by OPS_ (eg HcN_) and all spaces replaced by underscores (_) must equal the _id in the geojson file of the Feature Layer
    • mongoimport --db m4opsdb --collection FeatureLayers --mode upsert --file "C:\Users\Peter2\Documents\Mapping\Software\M4OPS\OPS\ENG England\HcN Holywell-cum-Needingworth\FromDev\ForMongo\Pubs.geojson"
    • and then:
      • Buildings.geojson"
      • Censuses.geojson"
      • HcN land ownership.geojson"
      • OSM20180209.geojson"
      • test03.geojson"
    • (adding --drop ensures the target instance drops the collection before importing the data from the input)
    • (adding --mode upsert to replace documents whose _id matches the document(s) in the import file)
  • (Note that the Studies collection is not used now)
  • Note that layerAndSourceVector.js prepends the OPS_ (eg HcN_) to the layer id before using it in the URL to get it from the database via the server
    • and the FromDev\ForMongo versions of the FeatureLayers have the extra _id field to correspond eg "_id":"HcN_Pubs"
  • maximum BSON (Mongodb) document size is 16 megabytes, or use GridFS API.

To create Places.json

  • using M4OPS-manage and with OPS.csv as chosen file compile all the studies
  • for each study upload the OPS.json file into the study's local folder
  • use concatOPS.bat to create output.txt concatenated from all of the individual OPS.json files
  • rename output.txt as Places.json

Mongoose

MongoDB Backups

Environment Variables

Envars General

Envars on Server

  • the environment variables we need are defined in .env.example ( as used by dotenv-safe)
    • PORT (only needed in development)
    • MONGO_DB_URL see doc
  • they are read before, or very early in, app.js starts
  • console.log(process.env) shows them

Envars on Server: PC Development

  • we use dotenv-safe to read from .env (untracked), and this is called if MONGO_DB_URL is not already set
  • they include (but we do not use) all the windows environment variables such as PATH

Envars on Server: Production

  • We use Claudia/AWS Lambda
  • .aws/credentials is stored in %UserProfile% (ie Peter)
  • environment variables are all in prod.json
  • these are read by claudia via --set-env-from-json

Envars on Client

  • In the client we use Vue CLI 3 standards - see VueCLI 3: Environment Variables and Modes
    • .env # loaded in all cases
    • .env.local # loaded in all cases, ignored by git
    • .env.[mode] # only loaded in specified mode
    • .env.[mode].local # only loaded in specified mode, ignored by git
    • (where [mode] is development, production or test)
    • Only variables that start with VUE_APP_ will be available (via process.env. ), plus BASE_URL & NODE_ENV
    • Computed env vars can be in vue.config.js (still prefixed with VUE_APP_)
  • Note that the older Vue CLI 2 had a config directory with dev.env.js and prod.env.js

Into Production

Principles

  • The front end (Vue.js) is built within VS Code then uploaded to an AWS S3 bucket
  • The (Node/Express) back end is constructed using claudia and uploaded into AWS Lamda
  • The (MongoDB) database behind that is implemented in Mongo Atlas

Where possible we use AWS REGION N. Virginia (us-east-1)

Guidance

Into Production - Client

Client Build

From Terminal: C:\projects\m4ops\client>npm run build

  • Compiled with 2 warnings
    • asset(s) exceed the recommended size limit (244 KiB).
      • js/chunk-vendors.2cee4a6c.js (1.2 MiB)
    • entrypoint(s) combined asset size exceeds the recommended limit (244 KiB)
      • app (1.51 MiB)
      • css/chunk-vendors.253ba11b.css
      • js/chunk-vendors.2cee4a6c.js
      • css/app.655d80b5.css
      • js/app.a9c51c81.js
  File                                      Size             Gzipped

  dist\js\chunk-vendors.2cee4a6c.js         1229.84 kb       347.15 kb
  dist\js\app.a9c51c81.js                   47.83 kb         14.06 kb
  dist\js\about.30fd5a98.js                 1.64 kb          0.68 kb
  dist\precache-manifest.7a2d93dc6fc2147    1.01 kb          0.40 kb
  a8173f6a6a2fd6cfb.js
  dist\service-worker.js                    0.94 kb          0.53 kb
  dist\css\app.655d80b5.css                 241.59 kb        31.64 kb
  dist\css\chunk-vendors.253ba11b.css       31.60 kb         5.53 kb
  dist\css\about.0e433876.css               0.00 kb          0.02 kb

Client Upload to AWS

See Vue CLI 3 deployment instructions - essentially drag and drop contents of dist folder into AWS bucket.

Our website

Into Production - Server

AWS Lambda

  • AWS Lambda needs AWS API Gateway to define how external services (or, in this case, a Vue.js client) can communicate with your serverless backend app - makes AWS Lambda not straightforward.

  • We use an open-source tool called Claudia.js.

  • To see what is happening in Lambda, and console messages, use AWS CloudWatch in the region we used (us-east-1) - prices

  • but see the serverless_computing_study

Claudia

Claudia and Express (now we use node v8.11.3) Use the claudia CLI tool to prepare a serverless proxy around Express API: claudia generate-serverless-express-proxy --express-module src/app

Then using claudia create:

claudia create --handler lambda.handler --deploy-proxy-api --region us-east-1 --set-env-from-json prod.json

us-east-1 is Virginia

To re-create use claudia update --set-env-from-json prod.json

See our lambda check costs !!!

General

  • Useful
  • Popular Express middleware to define backend endpoints
    • bodyParser (have) - Express middleware that parses request bodies so you can access JSON objects sent by clients
    • cors (have) - Express middleware to make your endpoint accept cross-origin requests
    • helmet - Express middleware that helps to secure your apps with various HTTP headers
    • morgan - HTTP request logger middleware for Node.js web apps
    • mongodb (we use mongoose??) - the MongoDB native driver for Node.js;

Useful but not used

Into Production - Database

MongoDB Atlas implementation

  • Documentation and pricing
  • Free tier has maximum 512 MB storage, no automatic backups - see limitations
  • login (manual) via RoboForm
  • mongoimport --host Cluster0-shard-0/cluster0-shard-00-00-bfjgs.mongodb.net:27017,cluster0-shard-00-01-bfjgs.mongodb.net:27017,cluster0-shard-00-02-bfjgs.mongodb.net:27017 --ssl --username m4ops_admin --password opl0oUiDw3w9FAH7 --authenticationDatabase admin --db m4opsdb --collection M4OPSData --drop --file C:\Users\Peter2\Documents\Mapping\Software\M4OPS2\M4OPS.json
    • also need --type csv if not json
    • --authenticationDatabase admin just means the user's details are in the admin db
  • Compass URI Connection String: mongodb+srv://m4ops_admin:opl0oUiDw3w9FAH7@cluster0-bfjgs.mongodb.net/admin
  • See also re Lambda
  • whitelist etc
  • Need VPC Peering and this for production, but
    • (free) M0 clusters don’t support VPC Peering so we need
    • whitelist 0.0.0.0/0 (access from anywhere)
  • older URI mongodb://m4ops_r:iS2NPR3HBmsSTheK@cluster0-shard-00-00-bfjgs.mongodb.net:27017,cluster0-shard-00-01-bfjgs.mongodb.net:27017,cluster0-shard-00-02-bfjgs.mongodb.net:27017/m4opsdb?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true
  • Check this
  • Also

Other Notes

  • The Power of Web Components
  • anatomy-of-a-url
  • Convert/prettify
  • Any locally installed command (eg xx) will be available at ./node_modules/.bin/xx in your project
    • node_modules/.bin directory will be added to system $PATH when your're running npm scripts, so you can directly use the local xx command there
  • Popular Systems in 2017 - especially Vue.js
  • We have a codeSandbox for trying things out
  • unpkg is a fast, global content delivery network for everything on npm - it makes every npm package available in the browser
  • Beware Memory Leaks
  • to install a specific version use eg npm install vuelayers@^0.11.0 --save
  • to reload not from cache use location.reload(true) - the parameter forces it to reload from the server
  • more slots, less props