From 892595a726bf98cf7ae4fde9b9490797f9052f69 Mon Sep 17 00:00:00 2001 From: Neil Goldader Date: Thu, 27 Feb 2020 15:01:31 -0500 Subject: [PATCH 01/18] [docs] Add docs for mds-web-sockets (#191) * Add MDS-Web-Sockets docs * Typo * Remove example response --- packages/mds-web-sockets/README.md | 45 ++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 packages/mds-web-sockets/README.md diff --git a/packages/mds-web-sockets/README.md b/packages/mds-web-sockets/README.md new file mode 100644 index 000000000..61f59afac --- /dev/null +++ b/packages/mds-web-sockets/README.md @@ -0,0 +1,45 @@ +# MDS Web Sockets Interface +Implementation of a basic websocket model for MDS + +## Package Content + +### Stable Content +N/A +### Experimental Content +This package implements a basic websocket client and server to be used with MDS. + +## Current usage +[MDS-Agency](../mds-agency/README.md) currently emits all incoming events and telemetry to the websocket server by implementating [client.ts](client.ts). + +## Protocol + +### Generic message structure +`CONTROL%ARG1%ARG2%...%ARGn` + +### Message Headers +* `AUTH`: Authentication Header. To be followed with a `SCHEME` and `TOKEN` in the format `AUTH%SCHEME TOKEN` +* `PING`: Ping message that expects a `PONG` response. Of the format `PING`. +* `PONG`: Pong message that responds to a `PING`. Of the format `PONG`. +* `SUB`: Subscription Header. To be followed with the entities to which you'd like to subscribe, in the format `SUB%ENTITY1%ENTITY2%...%ENTITYn` +* `PUSH`: Test message that will trigger an `EVENT` payload to be sent out to all subscribers of the `EVENTS` entity. + +### PING/PONG +PINGs/PONGs are managed internally by the `ws-heartbeat` library. PINGs will emit from the server, and the client should respond with PONGs. + +### Authentication +Currently, this package only supports the `Bearer` scheme for tokens. See [Message Headers](#message-headers) + +### Subscription Model +MDS-Web-Sockets uses a basic pub/sub model. For a client to subscribe to server-side entity updates, they must send a `SUB` message (see [Message Headers](#message-headers)). + +#### Subscribable Entities +| Entity | Response Type | Response Format | +|---------------|---------------|------------------------------| +| `EVENTS` | String | `EVENTS%VEHICLE_EVENT_JSON` | +| `TELEMETRIES` | String | `TELEMETRIES%TELEMETRY_JSON` | + +## Quick Start and Tests +See [Quick Start](../../README.md#Installation) + +## Build Deployment Package +See [Build](../../README.md#Build) \ No newline at end of file From a383814799abfaa824c46c71b58b21f7d5a805e4 Mon Sep 17 00:00:00 2001 From: Mark Maxham Date: Mon, 2 Mar 2020 16:13:02 -0800 Subject: [PATCH 02/18] add ARCHITECTURE.md twiddle README add NOT FOR CONTRIB to service-areas --- ARCHITECTURE.md | 97 ++++++++++++++++++++++++++ README.md | 37 +++++----- packages/ladot-service-areas/README.md | 1 + 3 files changed, 119 insertions(+), 16 deletions(-) create mode 100644 ARCHITECTURE.md create mode 100644 packages/ladot-service-areas/README.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 000000000..649812476 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,97 @@ +# OMF mds-core Architecture + +## Draft +# Background +The mds-core repo represents an implementation of the MDS Agency and Policy APIs, as well as some not-yet-ratified API implementations (Compliance, Audit, Geography, Jurisdictions), plus shared libraries, containerization, authentication, etc. + +The goal of this document is to give an overview of the technology choices, design, and implementation of these services. The mds-core repo will be a reference system that can be used as a basis for building and testing additional services, as well as a reasonable facsimile of what is running for the first city to deploy these services (Los Angeles). +# Languages +This implementation is built in [TypeScript](https://www.typescriptlang.org) v3.8. The first draft of the Agency service was built in vanilla EMCAScript 6, and types were added as the system grew. + +TypeScript has a large developer community, a comprehensive set of toolchains, wide support in cloud providers, and is reasonably performant. JavaScript is the [most popular](https://octoverse.github.com/) language inside GitHub, with TypeScript 7th. + +The mds-core endpoints are implemented using the popular framework [express.js](https://expressjs.com/). + +# Authentication & Identity +We authenticate and authorize access to APIs and public-facing websites using JSON Web Tokens (JWT). JWTs are issued by Auth0 using the OAuth 2.0 protocol. +## Auth0 +We use Auth0 for our Identity Provider (IdP). Our users, applications, and permissions are configured in Auth0. Our Frontend applications forward users to Auth0 for login. Upon returning from login, included token information is exchanged for an access token (JWT) and stored in the browser’s local storage. This access token is sent along with every API request. Our APIs validate the signature of access tokens cryptographically with keys provided by Auth0. +## The 2 façades of authentication +There are two use cases for requesting and using access tokens when interacting with our system. From our API’s perspective there isn’t much difference. However there are 2 types of clients for our API, and they authenticate differently. + +### SPA - Front-end applications, web, and mobile + + +Our front-end applications authenticate users by forwarding them to Auth0’s login page when token information is not found or is expired. Users login either by username/password credentials or using social connections. We use the authorization_code grant type which involves receiving a “code” query string parameter upon successful login. This code is then exchanged for an access token using Auth0’s API and the resulting access token gets stored in local storage and included as an “Authorization” header bearer token to every API request thereafter. We are using a PKCE compliant login flow, which let’s us avoid specifying the private key during this exchange which makes it a safe process for native mobile as well. + + +### M2M - Providers accessing APIs + + +The vehicle providers pushing data to the Agency API are similarly required to provide authorization headers with their requests. To obtain access tokens, providers issue a token request to Auth0’s API using the client_credentials grant type. This type of authentication requires a secret key, so all of the providers are partitioned by application which means each provider has their own unique Client Id and Secret. There are also additional claims in the access token that help identify providers by id. +## Istio Gateway +Token validation, signature, and expiration occur in the Istio Gateway. This becomes particularly useful for local development, as issuing requests to locally running APIs, in docker or otherwise, do still require access tokens for authorization, but they can be entirely fabricated. +## Identity +Everything to this point has indicated direct interaction between clients and Auth0, when in fact there is actually an IDP proxy in place between clients and Auth0 we are calling Lacuna-Identity or Identity Service. Lacuna-Identity is a very simple API shim that does a few things. It acts as the middleware for login/logout ingress/egress between front-end applications and Auth0. It also proxies the token exchange request. Currently it is only minimally involved, enough that if we were to decide to switch IDPs, our client applications would likely be unaffected. Since we have access points at both the token generation and validation, there is a high degree of potential to eventually switch from returning and validating Auth0 access tokens to our own signed access tokens. +## Role based access control (RBAC) +We restrict access and alter behavior in both front and back-end applications by the presence or absence of scopes in the access token. API endpoints are configured to block requests via middleware if indicated scopes are not present. Likewise, front-end applications will show or hide certain pages or functionality with the presence of indicated scopes. For example, you are required to have the “policies:read” scope to see and use Policy Manager. + +There are a couple different ways scopes are included in access tokens, and they differ between the SPA and M2M authentication scenarios. Both cases involve configuration with Auth0’s RBAC. Scopes are represented in Auth0 as permissions 1:1, and the scopes included in the access token are the intersection between assigned permissions and requested scopes. + +### SPA + +Since this authentication involves users, the permissions in consideration are the permissions directly assigned to the user or the permissions in any assigned roles. + +### M2M + +Since this authentication involves applications, the permissions in consideration are the permissions directly assigned to the application. +# Containerization +The mds-core services are packaged as Docker container images, utilizing [webpack](https://webpack.js.org/) for minification. The current base image is [node:12.14.1-alpine](https://github.com/mhart/alpine-node). +# Orchestration +[Kubernetes](https://kubernetes.io/) provides a declarative, cloud-neutral framework for specifying the infrastructure in which an application (or set of applications) runs. The extensible API allows platform-as-a-service (Paas) components such as NATS, Prometheus, and the core kubernetes operators to be included as a managed set of configuration files. + +Kubernetes is a datacenter-scale operating system. The configuration files for kubernetes allow for source-controlled management of datacenter-level functionality. This includes path-based routing of http requests to the various back-end applications; authentication of the same requests to a 3rd-party identity management service; monitoring of performance and health metrics, alerting on failure conditions; and demand-based scaling of compute resources. +# Deployment +We chose [kubernetes](https://kubernetes.io/) as a widely-supported, vendor-agnostic method of deploying containers into a cloud. A custom tool, mdsctl, provides a wrapper around [kubectl](https://github.com/kubernetes/kubectl) and (helm)[https://helm.sh/] enabling portable build, deployment, development and verification operations. +# Database & ORM +Persistent storage is contained in Postgres. The current interface for each service is performed via a hand-made ORM in the mds-db library, but we are in the process of replacing it with a more robust and FOSS library, TypeORM. See the documentation in the mds-db package for more details, although this will be phased out in the next release. + +DB credentials are injected to services via environment variables during the deployment cycle. +# Cache +For frequently-accessed data, such as the current state of all active vehicles in the system, the mds-cache library uses Redis under the hood. + +The interface to the cache is described in the package mds-cache. +# Messaging +Although the current services in MDS-core do not communicate with each other, LADOT is using some data-consumers that process Agency-ingested events and telemetry via a publish-and-subscribe message bus. + +Services in a service-oriented-architecture can communicate with each other via the same types of interfaces used by the outside clients (e.g. REST or SOAP), or they can use an internal message bus. We chose the latter for perceived wins in speed and efficiency. After considering knative eventing and kafka, we settled on [NATS](https://nats.io) as a more stable and low-operational-overhead solution. We have attempted to abstract away the message interface, however, such that we could later decide to plug in a different message bus. + +NATS is recognized as an incubating [CNCF](https://www.cncf.io) [open source project](https://github.com/nats-io) made available under the [Apache License](https://www.apache.org/licenses/LICENSE-2.0.html) and supporting a number of contemporary software languages. +# Testing & Validation +The repo currently has integration tests with code-coverage minimums on a per-project basis. [TODO add more] +# Versioning +The services in mds-core will follow the MDS spec’s requirements for [API versioning](https://github.com/openmobilityfoundation/mobility-data-specification/blob/dev/general-information.md). +# Shared Libraries +The mds-core repo contains a number of libraries, including the previously mentioned mds-db (postgres wrapper) and mds-cache (redis wrapper). +## mds-api-authorizer +Code to extract standard and custom claims from Basic or Bearer Authorization headers +## mds-api-helpers +Simple utilities for construction APIs. For example, processing paging parameters or formating JSON API links +## mds-api-server +Standard API server which implements middleware and endpoints that are common to all MDS APIs. +## mds-logger +Basic logging wrapper that includes integration with Slack and Pushover as well as standard console-output. + +## mds-orm +Code for managing connections to Postgres and schema migrations. +## mds-providers +Collection of information about supported micro-mobility providers. +## mds-stream +Collection of methods to send messages to NATS-Streaming. Currently supports sending vehicle events and telemetry. +## mds-types +Collection of TypeScript descriptions of types that are shared among components, including Agency’s notion of Device and VehicleEvent, event type enums for Provider and Agency, etc. +## mds-utils +Grab-bag of shared utility methods. Including, but not limited to: Programmatic representation of the MDS-Agency State Machine, date-time converters, and geospatial analysis methods. +## mds-web-sockets +Implementation of a basic websocket model for MDS. Enables real-time external consumption of data through a pub/sub model. See package documentation for specific examples. + diff --git a/README.md b/README.md index 95dc19aab..750b05fee 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ # Overview -Repo for LADOT MDS implementation for contribution to the Open Mobility Foundation. It represents what is currently up and running for Los Angeles production MDS as well as new features under development. +Repo for a reference implementation of MDS services intended for hosting by cities. Includes implementations of +Agency and Policy services, as well as Compliance and Audit services, plus some other tools and shared libraries. ## Contents @@ -14,16 +15,25 @@ Repo for LADOT MDS implementation for contribution to the Open Mobility Foundati 1. MDS-Agency `/stops` [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) 2. MDS-Audit [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) 3. MDS-Compliance [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) -4. MDS-Config -5. MDS-Daily -6. MDS-Metrics -7. MDS-Metrics-Sheet -8. MDS-Policy-Author -9. MDS-Web-Sockets -#### Backend Services -1. MDS-Event-Processor -2. MDS-Provider-Processor -3. MDS-Trip-Processor +4. MDS-Policy-Author TODO docs +5. MDS-Daily TODO docs +6. MDS-Web-Sockets TODO docs + +#### TO BE REMOVED BEFORE MERGE +1. MDS-Event-Processor (REMOVE) +2. MDS-Provider-Processor (REMOVE) +3. MDS-Trip-Processor (REMOVE) +4. MDS-Config (REMOVE) +5. MDS-Metrics (REMOVE) +6. MDS-Metrics-Sheet (REMOVE) + +## Contributing + +See [CONTRIBUTING.md](.github/CONTRIBUTING.md) + +## Security + +To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) ## Installation @@ -299,10 +309,5 @@ Display the complete set of operations: % ./bin/mdsctl uninstall:mds uninstall ``` -## Other -To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) -## Contributing - -See [CONTRIBUTING.md](.github/CONTRIBUTING.md) diff --git a/packages/ladot-service-areas/README.md b/packages/ladot-service-areas/README.md new file mode 100644 index 000000000..abada369b --- /dev/null +++ b/packages/ladot-service-areas/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION From fafca98271bf7a896987d01b789c509971cd8698 Mon Sep 17 00:00:00 2001 From: sean roberts Date: Tue, 3 Mar 2020 10:57:34 -0800 Subject: [PATCH 03/18] change readme to OMF project landing page --- DevWorkstationSetup..md | 313 ++++++++++++++++++++++++++++++++++ README.md | 366 ++++++---------------------------------- 2 files changed, 366 insertions(+), 313 deletions(-) create mode 100644 DevWorkstationSetup..md diff --git a/DevWorkstationSetup..md b/DevWorkstationSetup..md new file mode 100644 index 000000000..750b05fee --- /dev/null +++ b/DevWorkstationSetup..md @@ -0,0 +1,313 @@ +# Overview + +Repo for a reference implementation of MDS services intended for hosting by cities. Includes implementations of +Agency and Policy services, as well as Compliance and Audit services, plus some other tools and shared libraries. + +## Contents + +### Stable Content +#### APIs +1. MDS-Agency 0.4.0 Implementation +2. MDS-Policy 0.4.0 Implementation + +### Experimental Content +#### APIs +1. MDS-Agency `/stops` [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) +2. MDS-Audit [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) +3. MDS-Compliance [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) +4. MDS-Policy-Author TODO docs +5. MDS-Daily TODO docs +6. MDS-Web-Sockets TODO docs + +#### TO BE REMOVED BEFORE MERGE +1. MDS-Event-Processor (REMOVE) +2. MDS-Provider-Processor (REMOVE) +3. MDS-Trip-Processor (REMOVE) +4. MDS-Config (REMOVE) +5. MDS-Metrics (REMOVE) +6. MDS-Metrics-Sheet (REMOVE) + +## Contributing + +See [CONTRIBUTING.md](.github/CONTRIBUTING.md) + +## Security + +To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) + +## Installation + +### Dependencies + +* PostgreSQL +* Redis +* [Yarn](https://yarnpkg.com/en/docs/install#mac-stable) +* [NVM](https://github.com/nvm-sh/nvm#installation-and-update) + +#### Database config on macOS +If you haven't installed PostegreSQL and Redis you can install them with homebrew on macOS +``` +brew install postgresql +brew install redis +``` + +Make sure they are running before you run the tests +``` +brew services start postgresql +brew services start redis +``` + +If you encounter the following error: +`FATAL: database “” does not exist` + +The following command should fix your issue +`createdb -h localhost` + +To run tests, you will need this: +`createdb -h localhost mdstest` + +Then add `export PG_NAME=mdstest` to your shell's environment file. (The name is not important, but you'll need to point it somehwere.) + +#### Node setup + +You should have NVM already installed from the link above. The top level directory of the project has a `.nvmrc` file and you should be able to run `nvm install` to get the right version of Node. + +#### Package setup +Install [Lerna](https://lerna.js.org/) + +```sh +yarn global add lerna +``` + +Install all packages. Uses Yarn workspaces. + +```sh +yarn install +``` + +#### Launching a local server for a package +Now you can work with each package + +```sh +cd packages/mds-audit +yarn test +yarn start +``` + +#### Running the tests +You can also run all tests from the project root with +``` +yarn test +``` + +### Package Management - Lerna + +This repository is a monorepo and uses Lerna for working with its packages. + +#### Example commands + +Run all test suites at once + +```sh +lerna run test +``` + +Run all tests suites sequentially + +```sh +lerna run test --concurrency 1 +``` + +Run tests for a particular package + +```sh +lerna run test --scope mds-audit +``` + +Clean all dependencies + +```sh +lerna run clean +``` + +Format all files + +```sh +lerna run prettier +``` + +## Debugging with Visual Studio Code + +### Node.js: Express Server + +* Select the `Node.js Express Server` debug configuration +* Select the file that implements the Node/Express server for a package (typically `server.ts`) in the Explorer panel +* Press `F5` + +### Mocha Tests + +* Select the `Node.js: Mocha Tests` debug configuration +* Select any one of the files in a package's test folder +* Press `F5` + +### Kubernetes + +MDS can readily be provisioned to a [Kubernetes](https://kubernetes.io) capable cluster, be it a local or remote. The following steps describe how to build, deploy and operate against a local MDS cluster. + +#### Prerequisites + +Obtain a local working copy of MDS: + +```sh +% git clone https://github.com/lacuna-tech/mds-core +% cd mds-core +``` + +OSX (Linux and Windows tbd) + +Install [Docker Desktop](https://download.docker.com/mac/stable/Docker.dmg): + +```sh +% open https://download.docker.com/mac/stable/Docker.dmg +``` + +Start Docker-Desktop: + +```sh +% open /Applications/Docker.app +``` + +Lastly, configure Kubernetes: + +```txt +select the 'Preferences' option +select the 'Resources' option + apply the following minimal resource changes: + CPUs: 6 + Memory: 8G + Swap: 1G +select the 'Kubernetes' option + select 'Enable Kubernetes' option +select 'Apply & Restart' +``` + +Verify: + +```sh +% which kubectl +% kubectl config set-context docker-desktop +% kubectl cluster-info +``` + +#### Bootstrap : install operational dependencies + +In order to build and operate MDS, a number of suporting technologies are leveraged by ensuring they are installed and operational via a one-time `bootstap` process: + +```sh +% ./bin/mdsctl bootstrap +``` + +The principle tools are: [homebrew](https://brew.sh), [bash-4.x+](https://www.gnu.org/software/bash/), [oq](https://github.com/Blacksmoke16/oq), [jq](https://stedolan.github.io/jq/), [yarn](https://yarnpkg.com/), [nvm](https://github.com/nvm-sh/nvm), [helm-2.14.1](https://helm.sh), [k9s](https://github.com/derailed/k9s), [kubectx](https://github.com/ahmetb/kubectx), [git](https://git-scm.com/), [gcloud](https://cloud.google.com/sdk/) and [awscli](https://aws.amazon.com/cli/). Additionally the following services are provisioned: [istio](https://istio.io) and [nats](https://nats.io). + +Verify: + +```sh +% kubectl -n istio-system get pods +% kubectl -n nats get pods +% k9s & +``` + +#### Build : compile source into deployable images + +Compiling and packaging MDS into a deployable form is achived as follows: + +```sh +% ./bin/mdsctl build +``` + +Verify: + +```sh +% docker images | grep mds* +``` + +#### Run : install MDS + +(tbd: ?best profile?) + +```sh +% ./bin/mdsctl -p minimal install:mds +``` + +Verify: + +```sh +% curl localhost/agency +``` + +#### In-Cluster Development +Due to the nature of MDS-Core being a highly portable Typescript project that compiles down into minified javascript for its images, rapidly development in-cluster can be quite challenging. MDS-Core utilizes [Okteto](https://okteto.com) to enable developers to actively develop their code in-cluster. + +After following the above steps to set up a local MDS cluster, you can override an existing service's deployment with these steps. +1. Update `mds-core/okteto.yml`'s `name` field to be set to the service you wish to replace (e.g. `mds-agency`) +2. +```sh +% curl https://get.okteto.com -sSfL | sh +``` +3. Install the `Remote - Kubernetes` VSCode extension. +4. Run `> Okteto Up` from the VSCode command palette. +* After the remote session opens, execute this in the new shell window: +```sh +% yarn +% cd packages/${SERVICE_NAME} +% yarn start +``` +5. This session is now safe to close, and you can reattach with the `okteto.${SERVICE_NAME}` ssh profile automatically added for you using the VSCode `Remote - SSH` package. +6. When you're completely done with your session, run `> Okteto Down` from the VSCode command palette, or `okteto down` from terminal to revert the changes made by Okteto, and return your service to its previous deployment. + +#### MDS Operations + +MDS operates atop the following services: [Kubernetes](https://kubernetes.io), [Istio](https://istio.io), [NATS](https://nats.io), [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io). + +(tbd) + +#### Additional Considerations + +Access the database: + +```sh +% ./bin/mdsctl cli:postgresql +``` + +Access the cache: + +```sh +% ./bin/mdsctl cli:redis +``` + +(tbd) Access the event stream: + +```sh +% ./bin/mdsctl install:natsbox +``` + +Access the MDS cluster: + +```sh +% k9s +``` + +Display the complete set of operations: + +```sh +% ./bin/mdsctl +``` + +#### Cleanup + +```sh +% ./bin/mdsctl uninstall:mds uninstall +``` + + + diff --git a/README.md b/README.md index 750b05fee..17d8686e4 100644 --- a/README.md +++ b/README.md @@ -1,313 +1,53 @@ -# Overview - -Repo for a reference implementation of MDS services intended for hosting by cities. Includes implementations of -Agency and Policy services, as well as Compliance and Audit services, plus some other tools and shared libraries. - -## Contents - -### Stable Content -#### APIs -1. MDS-Agency 0.4.0 Implementation -2. MDS-Policy 0.4.0 Implementation - -### Experimental Content -#### APIs -1. MDS-Agency `/stops` [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) -2. MDS-Audit [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) -3. MDS-Compliance [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) -4. MDS-Policy-Author TODO docs -5. MDS-Daily TODO docs -6. MDS-Web-Sockets TODO docs - -#### TO BE REMOVED BEFORE MERGE -1. MDS-Event-Processor (REMOVE) -2. MDS-Provider-Processor (REMOVE) -3. MDS-Trip-Processor (REMOVE) -4. MDS-Config (REMOVE) -5. MDS-Metrics (REMOVE) -6. MDS-Metrics-Sheet (REMOVE) - -## Contributing - -See [CONTRIBUTING.md](.github/CONTRIBUTING.md) - -## Security - -To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) - -## Installation - -### Dependencies - -* PostgreSQL -* Redis -* [Yarn](https://yarnpkg.com/en/docs/install#mac-stable) -* [NVM](https://github.com/nvm-sh/nvm#installation-and-update) - -#### Database config on macOS -If you haven't installed PostegreSQL and Redis you can install them with homebrew on macOS -``` -brew install postgresql -brew install redis -``` - -Make sure they are running before you run the tests -``` -brew services start postgresql -brew services start redis -``` - -If you encounter the following error: -`FATAL: database “” does not exist` - -The following command should fix your issue -`createdb -h localhost` - -To run tests, you will need this: -`createdb -h localhost mdstest` - -Then add `export PG_NAME=mdstest` to your shell's environment file. (The name is not important, but you'll need to point it somehwere.) - -#### Node setup - -You should have NVM already installed from the link above. The top level directory of the project has a `.nvmrc` file and you should be able to run `nvm install` to get the right version of Node. - -#### Package setup -Install [Lerna](https://lerna.js.org/) - -```sh -yarn global add lerna -``` - -Install all packages. Uses Yarn workspaces. - -```sh -yarn install -``` - -#### Launching a local server for a package -Now you can work with each package - -```sh -cd packages/mds-audit -yarn test -yarn start -``` - -#### Running the tests -You can also run all tests from the project root with -``` -yarn test -``` - -### Package Management - Lerna - -This repository is a monorepo and uses Lerna for working with its packages. - -#### Example commands - -Run all test suites at once - -```sh -lerna run test -``` - -Run all tests suites sequentially - -```sh -lerna run test --concurrency 1 -``` - -Run tests for a particular package - -```sh -lerna run test --scope mds-audit -``` - -Clean all dependencies - -```sh -lerna run clean -``` - -Format all files - -```sh -lerna run prettier -``` - -## Debugging with Visual Studio Code - -### Node.js: Express Server - -* Select the `Node.js Express Server` debug configuration -* Select the file that implements the Node/Express server for a package (typically `server.ts`) in the Explorer panel -* Press `F5` - -### Mocha Tests - -* Select the `Node.js: Mocha Tests` debug configuration -* Select any one of the files in a package's test folder -* Press `F5` - -### Kubernetes - -MDS can readily be provisioned to a [Kubernetes](https://kubernetes.io) capable cluster, be it a local or remote. The following steps describe how to build, deploy and operate against a local MDS cluster. - -#### Prerequisites - -Obtain a local working copy of MDS: - -```sh -% git clone https://github.com/lacuna-tech/mds-core -% cd mds-core -``` - -OSX (Linux and Windows tbd) - -Install [Docker Desktop](https://download.docker.com/mac/stable/Docker.dmg): - -```sh -% open https://download.docker.com/mac/stable/Docker.dmg -``` - -Start Docker-Desktop: - -```sh -% open /Applications/Docker.app -``` - -Lastly, configure Kubernetes: - -```txt -select the 'Preferences' option -select the 'Resources' option - apply the following minimal resource changes: - CPUs: 6 - Memory: 8G - Swap: 1G -select the 'Kubernetes' option - select 'Enable Kubernetes' option -select 'Apply & Restart' -``` - -Verify: - -```sh -% which kubectl -% kubectl config set-context docker-desktop -% kubectl cluster-info -``` - -#### Bootstrap : install operational dependencies - -In order to build and operate MDS, a number of suporting technologies are leveraged by ensuring they are installed and operational via a one-time `bootstap` process: - -```sh -% ./bin/mdsctl bootstrap -``` - -The principle tools are: [homebrew](https://brew.sh), [bash-4.x+](https://www.gnu.org/software/bash/), [oq](https://github.com/Blacksmoke16/oq), [jq](https://stedolan.github.io/jq/), [yarn](https://yarnpkg.com/), [nvm](https://github.com/nvm-sh/nvm), [helm-2.14.1](https://helm.sh), [k9s](https://github.com/derailed/k9s), [kubectx](https://github.com/ahmetb/kubectx), [git](https://git-scm.com/), [gcloud](https://cloud.google.com/sdk/) and [awscli](https://aws.amazon.com/cli/). Additionally the following services are provisioned: [istio](https://istio.io) and [nats](https://nats.io). - -Verify: - -```sh -% kubectl -n istio-system get pods -% kubectl -n nats get pods -% k9s & -``` - -#### Build : compile source into deployable images - -Compiling and packaging MDS into a deployable form is achived as follows: - -```sh -% ./bin/mdsctl build -``` - -Verify: - -```sh -% docker images | grep mds* -``` - -#### Run : install MDS - -(tbd: ?best profile?) - -```sh -% ./bin/mdsctl -p minimal install:mds -``` - -Verify: - -```sh -% curl localhost/agency -``` - -#### In-Cluster Development -Due to the nature of MDS-Core being a highly portable Typescript project that compiles down into minified javascript for its images, rapidly development in-cluster can be quite challenging. MDS-Core utilizes [Okteto](https://okteto.com) to enable developers to actively develop their code in-cluster. - -After following the above steps to set up a local MDS cluster, you can override an existing service's deployment with these steps. -1. Update `mds-core/okteto.yml`'s `name` field to be set to the service you wish to replace (e.g. `mds-agency`) -2. -```sh -% curl https://get.okteto.com -sSfL | sh -``` -3. Install the `Remote - Kubernetes` VSCode extension. -4. Run `> Okteto Up` from the VSCode command palette. -* After the remote session opens, execute this in the new shell window: -```sh -% yarn -% cd packages/${SERVICE_NAME} -% yarn start -``` -5. This session is now safe to close, and you can reattach with the `okteto.${SERVICE_NAME}` ssh profile automatically added for you using the VSCode `Remote - SSH` package. -6. When you're completely done with your session, run `> Okteto Down` from the VSCode command palette, or `okteto down` from terminal to revert the changes made by Okteto, and return your service to its previous deployment. - -#### MDS Operations - -MDS operates atop the following services: [Kubernetes](https://kubernetes.io), [Istio](https://istio.io), [NATS](https://nats.io), [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io). - -(tbd) - -#### Additional Considerations - -Access the database: - -```sh -% ./bin/mdsctl cli:postgresql -``` - -Access the cache: - -```sh -% ./bin/mdsctl cli:redis -``` - -(tbd) Access the event stream: - -```sh -% ./bin/mdsctl install:natsbox -``` - -Access the MDS cluster: - -```sh -% k9s -``` - -Display the complete set of operations: - -```sh -% ./bin/mdsctl -``` - -#### Cleanup - -```sh -% ./bin/mdsctl uninstall:mds uninstall -``` - - - +# Mobility Data Specification +The Mobility Data Specification (**MDS**), a project of the [Open Mobility Foundation](http://www.openmobilityfoundation.org) (OMF), is a set of Application Programming Interfaces (APIs) focused on dockless e-scooters, bicycles, mopeds and carshare. Inspired by projects like [GTFS](https://developers.google.com/transit/gtfs/reference/) and [GBFS](https://github.com/NABSA/gbfs), the goals of MDS are to provide a standardized way for municipalities or other regulatory agencies to ingest, compare and analyze data from mobility service providers, and to give municipalities the ability to express regulation in machine-readable formats. + +**MDS** helps cities interact with companies who operate dockless scooters, bicycles, mopeds and carshare in the public right-of-way. MDS is a key piece of digital infrastructure that supports the effective implementation of mobility policies in cities around the world. + +**MDS** is an open-source project. It was originally created by the [Los Angeles Department of Transportation](http://ladot.io) (LADOT). In November 2019, stewardship of MDS and the ownership of this repository was transferred to the Open Mobility Foundation. GitHub automatically redirects any links to this repository in the `CityOfLosAngeles` organization to the `openmobilityfoundation` instead. MDS continues to be used by LADOT and many other municipalities. + +# mds-core Project Overview +mds-core: An open source reference implementation of the Mobility Data Specification released by the OMF. +* Agency service to ingest data from providers +* Policy service to express vehicle rules and geofences +* Audit service to support field verification of data (*) +* Compliance service to measure adherence to policies (*) +* Daily report generator +* Shared libraries used by multiple services +* Deployment scripts for Kubernetes and Istio +(*) - BASED ON A CANDIDATE API SPEC THAT IS NOT YET AN APPROVED PART OF MDS + +# mds-core is… +* a reference MDS implementation usable by cities +* on-ramp for developers joining MDS ecosystem +* a tool for validating software implementations and data + +# mds-core is not… +* the only implementation of MDS +* where the specification is officially defined (that’s here) +* a place to define local policies or performance metrics +* a service that will be operated by the OMF + +# MDS-core development guidelines +All stakeholders have equal input in WG / WGSC +Decisions made by consensus (except in rare cases where not achievable) +Code changes are proposed and discussed in granular fashion - avoid bulk PRs (exception for new module development) +New modules / major functionality from a single contributor should be discussed and approved by WG / WGSC well in advance +OK for mds-core to slightly lag release of new versions of API specifications +API specifications are the “source of truth” for API definitions, not the mds-core implementation +Any mds-core functionality built against unreleased/draft API specifications will be clearly tagged as “EXPERIMENTAL” or “CANDIDATE” + +# Learn More / Get Involved / Contributing +To stay up to date on MDS releases, meetings, and events, please **subscribe to the [mds-announce](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-announce) mailing list.** + +MDS-core is an open source project with all development taking place on GitHub. Comments and ideas can be shared by [creating an issue](https://github.com/openmobilityfoundation/mds-core/issues), and specific changes can be suggested by [opening a pull request](https://github.com/openmobilityfoundation/mds-core/pulls). Before contributing, please review our [CONTRIBUTING page](CONTRIBUTING.md) to understand guidelines and policies for participation and our [CODE OF CONDUCT page](CODE_OF_CONDUCT.md). + +You can also get involved in development by joining an OMF working group. The working groups maintain the OMF GitHub repositories and work through issues and pull requests. Each working group has its own mailing list for non-technical discussion and planning: + +Working Group | Mailing List | Description +--- | --- | --- +Provider Services | [mds-provider-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-provider-services) | Manages the [`provider`][provider] API within MDS. +City Services | [mds-city-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-city-services) | Manages the [`agency`][agency] and [`policy`][policy] APIs within MDS, as well as the [`mds-core`](https://github.com/openmobilityfoundation/mds-core) reference implementation. + +You can view info about past releases and planning calls in the [wiki](https://github.com/openmobilityfoundation/mobility-data-specification/wiki). + +For questions about MDS please contact [info@openmobilityfoundation.org](mailto:info@openmobilityfoundation.org). Media inquiries to [media@openmobilityfoundation.org](mailto:media@openmobilityfoundation.org) From 984e9c8685d1de7917f78a2319f588f2621cebe8 Mon Sep 17 00:00:00 2001 From: sarob Date: Fri, 6 Mar 2020 14:59:56 -0800 Subject: [PATCH 04/18] readme updates replace paragraphs 2,3 as too spec related with a link to spec readme instead move mds-core development guidelines to CONTRIBUTING.md add a missing link change the order of the table items make a command decision on using mds-core instead of MDS-core --- README.md | 58 +++++++++++++++++++++++++------------------------------ 1 file changed, 26 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 17d8686e4..2b28ce2e5 100644 --- a/README.md +++ b/README.md @@ -1,52 +1,46 @@ # Mobility Data Specification -The Mobility Data Specification (**MDS**), a project of the [Open Mobility Foundation](http://www.openmobilityfoundation.org) (OMF), is a set of Application Programming Interfaces (APIs) focused on dockless e-scooters, bicycles, mopeds and carshare. Inspired by projects like [GTFS](https://developers.google.com/transit/gtfs/reference/) and [GBFS](https://github.com/NABSA/gbfs), the goals of MDS are to provide a standardized way for municipalities or other regulatory agencies to ingest, compare and analyze data from mobility service providers, and to give municipalities the ability to express regulation in machine-readable formats. -**MDS** helps cities interact with companies who operate dockless scooters, bicycles, mopeds and carshare in the public right-of-way. MDS is a key piece of digital infrastructure that supports the effective implementation of mobility policies in cities around the world. +The Mobility Data Specification (**MDS**), a project of the [Open Mobility Foundation](http://www.openmobilityfoundation.org) (OMF), is a set of Application Programming Interfaces (APIs) focused on dockless e-scooters, bicycles, mopeds and carshare. Inspired by projects like [GTFS](https://developers.google.com/transit/gtfs/reference/) and [GBFS](https://github.com/NABSA/gbfs), the goals of MDS are to provide a standardized way for municipalities or other regulatory agencies to ingest, compare and analyze data from mobility service providers, and to give municipalities the ability to express regulation in machine-readable formats. -**MDS** is an open-source project. It was originally created by the [Los Angeles Department of Transportation](http://ladot.io) (LADOT). In November 2019, stewardship of MDS and the ownership of this repository was transferred to the Open Mobility Foundation. GitHub automatically redirects any links to this repository in the `CityOfLosAngeles` organization to the `openmobilityfoundation` instead. MDS continues to be used by LADOT and many other municipalities. +[Read more about the Mobility Data Specification](https://github.com/openmobilityfoundation/mds-core/blob/master/README.md) # mds-core Project Overview + mds-core: An open source reference implementation of the Mobility Data Specification released by the OMF. -* Agency service to ingest data from providers -* Policy service to express vehicle rules and geofences -* Audit service to support field verification of data (*) -* Compliance service to measure adherence to policies (*) -* Daily report generator -* Shared libraries used by multiple services -* Deployment scripts for Kubernetes and Istio -(*) - BASED ON A CANDIDATE API SPEC THAT IS NOT YET AN APPROVED PART OF MDS + +- Agency service to ingest data from providers +- Policy service to express vehicle rules and geofences +- Audit service to support field verification of data (\*) +- Compliance service to measure adherence to policies (\*) +- Daily report generator +- Shared libraries used by multiple services +- Deployment scripts for Kubernetes and Istio # mds-core is… -* a reference MDS implementation usable by cities -* on-ramp for developers joining MDS ecosystem -* a tool for validating software implementations and data + +- a reference MDS implementation usable by cities +- on-ramp for developers joining MDS ecosystem +- a tool for validating software implementations and data # mds-core is not… -* the only implementation of MDS -* where the specification is officially defined (that’s here) -* a place to define local policies or performance metrics -* a service that will be operated by the OMF - -# MDS-core development guidelines -All stakeholders have equal input in WG / WGSC -Decisions made by consensus (except in rare cases where not achievable) -Code changes are proposed and discussed in granular fashion - avoid bulk PRs (exception for new module development) -New modules / major functionality from a single contributor should be discussed and approved by WG / WGSC well in advance -OK for mds-core to slightly lag release of new versions of API specifications -API specifications are the “source of truth” for API definitions, not the mds-core implementation -Any mds-core functionality built against unreleased/draft API specifications will be clearly tagged as “EXPERIMENTAL” or “CANDIDATE” + +- the only implementation of MDS +- where the specification is officially defined (that’s [here](https://github.com/openmobilityfoundation/mobility-data-specification)) +- a place to define local policies or performance metrics +- a service that will be operated by the OMF # Learn More / Get Involved / Contributing + To stay up to date on MDS releases, meetings, and events, please **subscribe to the [mds-announce](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-announce) mailing list.** -MDS-core is an open source project with all development taking place on GitHub. Comments and ideas can be shared by [creating an issue](https://github.com/openmobilityfoundation/mds-core/issues), and specific changes can be suggested by [opening a pull request](https://github.com/openmobilityfoundation/mds-core/pulls). Before contributing, please review our [CONTRIBUTING page](CONTRIBUTING.md) to understand guidelines and policies for participation and our [CODE OF CONDUCT page](CODE_OF_CONDUCT.md). +mds-core is an open source project with all development taking place on GitHub. Comments and ideas can be shared by [creating an issue](https://github.com/openmobilityfoundation/mds-core/issues), and specific changes can be suggested by [opening a pull request](https://github.com/openmobilityfoundation/mds-core/pulls). Before contributing, please review our [CONTRIBUTING page](CONTRIBUTING.md) to understand guidelines and policies for participation and our [CODE OF CONDUCT page](CODE_OF_CONDUCT.md). You can also get involved in development by joining an OMF working group. The working groups maintain the OMF GitHub repositories and work through issues and pull requests. Each working group has its own mailing list for non-technical discussion and planning: -Working Group | Mailing List | Description ---- | --- | --- -Provider Services | [mds-provider-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-provider-services) | Manages the [`provider`][provider] API within MDS. -City Services | [mds-city-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-city-services) | Manages the [`agency`][agency] and [`policy`][policy] APIs within MDS, as well as the [`mds-core`](https://github.com/openmobilityfoundation/mds-core) reference implementation. +| Working Group | Mailing List | Description | +| ----------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| City Services | [mds-city-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-city-services) | Manages the [`mds-core`](https://github.com/openmobilityfoundation/mds-core) reference implementation, as well as the [`agency`][agency] and [`policy`][policy] APIs within MDS. | +| Provider Services | [mds-provider-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-provider-services) | Manages the [`provider`][provider] API within MDS. | You can view info about past releases and planning calls in the [wiki](https://github.com/openmobilityfoundation/mobility-data-specification/wiki). From 4d651f0d503a7e7fff364a7193960dff7888d265 Mon Sep 17 00:00:00 2001 From: sarob Date: Fri, 6 Mar 2020 15:21:57 -0800 Subject: [PATCH 05/18] add missing ref links to readme also eliminate garbage from bad markdown linter --- README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 2b28ce2e5..e6019070b 100644 --- a/README.md +++ b/README.md @@ -37,11 +37,15 @@ mds-core is an open source project with all development taking place on GitHub. You can also get involved in development by joining an OMF working group. The working groups maintain the OMF GitHub repositories and work through issues and pull requests. Each working group has its own mailing list for non-technical discussion and planning: -| Working Group | Mailing List | Description | -| ----------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| City Services | [mds-city-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-city-services) | Manages the [`mds-core`](https://github.com/openmobilityfoundation/mds-core) reference implementation, as well as the [`agency`][agency] and [`policy`][policy] APIs within MDS. | -| Provider Services | [mds-provider-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-provider-services) | Manages the [`provider`][provider] API within MDS. | +| Working Group | Mailing List | Description +| ------------- | ------------ | -------- +| City Services | [mds-city-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-city-services) | Manages the [`mds-core`](https://github.com/openmobilityfoundation/mds-core) reference implementation, as well as the [`agency`][agency] and [`policy`][policy] APIs within MDS. +| Provider Services | [mds-provider-services](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-provider-services) | Manages the [`provider`][provider] API within MDS. You can view info about past releases and planning calls in the [wiki](https://github.com/openmobilityfoundation/mobility-data-specification/wiki). For questions about MDS please contact [info@openmobilityfoundation.org](mailto:info@openmobilityfoundation.org). Media inquiries to [media@openmobilityfoundation.org](mailto:media@openmobilityfoundation.org) + +[agency]: https://github.com/openmobilityfoundation/mobility-data-specification/tree/master/agency/README.md +[provider]: https://github.com/openmobilityfoundation/mobility-data-specification/tree/master/provider/README.md +[policy]: https://github.com/openmobilityfoundation/mobility-data-specification/tree/master/policy/README.md From e25cdc748938348786377c013e951f2f7b8c694f Mon Sep 17 00:00:00 2001 From: Mark Maxham Date: Mon, 9 Mar 2020 11:05:51 -0700 Subject: [PATCH 06/18] make sure we have at least rudimentary README files in each package. in some cases, mark packages not intended for contrib. --- packages/mds-api-authorizer/README.md | 1 + packages/mds-api-helpers/README.md | 2 ++ packages/mds-api-server/README.md | 2 ++ packages/mds-cache/README.md | 1 + packages/mds-config-service/README.md | 1 + packages/mds-config/README.md | 1 + packages/mds-event-processor/README.md | 2 ++ packages/mds-event-server/README.md | 1 + packages/mds-logger/README.md | 2 ++ packages/mds-metrics/README.md | 1 + packages/mds-native/README.md | 2 ++ packages/mds-provider-processor/README.md | 1 + packages/mds-provider/README.md | 1 + packages/mds-providers/README.md | 1 + packages/mds-providers/index.ts | 2 +- packages/mds-schema-validators/README.md | 3 +++ packages/mds-stream/README.md | 2 ++ packages/mds-test-data/README.md | 3 +++ packages/mds-types/README.md | 2 ++ packages/mds-utils/README.md | 6 ++++++ 20 files changed, 36 insertions(+), 1 deletion(-) create mode 100644 packages/mds-api-authorizer/README.md create mode 100644 packages/mds-api-helpers/README.md create mode 100644 packages/mds-api-server/README.md create mode 100644 packages/mds-cache/README.md create mode 100644 packages/mds-config-service/README.md create mode 100644 packages/mds-config/README.md create mode 100644 packages/mds-event-server/README.md create mode 100644 packages/mds-logger/README.md create mode 100644 packages/mds-metrics/README.md create mode 100644 packages/mds-native/README.md create mode 100644 packages/mds-provider-processor/README.md create mode 100644 packages/mds-provider/README.md create mode 100644 packages/mds-providers/README.md create mode 100644 packages/mds-schema-validators/README.md create mode 100644 packages/mds-stream/README.md create mode 100644 packages/mds-test-data/README.md create mode 100644 packages/mds-types/README.md create mode 100644 packages/mds-utils/README.md diff --git a/packages/mds-api-authorizer/README.md b/packages/mds-api-authorizer/README.md new file mode 100644 index 000000000..2cd74ada0 --- /dev/null +++ b/packages/mds-api-authorizer/README.md @@ -0,0 +1 @@ +Code to extract standard and custom claims from Basic or Bearer Authorization headers diff --git a/packages/mds-api-helpers/README.md b/packages/mds-api-helpers/README.md new file mode 100644 index 000000000..620273589 --- /dev/null +++ b/packages/mds-api-helpers/README.md @@ -0,0 +1,2 @@ +Simple utilities for construction APIs. For example, processing paging parameters or formating JSON API links. + diff --git a/packages/mds-api-server/README.md b/packages/mds-api-server/README.md new file mode 100644 index 000000000..42ddf39e5 --- /dev/null +++ b/packages/mds-api-server/README.md @@ -0,0 +1,2 @@ +Standard API server which implements middleware and endpoints that are common to all MDS APIs. + diff --git a/packages/mds-cache/README.md b/packages/mds-cache/README.md new file mode 100644 index 000000000..2cc38bc1a --- /dev/null +++ b/packages/mds-cache/README.md @@ -0,0 +1 @@ +Library for transient low-latency storage for e.g. UIs. Backed up by database. Currently contains state of all non-deregistered vehicles. diff --git a/packages/mds-config-service/README.md b/packages/mds-config-service/README.md new file mode 100644 index 000000000..abada369b --- /dev/null +++ b/packages/mds-config-service/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION diff --git a/packages/mds-config/README.md b/packages/mds-config/README.md new file mode 100644 index 000000000..abada369b --- /dev/null +++ b/packages/mds-config/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION diff --git a/packages/mds-event-processor/README.md b/packages/mds-event-processor/README.md index c776cbd9a..4ab4dfc31 100644 --- a/packages/mds-event-processor/README.md +++ b/packages/mds-event-processor/README.md @@ -1,3 +1,5 @@ +# NOT FOR CONTRIBUTION + # Event Streaming Processor Included in this repo is the stream processor for vehicle event and telemetry data sent via cloudevents with Knative eventing: diff --git a/packages/mds-event-server/README.md b/packages/mds-event-server/README.md new file mode 100644 index 000000000..1898b015d --- /dev/null +++ b/packages/mds-event-server/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION \ No newline at end of file diff --git a/packages/mds-logger/README.md b/packages/mds-logger/README.md new file mode 100644 index 000000000..6626d9783 --- /dev/null +++ b/packages/mds-logger/README.md @@ -0,0 +1,2 @@ +Basic logging wrapper that includes integration with Slack and Pushover as well as standard console-output. + diff --git a/packages/mds-metrics/README.md b/packages/mds-metrics/README.md new file mode 100644 index 000000000..abada369b --- /dev/null +++ b/packages/mds-metrics/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION diff --git a/packages/mds-native/README.md b/packages/mds-native/README.md new file mode 100644 index 000000000..3eef4d6b8 --- /dev/null +++ b/packages/mds-native/README.md @@ -0,0 +1,2 @@ +# NOT FOR CONTRIBUTION + diff --git a/packages/mds-provider-processor/README.md b/packages/mds-provider-processor/README.md new file mode 100644 index 000000000..abada369b --- /dev/null +++ b/packages/mds-provider-processor/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION diff --git a/packages/mds-provider/README.md b/packages/mds-provider/README.md new file mode 100644 index 000000000..abada369b --- /dev/null +++ b/packages/mds-provider/README.md @@ -0,0 +1 @@ +# NOT FOR CONTRIBUTION diff --git a/packages/mds-providers/README.md b/packages/mds-providers/README.md new file mode 100644 index 000000000..e236c69c7 --- /dev/null +++ b/packages/mds-providers/README.md @@ -0,0 +1 @@ +Code form of the [list of mobility provider info](https://github.com/openmobilityfoundation/mobility-data-specification/blob/dev/providers.csv). Must be kept updated with changes to that document to stay useful. Also has some test `provider_id` values. \ No newline at end of file diff --git a/packages/mds-providers/index.ts b/packages/mds-providers/index.ts index 2a63c070f..2a51e2eb0 100644 --- a/packages/mds-providers/index.ts +++ b/packages/mds-providers/index.ts @@ -1,7 +1,7 @@ import { Provider } from '@mds-core/mds-types' // Officially recognized providers, from: -// https://github.com/CityOfLosAngeles/mobility-data-specification/blob/dev/providers.csv +// https://github.com/openmobilityfoundation/mobility-data-specification/blob/dev/providers.csv export const JUMP_PROVIDER_ID = 'c20e08cf-8488-46a6-a66c-5d8fb827f7e0' export const LIME_PROVIDER_ID = '63f13c48-34ff-49d2-aca7-cf6a5b6171c3' export const BIRD_PROVIDER_ID = '2411d395-04f2-47c9-ab66-d09e9e3c3251' diff --git a/packages/mds-schema-validators/README.md b/packages/mds-schema-validators/README.md new file mode 100644 index 000000000..01d6bca18 --- /dev/null +++ b/packages/mds-schema-validators/README.md @@ -0,0 +1,3 @@ +# MDS Schema Validators + +A collection of [Joi](https://github.com/hapijs/joi) validators for mds-core API data structures, including VehicleEvent, Telemetry, Policy, etc. Probably should be extracted from mds-core at some point for easier incorporation into stand-alone validators. \ No newline at end of file diff --git a/packages/mds-stream/README.md b/packages/mds-stream/README.md new file mode 100644 index 000000000..124aa01d7 --- /dev/null +++ b/packages/mds-stream/README.md @@ -0,0 +1,2 @@ +Collection of methods to send messages to NATS-Streaming. Currently supports sending vehicle events and telemetry. + diff --git a/packages/mds-test-data/README.md b/packages/mds-test-data/README.md new file mode 100644 index 000000000..0653fad4f --- /dev/null +++ b/packages/mds-test-data/README.md @@ -0,0 +1,3 @@ +# MDS Test Data + +A variety of JSON blobs representing data that can be used in unit and integration testing of a variety of MDS services. Includes functions to bulk-create random devices, telemetry for those devices, etc. \ No newline at end of file diff --git a/packages/mds-types/README.md b/packages/mds-types/README.md new file mode 100644 index 000000000..be7d57848 --- /dev/null +++ b/packages/mds-types/README.md @@ -0,0 +1,2 @@ +Collection of TypeScript descriptions of types that are shared among components, including Agency’s notion of Device and VehicleEvent, event type enums for Provider and Agency, etc. + diff --git a/packages/mds-utils/README.md b/packages/mds-utils/README.md new file mode 100644 index 000000000..cbea7c98d --- /dev/null +++ b/packages/mds-utils/README.md @@ -0,0 +1,6 @@ +# MDS Utils + +Utility functions and classes shared across various mds-core services, including +* a variety of standard Error subclasses +* an implementation of the Agency state-machine +* date and time utilities \ No newline at end of file From 68d3639ef476bd57d8292e4bb891c8b859b3f1d0 Mon Sep 17 00:00:00 2001 From: Mark Maxham Date: Mon, 9 Mar 2020 11:28:16 -0700 Subject: [PATCH 07/18] tiny WIP --- packages/mds-policy/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/mds-policy/README.md b/packages/mds-policy/README.md index 0b0c0296f..419ff4cdf 100644 --- a/packages/mds-policy/README.md +++ b/packages/mds-policy/README.md @@ -12,3 +12,7 @@ See [Quick Start](../../README.md#Installation) ## Build Deployment Package See [Build](../../README.md#Build) + +## Design + +TODO MAX \ No newline at end of file From 32eca8f58a0f0d1f7d0b3125d905b3f21f843ccd Mon Sep 17 00:00:00 2001 From: sarob Date: Mon, 16 Mar 2020 13:30:44 -0700 Subject: [PATCH 08/18] small adjustment to contribution page ref missing one ref and it didnt read well --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e6019070b..b1d289d09 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ mds-core: An open source reference implementation of the Mobility Data Specifica To stay up to date on MDS releases, meetings, and events, please **subscribe to the [mds-announce](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-announce) mailing list.** -mds-core is an open source project with all development taking place on GitHub. Comments and ideas can be shared by [creating an issue](https://github.com/openmobilityfoundation/mds-core/issues), and specific changes can be suggested by [opening a pull request](https://github.com/openmobilityfoundation/mds-core/pulls). Before contributing, please review our [CONTRIBUTING page](CONTRIBUTING.md) to understand guidelines and policies for participation and our [CODE OF CONDUCT page](CODE_OF_CONDUCT.md). +mds-core is an open source project with all development taking place on GitHub. Comments and ideas can be shared by [creating an issue](https://github.com/openmobilityfoundation/mds-core/issues), and specific changes can be suggested by [opening a pull request](https://github.com/openmobilityfoundation/mds-core/pulls). Before contributing, please review our [CONTRIBUTING page](CONTRIBUTING.md), [Release Guidelines page](ReleaseGuidelines.md), and [CODE OF CONDUCT page](CODE_OF_CONDUCT.md) to understand guidelines and policies for participation. You can also get involved in development by joining an OMF working group. The working groups maintain the OMF GitHub repositories and work through issues and pull requests. Each working group has its own mailing list for non-technical discussion and planning: From 8ae713414e4642fd8517a5f785f5e50fd3f01e94 Mon Sep 17 00:00:00 2001 From: Mark Maxham Date: Thu, 19 Mar 2020 10:45:03 -0700 Subject: [PATCH 09/18] placeholder --- packages/mds-geography/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/mds-geography/README.md b/packages/mds-geography/README.md index abada369b..23a4ac897 100644 --- a/packages/mds-geography/README.md +++ b/packages/mds-geography/README.md @@ -1 +1 @@ -# NOT FOR CONTRIBUTION +# Write dox FIXME TODO From 79f661dcdec07c8fa2db64f9cb82bf49e45dc9c3 Mon Sep 17 00:00:00 2001 From: Mark Maxham Date: Thu, 19 Mar 2020 11:08:34 -0700 Subject: [PATCH 10/18] trimming obsolete readmes --- packages/mds-event-processor/README.md | 60 -------------------------- packages/mds-provider/README.md | 1 - 2 files changed, 61 deletions(-) delete mode 100644 packages/mds-event-processor/README.md delete mode 100644 packages/mds-provider/README.md diff --git a/packages/mds-event-processor/README.md b/packages/mds-event-processor/README.md deleted file mode 100644 index 4ab4dfc31..000000000 --- a/packages/mds-event-processor/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# NOT FOR CONTRIBUTION - -# Event Streaming Processor -Included in this repo is the stream processor for vehicle event and telemetry data sent via cloudevents with Knative eventing: - -## Running the Processor -#### Setup & build -Dependencies: Istio, Knative, Natss -To install with the rest of the mds packages: -``` -mdsctl install:mds -``` -Several useful preset (-p) flags: - processors: Will only deploy mds-agency and the three pipeline processors to yourt cluster - data-external: For local development, pomts cluster to use local redis & postgres - -#### Logging -To view the Event-processor log: -``` -kubectl logs $(kubectl get pods -A -o=name | sed "s/^.\{4\}//" | grep '^event-processor') user-container -n mds -f -``` - -## Data Flow -#### Triggered by: -| type | name | description | -|------|-----|-------------| -| cloudevent | {jurisdiction_id}.event | cloudevent outputted by mds-agency of event from device | -| cloudevent | {jurisdiction_id}.telemetry | cloudevent outputted by mds-agency of telemetry from device | - -#### Output -| type | key/table | description | -|------|-----|-------------| -| postgres row | reports_device_states | entry of device id and processed event information | -| redis cache | device:state | hash set of every device's last event and metadata | -| redis cache | trips:events | hash set of every device's uncompleted trips containing their events | -| redis cache | trips:telemetry | hash set of every device's uncompleted trips containing their telemetry | -TO BE ADDED: -| redis cache | provider:state | hash set of all provider level metrics | - -#### Schema -REPORTS_DEVICE_STATES: -(PK= [timestamp, device_id, provider_id, type]) - -| Column Name (type) | Description | -| ------ | ------ | -| vehicle_type (VARCHAR) | vehicle mode (e.g. scooter) | -| type (VARCHAR) | event or telemetry | -| timestamp (BIGINT) | time of ping | -| device_id (UUID) | unique ID of device | -| provider_id (UUID) | unique ID associated with a provider | -| recorded (BIGINT) | time of insert into DB | -| annotation_version (SMALLINT) | version of geo service | -| annotation (JSON) | result of geo service that calculates service area based on lat/lng coords | -| gps (JSON) | gps data including: {lat, lng, altitude, heading, speed, accuracy} | -| service_area_id (UUID or null) | (EVENT SPECIFIC) unique ID associated with service area | -| charge (REAL) | charge of to battery on a 0 to 1 scale | -| state (VARCHAR or null) | (EVENT SPECIFIC) state of device, possibilites defined in EVENT_STATUS_MAP | -| event_type (VARCHAR or null) | (EVENT SPECIFIC) one of the possibilities defined in EVENT_STATUS_MAP | -| event_type_reason (VARCHAR or null) | (EVENT SPECIFIC) associated decorator of event_type | -| trip_id (UUID) | (EVENT SPECIFIC) unique ID of trip | diff --git a/packages/mds-provider/README.md b/packages/mds-provider/README.md deleted file mode 100644 index abada369b..000000000 --- a/packages/mds-provider/README.md +++ /dev/null @@ -1 +0,0 @@ -# NOT FOR CONTRIBUTION From 2d2dad0f0c29799282bcecd190db52616d3797c6 Mon Sep 17 00:00:00 2001 From: Mark Maxham Date: Thu, 19 Mar 2020 11:13:10 -0700 Subject: [PATCH 11/18] moar readmes --- packages/mds-event-server/README.md | 2 +- packages/mds-provider-processor/README.md | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) delete mode 100644 packages/mds-provider-processor/README.md diff --git a/packages/mds-event-server/README.md b/packages/mds-event-server/README.md index 1898b015d..23a4ac897 100644 --- a/packages/mds-event-server/README.md +++ b/packages/mds-event-server/README.md @@ -1 +1 @@ -# NOT FOR CONTRIBUTION \ No newline at end of file +# Write dox FIXME TODO diff --git a/packages/mds-provider-processor/README.md b/packages/mds-provider-processor/README.md deleted file mode 100644 index abada369b..000000000 --- a/packages/mds-provider-processor/README.md +++ /dev/null @@ -1 +0,0 @@ -# NOT FOR CONTRIBUTION From cd7575bf33522a6b0f4f375b886040aba079f2bb Mon Sep 17 00:00:00 2001 From: sarob Date: Sun, 22 Mar 2020 14:18:04 -0700 Subject: [PATCH 12/18] Update readme with running, testing mds-core also a bit about experimental content --- README.md | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 885bff8bd..0652d7737 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,8 @@ mds-core: An open source reference implementation of the Mobility Data Specifica - Agency service to ingest data from providers - Policy service to express vehicle rules and geofences -- Audit service to support field verification of data (\*) -- Compliance service to measure adherence to policies (\*) +- Audit service to support field verification of data +- Compliance service to measure adherence to policies - Daily report generator - Shared libraries used by multiple services - Deployment scripts for Kubernetes and Istio @@ -29,6 +29,27 @@ mds-core: An open source reference implementation of the Mobility Data Specifica - a place to define local policies or performance metrics - a service that will be operated by the OMF +# Running and Testing mds-core + +- Operators and developers that want to run mds-core locally use the [Developers Workstation Setup Guide](DevWorkstationSetup.md) +- Operators that want to want to run mds-core remotely follow the instructions by infrastructure types + * Bare metal servers + * Amazon Web Services + * Azure +- Developers that want to test mds-core code changes [follow the instructions in the Developers Workstation Setup Guide](DevWorkstationSetup.md#debugging-with-visual-studio-code) + +# Contributing to Experimental Content + +There are various parts of mds-core in development. Here is a partial list: +1. [Updating MDS-Agency API with `/stops` PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) +2. [Adding MDS-Audit API PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) +3. [Adding MDS-Compliance API PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) +4. MDS-Config **pending** +5. MDS-Daily **pending** +6. MDS-Metrics-Sheet **pending** +7. MDS-Policy-Author **pending** +8. MDS-Web-Sockets **pending** + # Learn More / Get Involved / Contributing To stay up to date on MDS releases, meetings, and events, please **subscribe to the [mds-announce](https://groups.google.com/a/groups.openmobilityfoundation.org/forum/#!forum/mds-announce) mailing list.** @@ -49,17 +70,3 @@ For questions about MDS please contact [info@openmobilityfoundation.org](mailto: [agency]: https://github.com/openmobilityfoundation/mobility-data-specification/tree/master/agency/README.md [provider]: https://github.com/openmobilityfoundation/mobility-data-specification/tree/master/provider/README.md [policy]: https://github.com/openmobilityfoundation/mobility-data-specification/tree/master/policy/README.md - - -TODO where should this go? Seems out of place in this trimmed-down doc. @seanr - -### Experimental Content -#### APIs -1. MDS-Agency `/stops` [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) -2. MDS-Audit [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) -3. MDS-Compliance [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) -4. MDS-Config -5. MDS-Daily -6. MDS-Metrics-Sheet -7. MDS-Policy-Author -8. MDS-Web-Sockets From 2b23a735a48408a5b85ff514bbfb81b3cb9a38d5 Mon Sep 17 00:00:00 2001 From: sarob Date: Thu, 26 Mar 2020 14:01:51 -0700 Subject: [PATCH 13/18] first pass at dev workstation setup some basic reorganization so it is easier to follow steps added index ill start verifying steps next after that a pass at vm in win, vm in macos --- DevWorkstationSetup..md | 106 ++++++++++++++++++++-------------------- README.md | 12 +++-- 2 files changed, 61 insertions(+), 57 deletions(-) diff --git a/DevWorkstationSetup..md b/DevWorkstationSetup..md index 750b05fee..322e23792 100644 --- a/DevWorkstationSetup..md +++ b/DevWorkstationSetup..md @@ -3,48 +3,36 @@ Repo for a reference implementation of MDS services intended for hosting by cities. Includes implementations of Agency and Policy services, as well as Compliance and Audit services, plus some other tools and shared libraries. -## Contents +[More information on contributing found here](README.md#learn-more--get-involved--contributing) -### Stable Content -#### APIs -1. MDS-Agency 0.4.0 Implementation -2. MDS-Policy 0.4.0 Implementation - -### Experimental Content -#### APIs -1. MDS-Agency `/stops` [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) -2. MDS-Audit [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) -3. MDS-Compliance [PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) -4. MDS-Policy-Author TODO docs -5. MDS-Daily TODO docs -6. MDS-Web-Sockets TODO docs +To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) -#### TO BE REMOVED BEFORE MERGE -1. MDS-Event-Processor (REMOVE) -2. MDS-Provider-Processor (REMOVE) -3. MDS-Trip-Processor (REMOVE) -4. MDS-Config (REMOVE) -5. MDS-Metrics (REMOVE) -6. MDS-Metrics-Sheet (REMOVE) +# Install Dependencies -## Contributing +- [PostgreSQL and Redis](#PostgreSQL-and-Redis) +- [Yarn](#Yarn) +- [NVM](#NVM) +- [Lerna](#Lerna) +- [Docker](#Docker) +- [mds-core codebase](#mds-core-codebase) +- [Kubernetes](#Kubernetes) +- [Okteto](#Okteto) -See [CONTRIBUTING.md](.github/CONTRIBUTING.md) +# Operations to support development -## Security +- [Build mds-core](#build-mds-core) +- [Run mds-core](#run-mds-core) +- [mds-core Operations](#mds-core-operations) +- [Additional Considerations](#additional-considerations) -To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) + -## Installation +# Contents -### Dependencies +## PostgreSQL and Redis -* PostgreSQL -* Redis -* [Yarn](https://yarnpkg.com/en/docs/install#mac-stable) -* [NVM](https://github.com/nvm-sh/nvm#installation-and-update) +### MacOS -#### Database config on macOS If you haven't installed PostegreSQL and Redis you can install them with homebrew on macOS ``` brew install postgresql @@ -68,11 +56,20 @@ To run tests, you will need this: Then add `export PG_NAME=mdstest` to your shell's environment file. (The name is not important, but you'll need to point it somehwere.) -#### Node setup +### Yarn + +[Installing Yarn](https://yarnpkg.com/en/docs/install#mac-stable) + +### NVM + +[Installing NVM](https://github.com/nvm-sh/nvm#installation-and-update) You should have NVM already installed from the link above. The top level directory of the project has a `.nvmrc` file and you should be able to run `nvm install` to get the right version of Node. -#### Package setup +### Lerna + +#### MacOS + Install [Lerna](https://lerna.js.org/) ```sh @@ -100,7 +97,7 @@ You can also run all tests from the project root with yarn test ``` -### Package Management - Lerna +#### Package Management - Lerna This repository is a monorepo and uses Lerna for working with its packages. @@ -136,25 +133,21 @@ Format all files lerna run prettier ``` -## Debugging with Visual Studio Code +#### Debugging with Visual Studio Code -### Node.js: Express Server +##### Node.js: Express Server * Select the `Node.js Express Server` debug configuration * Select the file that implements the Node/Express server for a package (typically `server.ts`) in the Explorer panel * Press `F5` -### Mocha Tests +##### Mocha Tests * Select the `Node.js: Mocha Tests` debug configuration * Select any one of the files in a package's test folder * Press `F5` -### Kubernetes - -MDS can readily be provisioned to a [Kubernetes](https://kubernetes.io) capable cluster, be it a local or remote. The following steps describe how to build, deploy and operate against a local MDS cluster. - -#### Prerequisites +## mds-core codebase Obtain a local working copy of MDS: @@ -162,8 +155,9 @@ Obtain a local working copy of MDS: % git clone https://github.com/lacuna-tech/mds-core % cd mds-core ``` +## Docker -OSX (Linux and Windows tbd) +### MacOS Install [Docker Desktop](https://download.docker.com/mac/stable/Docker.dmg): @@ -177,7 +171,13 @@ Start Docker-Desktop: % open /Applications/Docker.app ``` -Lastly, configure Kubernetes: +## Kubernetes + + + +MDS can readily be provisioned to a [Kubernetes](https://kubernetes.io) capable cluster, be it a local or remote. The following steps describe how to build, deploy and operate against a local MDS cluster. + +### Kubernetes through Docker: ```txt select the 'Preferences' option @@ -199,7 +199,7 @@ Verify: % kubectl cluster-info ``` -#### Bootstrap : install operational dependencies +### Bootstrap : install operational dependencies In order to build and operate MDS, a number of suporting technologies are leveraged by ensuring they are installed and operational via a one-time `bootstap` process: @@ -217,9 +217,9 @@ Verify: % k9s & ``` -#### Build : compile source into deployable images +## Build mds-core -Compiling and packaging MDS into a deployable form is achived as follows: +Compiling and packaging MDS into a deployable images is as follows: ```sh % ./bin/mdsctl build @@ -231,7 +231,7 @@ Verify: % docker images | grep mds* ``` -#### Run : install MDS +## Run mds-core (tbd: ?best profile?) @@ -245,7 +245,7 @@ Verify: % curl localhost/agency ``` -#### In-Cluster Development +## Okteto Due to the nature of MDS-Core being a highly portable Typescript project that compiles down into minified javascript for its images, rapidly development in-cluster can be quite challenging. MDS-Core utilizes [Okteto](https://okteto.com) to enable developers to actively develop their code in-cluster. After following the above steps to set up a local MDS cluster, you can override an existing service's deployment with these steps. @@ -265,13 +265,13 @@ After following the above steps to set up a local MDS cluster, you can override 5. This session is now safe to close, and you can reattach with the `okteto.${SERVICE_NAME}` ssh profile automatically added for you using the VSCode `Remote - SSH` package. 6. When you're completely done with your session, run `> Okteto Down` from the VSCode command palette, or `okteto down` from terminal to revert the changes made by Okteto, and return your service to its previous deployment. -#### MDS Operations +## mds-core Operations MDS operates atop the following services: [Kubernetes](https://kubernetes.io), [Istio](https://istio.io), [NATS](https://nats.io), [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io). (tbd) -#### Additional Considerations +## Additional Considerations Access the database: @@ -303,7 +303,7 @@ Display the complete set of operations: % ./bin/mdsctl ``` -#### Cleanup +### Cleanup ```sh % ./bin/mdsctl uninstall:mds uninstall diff --git a/README.md b/README.md index 0652d7737..e2ca4e305 100644 --- a/README.md +++ b/README.md @@ -38,12 +38,16 @@ mds-core: An open source reference implementation of the Mobility Data Specifica * Azure - Developers that want to test mds-core code changes [follow the instructions in the Developers Workstation Setup Guide](DevWorkstationSetup.md#debugging-with-visual-studio-code) -# Contributing to Experimental Content +## Stable Content +1. MDS-Agency 0.4.0 Implementation +2. MDS-Policy 0.4.0 Implementation + +## Contributing to Experimental Content There are various parts of mds-core in development. Here is a partial list: -1. [Updating MDS-Agency API with `/stops` PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) -2. [Adding MDS-Audit API PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) -3. [Adding MDS-Compliance API PR](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) +1. [Updating MDS-Agency API with `/stops` PR #430](https://github.com/openmobilityfoundation/mobility-data-specification/pull/430) +2. [Adding MDS-Audit API PR #326](https://github.com/openmobilityfoundation/mobility-data-specification/pull/326) +3. [Adding MDS-Compliance API PR #333](https://github.com/openmobilityfoundation/mobility-data-specification/pull/333) 4. MDS-Config **pending** 5. MDS-Daily **pending** 6. MDS-Metrics-Sheet **pending** From 9e50c8ee68b6b9a9357cd69cf1c53352b05b3e2f Mon Sep 17 00:00:00 2001 From: sarob Date: Fri, 27 Mar 2020 15:02:20 -0700 Subject: [PATCH 14/18] second pass at devworkstation cleanup Using a few other projects as a guide --- DevWorkstationSetup..md | 51 ++++++++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/DevWorkstationSetup..md b/DevWorkstationSetup..md index 322e23792..05f82b367 100644 --- a/DevWorkstationSetup..md +++ b/DevWorkstationSetup..md @@ -1,14 +1,12 @@ -# Overview +# Making Your First mds-core Contribution -Repo for a reference implementation of MDS services intended for hosting by cities. Includes implementations of -Agency and Policy services, as well as Compliance and Audit services, plus some other tools and shared libraries. - -[More information on contributing found here](README.md#learn-more--get-involved--contributing) - -To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) +This document will guide you through making your first mds-core contribution. Before you follow these steps, make sure you understand the mobility data specification core and community supporting it. +- [What is mds-core](README.md#mds-core-project-overview) +- [Learn More / Get Involved / Contributing](README.md#learn-more--get-involved--contributing) # Install Dependencies +- [Pre-commit](#pre-commit) - [PostgreSQL and Redis](#PostgreSQL-and-Redis) - [Yarn](#Yarn) - [NVM](#NVM) @@ -29,6 +27,12 @@ To commit code, you will need the pre-commit tool, which can be installed via `b # Contents +## Pre-commit + +To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) + +[Back to top](#install-dependencies) + ## PostgreSQL and Redis ### MacOS @@ -55,20 +59,26 @@ To run tests, you will need this: `createdb -h localhost mdstest` Then add `export PG_NAME=mdstest` to your shell's environment file. (The name is not important, but you'll need to point it somehwere.) + + [Back to top](#install-dependencies) -### Yarn +## Yarn [Installing Yarn](https://yarnpkg.com/en/docs/install#mac-stable) -### NVM +[Back to top](#install-dependencies) + +## NVM [Installing NVM](https://github.com/nvm-sh/nvm#installation-and-update) You should have NVM already installed from the link above. The top level directory of the project has a `.nvmrc` file and you should be able to run `nvm install` to get the right version of Node. -### Lerna +[Back to top](#install-dependencies) -#### MacOS +## Lerna + +### MacOS Install [Lerna](https://lerna.js.org/) @@ -147,6 +157,8 @@ lerna run prettier * Select any one of the files in a package's test folder * Press `F5` +[Back to top](#install-dependencies) + ## mds-core codebase Obtain a local working copy of MDS: @@ -155,6 +167,9 @@ Obtain a local working copy of MDS: % git clone https://github.com/lacuna-tech/mds-core % cd mds-core ``` + +[Back to top](#install-dependencies) + ## Docker ### MacOS @@ -171,6 +186,8 @@ Start Docker-Desktop: % open /Applications/Docker.app ``` +[Back to top](#install-dependencies) + ## Kubernetes @@ -217,6 +234,8 @@ Verify: % k9s & ``` +[Back to top](#install-dependencies) + ## Build mds-core Compiling and packaging MDS into a deployable images is as follows: @@ -245,7 +264,10 @@ Verify: % curl localhost/agency ``` +[Back to top](#install-dependencies) + ## Okteto + Due to the nature of MDS-Core being a highly portable Typescript project that compiles down into minified javascript for its images, rapidly development in-cluster can be quite challenging. MDS-Core utilizes [Okteto](https://okteto.com) to enable developers to actively develop their code in-cluster. After following the above steps to set up a local MDS cluster, you can override an existing service's deployment with these steps. @@ -265,12 +287,16 @@ After following the above steps to set up a local MDS cluster, you can override 5. This session is now safe to close, and you can reattach with the `okteto.${SERVICE_NAME}` ssh profile automatically added for you using the VSCode `Remote - SSH` package. 6. When you're completely done with your session, run `> Okteto Down` from the VSCode command palette, or `okteto down` from terminal to revert the changes made by Okteto, and return your service to its previous deployment. +[Back to top](#install-dependencies) + ## mds-core Operations MDS operates atop the following services: [Kubernetes](https://kubernetes.io), [Istio](https://istio.io), [NATS](https://nats.io), [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io). (tbd) +[Back to top](#install-dependencies) + ## Additional Considerations Access the database: @@ -309,5 +335,4 @@ Display the complete set of operations: % ./bin/mdsctl uninstall:mds uninstall ``` - - +[Back to top](#install-dependencies) From 9e10c9a3f24884fcb70000af00a11ed377c6ce42 Mon Sep 17 00:00:00 2001 From: sarob Date: Fri, 27 Mar 2020 16:00:40 -0700 Subject: [PATCH 15/18] fix bad filename --- DevWorkstationSetup..md => DevWorkstationSetup.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename DevWorkstationSetup..md => DevWorkstationSetup.md (100%) diff --git a/DevWorkstationSetup..md b/DevWorkstationSetup.md similarity index 100% rename from DevWorkstationSetup..md rename to DevWorkstationSetup.md From 4067c4cfba37f91fbc6af7f40e63d15884ef74d0 Mon Sep 17 00:00:00 2001 From: sarob Date: Fri, 27 Mar 2020 18:33:58 -0700 Subject: [PATCH 16/18] Added references to technical details plus a few fix ups --- DevWorkstationSetup.md | 16 +++++++++++++--- README.md | 6 ++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/DevWorkstationSetup.md b/DevWorkstationSetup.md index 05f82b367..2d93d02a1 100644 --- a/DevWorkstationSetup.md +++ b/DevWorkstationSetup.md @@ -3,6 +3,16 @@ This document will guide you through making your first mds-core contribution. Before you follow these steps, make sure you understand the mobility data specification core and community supporting it. - [What is mds-core](README.md#mds-core-project-overview) - [Learn More / Get Involved / Contributing](README.md#learn-more--get-involved--contributing) +- [Technical Details](README/md#technical-details) + +# Assumptions + +All operations assume you have: + +- Linux (Red Hat or Debian based), Windows, or macOS +- git +- homebrew (macOS) +- yum or apt-get (Linux) # Install Dependencies @@ -35,7 +45,7 @@ To commit code, you will need the pre-commit tool, which can be installed via `b ## PostgreSQL and Redis -### MacOS +### macOS If you haven't installed PostegreSQL and Redis you can install them with homebrew on macOS ``` @@ -78,7 +88,7 @@ You should have NVM already installed from the link above. The top level direct ## Lerna -### MacOS +### macOS Install [Lerna](https://lerna.js.org/) @@ -172,7 +182,7 @@ Obtain a local working copy of MDS: ## Docker -### MacOS +### macOS Install [Docker Desktop](https://download.docker.com/mac/stable/Docker.dmg): diff --git a/README.md b/README.md index e2ca4e305..b53151d36 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ mds-core: An open source reference implementation of the Mobility Data Specifica - Shared libraries used by multiple services - Deployment scripts for Kubernetes and Istio + # mds-core is… - a reference MDS implementation usable by cities @@ -29,6 +30,11 @@ mds-core: An open source reference implementation of the Mobility Data Specifica - a place to define local policies or performance metrics - a service that will be operated by the OMF +# Technical Details + - [mds-core architecture](ARCHITECTURE.md) + - COMING SOON - Vehicle states and event types explained through their implementations + - Provider and Agency API specification reconcilation [notes](https://docs.google.com/document/d/1-KggZnPtdmiDvJJOzb4NWa8v7MfdNK88quWc0y7dzyo/edit?usp=sharing) and [spreadsheet](https://docs.google.com/spreadsheets/d/1rs6pGI3LudXIFQfZLkqE5sJ4A6_WfM1439hG4IK6B0Y/edit#gid=0). These are histortical documents that show how much time and effort is going into maintaining the specification resulting from real world implementations. + # Running and Testing mds-core - Operators and developers that want to run mds-core locally use the [Developers Workstation Setup Guide](DevWorkstationSetup.md) From 3f4819c1f5591b2a12c6fdfa4250b40172f56351 Mon Sep 17 00:00:00 2001 From: sarob Date: Fri, 10 Apr 2020 15:44:27 -0700 Subject: [PATCH 17/18] More details for DevWorkstationSetup.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - shell commands and multi line outputs are bracketed as code - single line outputs are bracketed as inline code - Moved a few sections to operations - Added package descriptions - Added expected install results Some errors were logged as part of the mds-core bootstrap script. I will post these to the PR. I got all the way to the mds-core build section, where I get `curl: (52) Empty reply from server` after running 'curl localhost/agency’. --- DevWorkstationSetup.md | 432 ++++++++++++++++++++++++++--------------- 1 file changed, 280 insertions(+), 152 deletions(-) diff --git a/DevWorkstationSetup.md b/DevWorkstationSetup.md index 2d93d02a1..93ecd35b4 100644 --- a/DevWorkstationSetup.md +++ b/DevWorkstationSetup.md @@ -5,17 +5,9 @@ This document will guide you through making your first mds-core contribution. Be - [Learn More / Get Involved / Contributing](README.md#learn-more--get-involved--contributing) - [Technical Details](README/md#technical-details) -# Assumptions - -All operations assume you have: - -- Linux (Red Hat or Debian based), Windows, or macOS -- git -- homebrew (macOS) -- yum or apt-get (Linux) - -# Install Dependencies - +# Using natively installed development tools for macOS, Linux +- [Assumptions](#assumptions) +- [Homebrew (macOS)](#homebrew-(macos)) - [Pre-commit](#pre-commit) - [PostgreSQL and Redis](#PostgreSQL-and-Redis) - [Yarn](#Yarn) @@ -24,325 +16,461 @@ All operations assume you have: - [Docker](#Docker) - [mds-core codebase](#mds-core-codebase) - [Kubernetes](#Kubernetes) -- [Okteto](#Okteto) + +# Using Container based Development tools + +- coming soon, currently under development # Operations to support development +- [Bootstrap Operational Dependancies](#bootstrap-operational-dependencies) - [Build mds-core](#build-mds-core) - [Run mds-core](#run-mds-core) -- [mds-core Operations](#mds-core-operations) +- [Okteto](#Okteto) +- [mds-core Operations (under development)](#mds-core-operations) +- [Launching a local server for a package](#launching-a-local-server-for-a-package) +- [Package Management with Lerna](#package-management-with-lerna) +- [Debugging with Visual Studio Code](#debugging-with-visual-studio-code) - [Additional Considerations](#additional-considerations) # Contents -## Pre-commit +## Assumptions -To commit code, you will need the pre-commit tool, which can be installed via `brew install pre-commit`. For more information, see [SECURITY.md](.github/SECURITY.md) +All operations assume you are running the commands in a shell or console and you have: -[Back to top](#install-dependencies) +- Linux (Red Hat or Debian based) OR macOS +- [git](https://git-scm.com/downloads) revision control is generally part of any standard install +- yum or apt-get (Linux) package management is generally part of any standard install +- All commands by default are executed through a terminal shell in your $HOME directory. Find by `echo $HOME`. +- Determine where you want your git repositories to be. A common location is your user or $HOME directory. -## PostgreSQL and Redis +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -### macOS +## Homebrew (macOS) -If you haven't installed PostegreSQL and Redis you can install them with homebrew on macOS -``` -brew install postgresql -brew install redis -``` +Homebrew simplifies installing complex packages on macOS. -Make sure they are running before you run the tests -``` -brew services start postgresql -brew services start redis +new install +```sh +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" ``` + OR verify existing install is up-to-date + ```sh + brew upgrade + ``` -If you encounter the following error: -`FATAL: database “” does not exist` +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -The following command should fix your issue -`createdb -h localhost` - -To run tests, you will need this: -`createdb -h localhost mdstest` +## Pre-commit -Then add `export PG_NAME=mdstest` to your shell's environment file. (The name is not important, but you'll need to point it somehwere.) - - [Back to top](#install-dependencies) +To perform checks of your code before committing to github, you will need the pre-commit tool. +```sh +brew install pre-commit -## Yarn +brew upgrade +``` -[Installing Yarn](https://yarnpkg.com/en/docs/install#mac-stable) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -[Back to top](#install-dependencies) +## PostgreSQL and Redis -## NVM +You will need a database and in-memory data store. Install PostegreSQL and Redis with homebrew on macOS +```sh +brew install postgresql -[Installing NVM](https://github.com/nvm-sh/nvm#installation-and-update) +brew install redis +``` -You should have NVM already installed from the link above. The top level directory of the project has a `.nvmrc` file and you should be able to run `nvm install` to get the right version of Node. +Always run the services +```sh +brew services start postgresql -[Back to top](#install-dependencies) +brew services start redis +``` +OR run start manually each time +```sh +pg_ctl -D /usr/local/var/postgres start -## Lerna +redis-server /usr/local/etc/redis.conf +``` -### macOS +If you encounter the following error: +`FATAL: database “” does not exist` -Install [Lerna](https://lerna.js.org/) +- The following command should fix your issue +```sh +createdb -h localhost +``` +To run tests, you will need this: ```sh -yarn global add lerna +createdb -h localhost mdstest ``` -Install all packages. Uses Yarn workspaces. +- Then add `export PG_NAME=mdstest` to your shell's environment file. ```sh -yarn install +echo "export PG_NAME=mdstest" >> ~/.bash_profile ``` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -#### Launching a local server for a package -Now you can work with each package +## Yarn + +Yarn is used for package management. For macOS install by ```sh -cd packages/mds-audit -yarn test -yarn start +brew install yarn ``` -#### Running the tests -You can also run all tests from the project root with -``` -yarn test -``` +Otherwise [follow the platform specific installation instructions](https://yarnpkg.com/en/docs/install#mac-stable) -#### Package Management - Lerna +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -This repository is a monorepo and uses Lerna for working with its packages. +## NVM -#### Example commands +nvm is a version manager for node.js. [Install NVM through a series of commands.](https://github.com/nvm-sh/nvm#installation-and-update) -Run all test suites at once + +To verify the installation was successful, run ```sh -lerna run test -``` +. ~/.bash_profile -Run all tests suites sequentially +nvm --version +``` +to reload the bash profile and return the version of Node installed. -```sh -lerna run test --concurrency 1 -``` +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -Run tests for a particular package +## Lerna +Lerna optimizes workflow for multi-package repositories for git and npm. ```sh -lerna run test --scope mds-audit -``` +npm install --global lerna -Clean all dependencies +git init lerna-repo && cd lerna-repo -```sh -lerna run clean +lerna init ``` -Format all files - +Configure yarn to use lerna ```sh -lerna run prettier +yarn global add lerna ``` -#### Debugging with Visual Studio Code +Install all packages. Uses Yarn workspaces. -##### Node.js: Express Server +```sh +yarn install +``` -* Select the `Node.js Express Server` debug configuration -* Select the file that implements the Node/Express server for a package (typically `server.ts`) in the Explorer panel -* Press `F5` +If you have any problems or want to learn more, review your options at the [Lerna website](https://lerna.js.org/). -##### Mocha Tests - -* Select the `Node.js: Mocha Tests` debug configuration -* Select any one of the files in a package's test folder -* Press `F5` - -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## mds-core codebase -Obtain a local working copy of MDS: +Obtain a local working copy of mds-core: ```sh -% git clone https://github.com/lacuna-tech/mds-core -% cd mds-core +git clone https://github.com/lacuna-tech/mds-core + +cd mds-core ``` -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Docker -### macOS - -Install [Docker Desktop](https://download.docker.com/mac/stable/Docker.dmg): +Install Docker Desktop for macOS ```sh -% open https://download.docker.com/mac/stable/Docker.dmg +open https://download.docker.com/mac/stable/Docker.dmg ``` -Start Docker-Desktop: +Start Docker-Desktop +- Through the Finder execute /Downloads/Docker.dmg +- Drag and drop Docker.app to Applications in the popup panel +- Accept the notice `Docker.app is an app downloaded from the internet` +- You can ignore the Docker Hub logon request -```sh -% open /Applications/Docker.app -``` +If you have problems or are running a different OS, [review the Docker installation documentation](https://docs.docker.com/get-docker/). -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Kubernetes -MDS can readily be provisioned to a [Kubernetes](https://kubernetes.io) capable cluster, be it a local or remote. The following steps describe how to build, deploy and operate against a local MDS cluster. +mds-core can readily be provisioned to a [Kubernetes](https://kubernetes.io) capable cluster, be it a local or remote. The following steps describe how to build, deploy and operate against a local MDS cluster. ### Kubernetes through Docker: -```txt -select the 'Preferences' option -select the 'Resources' option - apply the following minimal resource changes: - CPUs: 6 - Memory: 8G - Swap: 1G -select the 'Kubernetes' option - select 'Enable Kubernetes' option -select 'Apply & Restart' -``` +In Docker +- select the 'Preferences' option +- select the 'Resources' option + - apply the following minimal resource changes: + - CPUs: 6 + - Memory: 8G + - Swap: 1G +- select the 'Kubernetes' option + - select 'Enable Kubernetes' option +- select 'Apply & Restart' -Verify: +Verify Kebernetes is working properly ```sh -% which kubectl -% kubectl config set-context docker-desktop -% kubectl cluster-info +which kubectl + +kubectl config set-context docker-desktop + +kubectl cluster-info ``` -### Bootstrap : install operational dependencies +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) -In order to build and operate MDS, a number of suporting technologies are leveraged by ensuring they are installed and operational via a one-time `bootstap` process: +## Bootstrap Operational Dependencies +In order to build and operate mds-core, a number of suporting technologies are leveraged. They are installed and made operational via a one-time `bootstap` process. + +While in the mds-core repository ```sh -% ./bin/mdsctl bootstrap +./bin/mdsctl bootstrap ``` +- Accept Helm accepting incoming network connection notice. -The principle tools are: [homebrew](https://brew.sh), [bash-4.x+](https://www.gnu.org/software/bash/), [oq](https://github.com/Blacksmoke16/oq), [jq](https://stedolan.github.io/jq/), [yarn](https://yarnpkg.com/), [nvm](https://github.com/nvm-sh/nvm), [helm-2.14.1](https://helm.sh), [k9s](https://github.com/derailed/k9s), [kubectx](https://github.com/ahmetb/kubectx), [git](https://git-scm.com/), [gcloud](https://cloud.google.com/sdk/) and [awscli](https://aws.amazon.com/cli/). Additionally the following services are provisioned: [istio](https://istio.io) and [nats](https://nats.io). +The principle tools are: [homebrew](https://brew.sh), [bash-4.x+](https://www.gnu.org/software/bash/), [oq](https://github.com/Blacksmoke16/oq), [jq](https://stedolan.github.io/jq/), [yarn](https://yarnpkg.com/), [nvm](https://github.com/nvm-sh/nvm), [helm-2.14.1](https://helm.sh), [k9s](https://github.com/derailed/k9s), [kubectx](https://github.com/ahmetb/kubectx), [git](https://git-scm.com/), [gcloud](https://cloud.google.com/sdk/) and [awscli](https://aws.amazon.com/cli/). Additionally the [istio](https://istio.io) and [nats](https://nats.io) services are provisioned. -Verify: +Verify the installation was successful ```sh -% kubectl -n istio-system get pods -% kubectl -n nats get pods -% k9s & +kubectl -n istio-system get pods +``` +You should get +```sh +NAME READY STATUS RESTARTS AGE +istio-citadel-74f7d4957f-fltpn 1/1 Running 0 3m41s +istio-galley-678f65b8d9-kbtsd 1/1 Running 0 3m41s +istio-ingressgateway-745c9b488d-ckjpt 1/1 Running 0 3m41s +istio-init-crd-10-1.3.6-cj5js 0/1 Completed 0 3m59s +istio-init-crd-11-1.3.6-xnnwj 0/1 Completed 0 3m59s +istio-init-crd-12-1.3.6-stssx 0/1 Completed 0 3m59s +istio-pilot-d6dc8547b-prh8b 2/2 Running 0 3m41s +istio-policy-7b5cccd8cd-9r995 2/2 Running 3 3m41s +istio-sidecar-injector-69579b5cc4-n75dt 1/1 Running 0 3m41s +istio-telemetry-8686476f99-zljtd 2/2 Running 3 3m41s +prometheus-7d7b9f7844-lhhjk 1/1 Running 0 3m41s +``` +And +```sh +kubectl -n nats get pods +``` +You should get +```sh +NAME READY STATUS RESTARTS AGE +nats-operator-b8f4977fc-ng87n 1/1 Running 0 4m53s +nats-streaming-stateful-0 0/1 CrashLoopBackOff 5 4m49s ``` -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Build mds-core -Compiling and packaging MDS into a deployable images is as follows: +Compiling and packaging mds-core into a deployable images ```sh -% ./bin/mdsctl build +./bin/mdsctl build ``` -Verify: +Verify it worked ```sh -% docker images | grep mds* +docker images | grep mds* ``` +You should get +```sh +mds-jurisdiction 0.0.1-docs-omf-release-0-1-9c71905f c8b1e2eb34ec 7 minutes ago 89.6MB +mds-metrics-sheet 0.1.27-docs-omf-release-0-1-9c71905f 19370c31dd6f 7 minutes ago 88.3MB +mds-web-sockets 0.0.1-docs-omf-release-0-1-9c71905f 6bb46a44abf3 7 minutes ago 88.3MB +mds-policy-author 0.1.26-docs-omf-release-0-1-9c71905f cfebe57d6cb6 8 minutes ago 87.5MB +mds-native 0.0.24-docs-omf-release-0-1-9c71905f a920a1019959 8 minutes ago 87.5MB +mds-policy 0.1.26-docs-omf-release-0-1-9c71905f 06e9e3a6abc3 8 minutes ago 87.5MB +mds-audit 0.1.27-docs-omf-release-0-1-9c71905f 66403a69b00b 9 minutes ago 143MB +mds-agency 0.1.26-docs-omf-release-0-1-9c71905f 669ad291a73c 10 minutes ago 89MB +mds-compliance 0.1.26-docs-omf-release-0-1-9c71905f 6d7ac8153b83 11 minutes ago 87.6MB +mds-daily 0.1.26-docs-omf-release-0-1-9c71905f 818df45273da 11 minutes ago 87.6MB +mds-event-processor 0.0.1-docs-omf-release-0-1-9c71905f 9eacd480471f 11 minutes ago 87.8MB +mds-geography 0.0.1-docs-omf-release-0-1-9c71905f d3520918208e 11 minutes ago 87.3MB +mds-geography-author 0.0.1-docs-omf-release-0-1-9c71905f 7859d3d522e9 11 minutes ago 87.5MB +mds-config 0.0.2-docs-omf-release-0-1-9c71905f aa2a1a99ecde 11 minutes ago 87.2MB +``` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Run mds-core -(tbd: ?best profile?) + ```sh -% ./bin/mdsctl -p minimal install:mds +./bin/mdsctl -p minimal install:mds ``` Verify: ```sh -% curl localhost/agency +curl localhost/agency ``` -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Okteto -Due to the nature of MDS-Core being a highly portable Typescript project that compiles down into minified javascript for its images, rapidly development in-cluster can be quite challenging. MDS-Core utilizes [Okteto](https://okteto.com) to enable developers to actively develop their code in-cluster. +Due to the nature of mds-core being a highly portable Typescript project that compiles down into minified javascript for its images, rapidly development in-cluster can be quite challenging. mds-core utilizes [Okteto](https://okteto.com) to enable developers to actively develop their code in-cluster. After following the above steps to set up a local MDS cluster, you can override an existing service's deployment with these steps. 1. Update `mds-core/okteto.yml`'s `name` field to be set to the service you wish to replace (e.g. `mds-agency`) 2. ```sh -% curl https://get.okteto.com -sSfL | sh +curl https://get.okteto.com -sSfL | sh ``` 3. Install the `Remote - Kubernetes` VSCode extension. 4. Run `> Okteto Up` from the VSCode command palette. * After the remote session opens, execute this in the new shell window: ```sh -% yarn -% cd packages/${SERVICE_NAME} -% yarn start +yarn + +cd packages/${SERVICE_NAME} + +yarn start ``` 5. This session is now safe to close, and you can reattach with the `okteto.${SERVICE_NAME}` ssh profile automatically added for you using the VSCode `Remote - SSH` package. 6. When you're completely done with your session, run `> Okteto Down` from the VSCode command palette, or `okteto down` from terminal to revert the changes made by Okteto, and return your service to its previous deployment. -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## mds-core Operations MDS operates atop the following services: [Kubernetes](https://kubernetes.io), [Istio](https://istio.io), [NATS](https://nats.io), [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io). -(tbd) +More details to follow + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + +## Launching a local server for a package +Now you can work with each package + +```sh +cd packages/mds-audit + +yarn test + +yarn start +``` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + +## Running the tests +You can also run all tests from the project root with +```sh +yarn test +``` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + +## Package Management with Lerna + +This repository is a monorepo and uses Lerna for working with its packages. + +### Example commands + +Run all test suites at once + +```sh +lerna run test +``` + +Run all tests suites sequentially + +```sh +lerna run test --concurrency 1 +``` + +Run tests for a particular package + +```sh +lerna run test --scope mds-audit +``` + +Clean all dependencies + +```sh +lerna run clean +``` + +Format all files + +```sh +lerna run prettier +``` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + +## Debugging with Visual Studio Code + +### Node.js: Express Server + +* Select the `Node.js Express Server` debug configuration +* Select the file that implements the Node/Express server for a package (typically `server.ts`) in the Explorer panel +* Press `F5` + +### Mocha Tests + +* Select the `Node.js: Mocha Tests` debug configuration +* Select any one of the files in a package's test folder +* Press `F5` -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Additional Considerations Access the database: ```sh -% ./bin/mdsctl cli:postgresql +./bin/mdsctl cli:postgresql ``` Access the cache: ```sh -% ./bin/mdsctl cli:redis +./bin/mdsctl cli:redis ``` (tbd) Access the event stream: ```sh -% ./bin/mdsctl install:natsbox +./bin/mdsctl install:natsbox ``` Access the MDS cluster: ```sh -% k9s +k9s ``` Display the complete set of operations: ```sh -% ./bin/mdsctl +./bin/mdsctl ``` ### Cleanup ```sh -% ./bin/mdsctl uninstall:mds uninstall +./bin/mdsctl uninstall:mds uninstall ``` -[Back to top](#install-dependencies) +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) From d8429175e35cce91929478c6c084d74947bb270e Mon Sep 17 00:00:00 2001 From: sarob Date: Sun, 19 Apr 2020 17:59:01 -0700 Subject: [PATCH 18/18] More DevWorkstationUpdate work I needed to manually reset the node version for yarn. Once past that the install part of the script appears to exit clean. However, during the build it existed with @container-images/mds-geography-author: ../../node_modules/@mds-core/mds-types/dist/index.d.ts(2,72): error TS2306: File '/Users/sarob/code/lacuna-tech-mds-core/node_modules/@mds-core/mds-types/dist/scopes.d.ts' is not a module. I will include the updated config files this time. --- DevWorkstationSetup.md | 154 ++++++++++++++++++++++++++++++++++++++++- package.json | 2 +- yarn.lock | 74 ++++++++------------ 3 files changed, 179 insertions(+), 51 deletions(-) diff --git a/DevWorkstationSetup.md b/DevWorkstationSetup.md index 93ecd35b4..ebcbf6d2a 100644 --- a/DevWorkstationSetup.md +++ b/DevWorkstationSetup.md @@ -24,10 +24,13 @@ This document will guide you through making your first mds-core contribution. Be # Operations to support development - [Bootstrap Operational Dependancies](#bootstrap-operational-dependencies) +- [Bootstrap script Help](#bootstrap-script-help) +- [In case of Bootstrap script install errors, break glass](#in-case-of-bootstrap-script-install-errors-break-glass) - [Build mds-core](#build-mds-core) +- [In case of Bootstrap script build errors, break glass](#in-case-of-bootstrap-script-build-errors-break-glass) - [Run mds-core](#run-mds-core) - [Okteto](#Okteto) -- [mds-core Operations (under development)](#mds-core-operations) +- [mds-core Operations](#mds-core-operations) - [Launching a local server for a package](#launching-a-local-server-for-a-package) - [Package Management with Lerna](#package-management-with-lerna) - [Debugging with Visual Studio Code](#debugging-with-visual-studio-code) @@ -243,7 +246,21 @@ While in the mds-core repository ``` - Accept Helm accepting incoming network connection notice. -The principle tools are: [homebrew](https://brew.sh), [bash-4.x+](https://www.gnu.org/software/bash/), [oq](https://github.com/Blacksmoke16/oq), [jq](https://stedolan.github.io/jq/), [yarn](https://yarnpkg.com/), [nvm](https://github.com/nvm-sh/nvm), [helm-2.14.1](https://helm.sh), [k9s](https://github.com/derailed/k9s), [kubectx](https://github.com/ahmetb/kubectx), [git](https://git-scm.com/), [gcloud](https://cloud.google.com/sdk/) and [awscli](https://aws.amazon.com/cli/). Additionally the [istio](https://istio.io) and [nats](https://nats.io) services are provisioned. +The principle tools are: +- [homebrew](https://brew.sh) +- [bash-4.x+](https://www.gnu.org/software/bash/) +- [oq](https://github.com/Blacksmoke16/oq) +- [jq](https://stedolan.github.io/jq/) +- [yarn](https://yarnpkg.com/) +- [nvm](https://github.com/nvm-sh/nvm) +- [helm-2.14.1](https://helm.sh) +- [k9s](https://github.com/derailed/k9s) +- [kubectx](https://github.com/ahmetb/kubectx) +- [git](https://git-scm.com/) +- [gcloud](https://cloud.google.com/sdk/) +- [awscli](https://aws.amazon.com/cli/). +- [istio](https://istio.io) +- [nats](https://nats.io) Verify the installation was successful @@ -276,6 +293,131 @@ nats-operator-b8f4977fc-ng87n 1/1 Running 0 4m53s nats-streaming-stateful-0 0/1 CrashLoopBackOff 5 4m49s ``` +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + +## Bootstrap script Help + +``` +$ ./mdsctl --help + +usage: mdsctl [--options] [commands] + +options: + [-c | --configure] [{key}={value}[,{key}={value}] : set configuration value(s) by key(s) + [{key}+={value}[,{key}+={value}] : append configuraiton value(s) by key(s) + [{key}= : clear configuration value(s) by key(s) + [-p | --preset] [preset-key[,preset-key]] : preset configuration(s) + [-w | --workdir] [WORK-DIR] : specify working directory + [-s | --sleep] [PAUSE] : specify pause time + [-q | --quiet] : less verbose + [-h | --help] : usage + +where preset-key in: + minimal : minimal service deployment; default: preset(local) + mds-agency, postgresql, redis + local : local resource (cpu, memory) deployment; default: limitsCpu=200m, limitsMemory=200Mi, requestsCpu=20m, requestsMemory=32Mi + disabled : disable service deployment; default: all + no-persistence : disable persistence + +commands: + bootstrap : install dependencies; default: helm,pgcli,helm,istio,nats,stan + build : build project + install[:{service[,{service}]}] : install specified service; default: mds + cycle[]:[app[,app]]] : restart app; default: cycle all services + test[:unit,integration] : preform specified tests; default: unit,integration + open:[app[,{app}]] : opens a browser for the provided application(s) + forward[:{app[,{app}]}] : register port-forwarding for the provided application(s); default: default + unforward[:{app[,{app}]}] : deregister port-forwarding for the provided application(s) + cli:[postgresql,redis] : create a cli console for the provided service + uninstall[:{service[,{service}]}] : uninstall specified service(s); default: logging,mds,stan,nats,istio,helm + reinstall[:{service[,{service}]}] : reinstall specified service(s); default: mds + home : return installation directory + completion : return bash-completion + +where service in: + helm + kiali + istio + mds + fallbackCertificate : requires ingress domain; eg: -c:ingress-domain=[DOMAIN] + ingressGatewayCertificate : requires certificate path; eg: -c:ingress-gateway-key-path=[KEY-PATH],ingress-gateway-certificate-path=[CERT-PATH] + metricsAdapter + logging + metrics + nats + stan + grafana + dns + curl + +where app in: + mds-agency + mds-audit + mds-compliance + mds-daily + mds-event-processor + mds-geography + mds-geography-author + mds-native + mds-policy + mds-policy-author + mds-postgresql + mds-redis-headless + grafana : grafana; see: https://grafana.com + kibana : kibana; see https://www.elastic.co/products/kibana + prometheus : prometheus; see https://prometheus.io + jaeger : jaeger; see https://www.jaegertracing.io + kiali : kiali; see https://www.kiali.io + +example: + % ./bin/mdsctl bootstrap build install:mds test:integration + +pre-requisites: + docker desktop with kubernetes : see https://www.docker.com/products/docker-desktop + +``` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + +## In case of Bootstrap script install errors, break glass + +Bootstrap commands one by one + +`Note that we could fully document the bootstrap script inline as an alternative to here.` + +First, if the file package.json text is not +``` +"engines": { + "node": "12.14.1" +``` +then reset back to these values or yarn will not install the packages. You can attempt to use `./bin/mdsctl bootstrap` again. + +1. brew install -oq jq envsubst yarn nvm helm k9s kubectx git awscli +1. brew-cask install -gcloud +1. nvm install -12.14.1 +1. nvm alias default -12.14.1 +1. yarn lerna --version +1. yarn cypress --version +1. yarn mocha --version +1. yarn chai --version +1. yarn mochawesome --version +1. helm init --force-upgrade || usage "helm intialization failure" +1. helm repo add https://kubernetes-charts.storage.googleapis.com +1. helm repo add https://storage.googleapis.com/istio-release/releases/1.3.6/charts +1. helm repo add https://kubernetes-charts.banzaicloud.com +1. helm repo add https://helm.elastic.co +1. helm repo add https://charts.bitnami.com/bitnami +1. helm dependency update [[ $(helm plugin list | grep unittest) ]] || helm plugin install https://github.com/lrills/helm-unittest +1. mkdir /tmp/mds/tools +1. cd /tmp/mds/tools +1. curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.3.6 + +`Continue working on this manual bootstrap. Stopped at line 528` + +1. nats +1. stan + + + [Back to top](#using-natively-installed-development-tools-for-macOS-linux) ## Build mds-core @@ -311,6 +453,12 @@ mds-config 0.0.2-docs-omf-release-0-1-9c71905f aa2a [Back to top](#using-natively-installed-development-tools-for-macOS-linux) +## In case of Bootstrap script build errors, break glass + +`TBD` + +[Back to top](#using-natively-installed-development-tools-for-macOS-linux) + ## Run mds-core @@ -356,7 +504,7 @@ yarn start MDS operates atop the following services: [Kubernetes](https://kubernetes.io), [Istio](https://istio.io), [NATS](https://nats.io), [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io). -More details to follow +`More details to follow` [Back to top](#using-natively-installed-development-tools-for-macOS-linux) diff --git a/package.json b/package.json index 6ca6b7a9c..84ef3ac85 100644 --- a/package.json +++ b/package.json @@ -55,7 +55,7 @@ "husky": "4.2.3", "lerna": "3.20.2", "mocha": "7.1.0", - "mochawesome": "5.0.0", + "mochawesome": "6.1.0", "mockdate": "2.0.5", "node-rsa": "1.0.7", "nodemon": "2.0.2", diff --git a/yarn.lock b/yarn.lock index 6ede2a538..cbdc3e4ba 100644 --- a/yarn.lock +++ b/yarn.lock @@ -2834,6 +2834,14 @@ chalk@^3.0.0: ansi-styles "^4.1.0" supports-color "^7.1.0" +chalk@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/chalk/-/chalk-4.0.0.tgz#6e98081ed2d17faab615eb52ac66ec1fe6209e72" + integrity sha512-N9oWFcegS0sFr9oh1oz2d7Npos6vNoWW9HvtCg5N1KRFpUhaAhvTv5Y58g880fZaEYSNm3qDz8SU1UrGvp+n7A== + dependencies: + ansi-styles "^4.1.0" + supports-color "^7.1.0" + chardet@^0.7.0: version "0.7.0" resolved "https://registry.yarnpkg.com/chardet/-/chardet-0.7.0.tgz#90094849f0937f2eedc2425d0d28a9e5f0cbad9e" @@ -4054,7 +4062,7 @@ es6-promisify@^5.0.0: dependencies: es6-promise "^4.0.3" -escape-html@~1.0.3: +escape-html@^1.0.3, escape-html@~1.0.3: version "1.0.3" resolved "https://registry.yarnpkg.com/escape-html/-/escape-html-1.0.3.tgz#0258eae4d3d0c0974de1c169188ef0051d1d1988" integrity sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg= @@ -6883,7 +6891,7 @@ long@^4.0.0: resolved "https://registry.yarnpkg.com/long/-/long-4.0.0.tgz#9a7b71cfb7d361a194ea555241c92f7468d5bf28" integrity sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA== -loose-envify@^1.1.0, loose-envify@^1.4.0: +loose-envify@^1.4.0: version "1.4.0" resolved "https://registry.yarnpkg.com/loose-envify/-/loose-envify-1.4.0.tgz#71ee51fa7be4caec1a63839f7e682d8132d30caf" integrity sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q== @@ -7337,40 +7345,39 @@ mocha@7.1.0: yargs-parser "13.1.1" yargs-unparser "1.6.0" -mochawesome-report-generator@^4.1.0: - version "4.1.0" - resolved "https://registry.yarnpkg.com/mochawesome-report-generator/-/mochawesome-report-generator-4.1.0.tgz#f303e6acb5b30fd900f2ed8a0ba2054a034c9c39" - integrity sha512-8diUnfzLqMPybIsq3aw3Zc4Npw9W2ZCx8/fMR0ShAXfDTtPH4t2mRykXEWBhsBA5+jM74mjWpwEqY6Pmz+pCsw== +mochawesome-report-generator@^5.1.0: + version "5.1.0" + resolved "https://registry.yarnpkg.com/mochawesome-report-generator/-/mochawesome-report-generator-5.1.0.tgz#b8809e7661ac31732fa7ae7210380f704f7c68f6" + integrity sha512-5cI4Jh+sD+jIxc7q94961vnm/6VKDI7TFUPt9dps6oAc4y4WMpEeeOlmgKKM81q2eGaviNUYw+acFalGK6EJ9g== dependencies: chalk "^2.4.2" dateformat "^3.0.2" + escape-html "^1.0.3" fs-extra "^7.0.0" fsu "^1.0.2" lodash.isfunction "^3.0.8" opener "^1.4.2" prop-types "^15.7.2" - react "^16.8.5" - react-dom "^16.8.5" tcomb "^3.2.17" tcomb-validation "^3.3.0" validator "^10.11.0" yargs "^13.2.2" -mochawesome@5.0.0: - version "5.0.0" - resolved "https://registry.yarnpkg.com/mochawesome/-/mochawesome-5.0.0.tgz#eb3d96e589ccdd30146bbde581fb3deec01bcbec" - integrity sha512-1Vb0G8rqURptOZUmU5xLkLUWKYlx97IoMF2/xW18tL08Z1CZaUbzLAGbgq/s3DCn/vOrb8Dy7swb/cszp3Ylpg== +mochawesome@6.1.0: + version "6.1.0" + resolved "https://registry.yarnpkg.com/mochawesome/-/mochawesome-6.1.0.tgz#3da365a17d4e6e754df01969bdf94797fef4a83a" + integrity sha512-xbVA+sXuyF9nk/o70DzQnxdro8LuBA7Tf4h0Bdvqg5r5OyhVa12t0OWAMqdGz6UXEJxKD+yXyqB49iokE38UBg== dependencies: - chalk "^3.0.0" + chalk "^4.0.0" diff "^4.0.1" json-stringify-safe "^5.0.1" lodash.isempty "^4.4.0" lodash.isfunction "^3.0.9" lodash.isobject "^3.0.2" lodash.isstring "^4.0.1" - mochawesome-report-generator "^4.1.0" + mochawesome-report-generator "^5.1.0" strip-ansi "^6.0.0" - uuid "^7.0.0" + uuid "^7.0.3" mockdate@2.0.5: version "2.0.5" @@ -8669,7 +8676,7 @@ promzard@^0.3.0: dependencies: read "1" -prop-types@^15.6.2, prop-types@^15.7.2: +prop-types@^15.7.2: version "15.7.2" resolved "https://registry.yarnpkg.com/prop-types/-/prop-types-15.7.2.tgz#52c41e75b8c87e72b9d9360e0206b99dcbffa6c5" integrity sha512-8QQikdH7//R2vurIJSutZ1smHYTcLpRWEOlHnzcWHmBYrOGUysKwSsrC89BCiFj3CbrfJ/nXFdJepOVrY1GCHQ== @@ -8855,30 +8862,11 @@ rc@^1.0.1, rc@^1.1.6, rc@^1.2.7: minimist "^1.2.0" strip-json-comments "~2.0.1" -react-dom@^16.8.5: - version "16.13.0" - resolved "https://registry.yarnpkg.com/react-dom/-/react-dom-16.13.0.tgz#cdde54b48eb9e8a0ca1b3dc9943d9bb409b81866" - integrity sha512-y09d2c4cG220DzdlFkPTnVvGTszVvNpC73v+AaLGLHbkpy3SSgvYq8x0rNwPJ/Rk/CicTNgk0hbHNw1gMEZAXg== - dependencies: - loose-envify "^1.1.0" - object-assign "^4.1.1" - prop-types "^15.6.2" - scheduler "^0.19.0" - react-is@^16.8.1: version "16.13.0" resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.0.tgz#0f37c3613c34fe6b37cd7f763a0d6293ab15c527" integrity sha512-GFMtL0vHkiBv9HluwNZTggSn/sCyEt9n02aM0dSAjGGyqyNlAyftYm4phPxdvCigG15JreC5biwxCgTAJZ7yAA== -react@^16.8.5: - version "16.13.0" - resolved "https://registry.yarnpkg.com/react/-/react-16.13.0.tgz#d046eabcdf64e457bbeed1e792e235e1b9934cf7" - integrity sha512-TSavZz2iSLkq5/oiE7gnFzmURKZMltmi193rm5HEoUDAXpzT9Kzw6oNZnGoai/4+fUnm7FqS5dwgUL34TujcWQ== - dependencies: - loose-envify "^1.1.0" - object-assign "^4.1.1" - prop-types "^15.6.2" - read-cmd-shim@^1.0.1: version "1.0.5" resolved "https://registry.yarnpkg.com/read-cmd-shim/-/read-cmd-shim-1.0.5.tgz#87e43eba50098ba5a32d0ceb583ab8e43b961c16" @@ -9402,14 +9390,6 @@ sax@>=0.6.0: resolved "https://registry.yarnpkg.com/sax/-/sax-1.2.4.tgz#2816234e2378bddc4e5354fab5caa895df7100d9" integrity sha512-NqVDv9TpANUjFm0N8uM5GxL36UgKi9/atZw+x7YFnQ8ckwFGKrl4xX4yWtrey3UJm5nP1kUbnYgLopqWNSRhWw== -scheduler@^0.19.0: - version "0.19.0" - resolved "https://registry.yarnpkg.com/scheduler/-/scheduler-0.19.0.tgz#a715d56302de403df742f4a9be11975b32f5698d" - integrity sha512-xowbVaTPe9r7y7RUejcK73/j8tt2jfiyTednOvHbA8JoClvMYCp+r8QegLwK/n8zWQAtZb1fFnER4XLBZXrCxA== - dependencies: - loose-envify "^1.1.0" - object-assign "^4.1.1" - schema-utils@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/schema-utils/-/schema-utils-1.0.0.tgz#0b79a93204d7b600d4b2850d1f66c2a34951c770" @@ -10877,10 +10857,10 @@ uuid@^3.0.1, uuid@^3.3.2, uuid@^3.3.3: resolved "https://registry.yarnpkg.com/uuid/-/uuid-3.4.0.tgz#b23e4358afa8a202fe7a100af1f5f883f02007ee" integrity sha512-HjSDRw6gZE5JMggctHBcjVak08+KEVhSIiDzFnT9S9aegmp85S/bReBVTb4QTFaRNptJ9kuYaNhnbNEOkbKb/A== -uuid@^7.0.0: - version "7.0.1" - resolved "https://registry.yarnpkg.com/uuid/-/uuid-7.0.1.tgz#95ed6ff3d8c881cbf85f0f05cc3915ef994818ef" - integrity sha512-yqjRXZzSJm9Dbl84H2VDHpM3zMjzSJQ+hn6C4zqd5ilW+7P4ZmLEEqwho9LjP+tGuZlF4xrHQXT0h9QZUS/pWA== +uuid@^7.0.3: + version "7.0.3" + resolved "https://registry.yarnpkg.com/uuid/-/uuid-7.0.3.tgz#c5c9f2c8cf25dc0a372c4df1441c41f5bd0c680b" + integrity sha512-DPSke0pXhTZgoF/d+WSt2QaKMCFSfx7QegxEWT+JOuHF5aWrKEn0G+ztjuJg/gG8/ItK+rbPCD/yNv8yyih6Cg== v8-compile-cache@2.0.3: version "2.0.3"