Shopline Nodejs Framework

Objective

This framework sets up an organized NodeJS project in an easier and 'classier' manner. Since this is being used internally, we added our way of standardizing the code structure.

This framework wraps around ExpressJS, with the following additions:

1. App class - A big 'motherboard'
2. A router
3. some default middlewares
4. Dockerfile
5. project folder structure

App class

This framework has an App Class which acts as a 'motherboard' of the whole application. It controls the phases of the app in the following sequence:

loading phase

1. load configurations from /config folder
2. load framework related models and export it to a context(default to be global)
3. import services folder and export to the context
4. import models folder and export to the context
5. import viewModels folder and export as context.ViewModels
6. import controllers folder and export to the context

starting phase

1. connect to dependent services (like mongo, redis)
2. start the service ( default to starting an express server )

stopping phase

1. stop the service ( defaut to stopping an express server )
2. disconnect from dependent services

Router

Router is a simple component of the framework. It intends to make routing easier. It intakes a big object, parses it and call the express router to do the routing.

Most of the case, the big routing object will be the config/routes.js

middlewares

The middleware can be called before and after request. the name of the middleware will be same as its filename.

Dockerfile

The framework will provide the a Dockerfile of building a nodejs application image

View models

TBC

Getting Started with an example

installing sl-express framework

1. Install NodeJS on your machine
2. Install @shopline/sl-express on your machine npm i -g @shopline/sl-express
3. Clone this repo to your machine
4. Go into the example/basic directory on your terminal and execute npm install

starting a sample express server

1. Run sl-express start on your terminal
2. Open your browser and go to http://localhost:3000
3. You should also see a log on terminal that the route has been requested

starting a sample express console

1. Run sl-express console on your terminal and this will bring you into the express console
2. Type app.id on your express console to see the time when you start your console

starting a sample express console with async mode

1. Run sl-express asyncConsole on your terminal and this will bring you into the express console with async mode
2. Type app.id on your express console to see the time when you start your console

starting the express server with docker run

example/basic/Dockerfile has already been configured

1. Install Homebrew on your machine
2. Install docker via Homebrew brew cask install docker
3. Go into the example/basic directory on your terminal and build a docker image by docker build --tag=test-app .
4. Create a docker container by docker run -p 3000:3000 test-app
5. Open your browser and go to http://localhost:3000
6. Run ctrl+c to stop the container. You can check all the containers you have with docker ps -a
7. docker rm <CONTAINER_ID> to remove the container

starting the express server with docker-compose

example/basic/docker-compose has already been configured

1. Go into the example/basic directory on your terminal and build the images by docker-compose build
2. Create docker containers by docker-compose up
3. Open your browser and go to http://localhost:3000
4. Run ctrl+c to stop the container or you can run docker-compose stop

Use cases

customizing the app phases

There are a few phases you can customize by doing method overriding. Always remember to manage the super method

app.prepare()

This phase actually doing the loading phase of the app. most of the case, you would do:

  prepare() {
    super.prepare()
    /* your extra loading here */
  }

app.connectDependencies()

This phase handle the connection to other service like mongo, redis and rabbitmq. By default, it connects to no services. My suggestion will be:

  async connectAwesomeService() {
    /* connect... connect... connect... */
  }

  async disconnectAwesomeService() {
    /* disconnect... disconnect... disconnect... */
  }
  async connectDependencies() {
    super.connectDependencies()
    /* your other connections here */
    try { connectAwesomeService() } catch(e) { throw e }
  }

app.disconnectDependencies()

This phase handle the connection from other service like mongo, redis and rabbitmq. By default, it disconnect from no services. My suggestion will be:

  async connectAwesomeService() {
    /* connect... connect... connect... */
  }

  async disconnectAwesomeService() {
    /* disconnect... disconnect... disconnect... */
  }
  async disconnectDependencies() {
    /* the best practice is to diconnect in a reversed sequence of connections
    /* your other connections here */
    try { disconnectAwesomeService() } catch(e) { throw e }
    super.connectDependencies()

  }

app.startService()

The phase that really start the service. By default, it will start the express server. You can do a customization by condition

async startService() {

  if (/*this.role == 'SERVER'*/) {
    await this.startExpress()
    return
  }

  if (/*this.role == 'WORKER'*/) {
    /* start consuming queue */
    return
  }

}

app.stopService()

The phase to stop the service. By default, it is stopping nothing. You can also do a customization by condition

async stopService() {

  if (/*this.role == 'SERVER'*/) {
    await this.stopExpress()
    return
  }

  if (/*this.role == 'WORKER'*/) {
    /* stop consuming queue */
    return
  }

}

serving an api

routing and middlewares

To add a route, you can simply add a string to routes: []. It will split it by spaces.

The pattern should be:

HTTP_METHOD URI middleware middleware Controller.action

Sometimes you may not want to insert middleware one by one. Then you can use the preMiddlewares. Please check: https://expressjs.com/en/guide/using-middleware.html

The pattern should be:

REGEX middleware middleware

Please also reference: https://expressjs.com/en/guide/routing.html

module.exports = {

  preMiddlewares: [
    '* middleware middleware'
  ],

  routes: [
    'GET /index PublicController.index'
  ]

  postMiddlewares: [],
}

add a controller

This is how a controller should be added to the api/controllers directory.

In this example, router.js can reference the index controller by PublicController.index.

Class PublicController {

  async index(req, res) {

    return res.send('hello world')

  }

}

module.exports = PublicController

connecting to new services

creating new Service

first you will need to create a class under api/services directory.

api/services/AwesomeService.js

let _theLibYouUse = null;
let _sharedAwesomeService = null;

class AwesomeService {
    /* a lazy loading singleton. It ensures the lib would not be required if the service is not used. It may seems a bit dirty for requiring lib in functions. But it makes this service able to move into the core framework some days.
     */

    static get theLibYouUse() {
        if (!_theLibYouUse) {
            _theLibYouUse = require('theLibYouUse');
        }

        return _theLibYouUse;
    }

    /* A singleton. Most of the case you will just need to init one Service instance. You still better do a signleton pattern so that you can do stubbing easily when doing unit test on Model methods that make use of this service */

    static get sharedAwesomeService() {
        if (!_sharedAwesomeService) {
            _sharedAwesomeService = new AwesomeService();
        }

        return _sharedAwesomeService;
    }

    /* As singleton is used, it will be hard to pass the config when initializing the service. That's why we use init instead of constructor. Besides, we may not want to set the config or directly get the global config inside this class because it's better to keep it with fewer dependencies. The config should be passed to the signleton in the motherbroad */

    init(config) {
        this.endpoint = config.endpoint;
        this.abc = config.abc;
    }
}

module.exports = AwesomeService;

config/awesomeService.js

module.exports = {
    endpoint: process.env.AWESOME_SERVICE_ENPOINT
};

.env

/* all env-dependent variable should put in .env file */
AWESOME_SERVICE_ENPOINT=http://awesomeservice.com/api

app.js

  async connectAwesomeService() {

    await AwesomeService.sharedAwesomeService.init(this.config.awesomeService).connect()

  }

  async disconnectAwesomeService() {

    await AwesomeService.SharedAwesomeService.disconnect()

  }

  async connectDependencies() {

    await super.connectDependencies()
    await this.connectAwesomeService()

  }

  async disconnectDependencies() {

    await this.disconnectAwesomeService()
    await super.disconnectDependencies()

  }

config

In most of the frameworks, they like to do a structure like

- config/
  - env/
    - development.js
    - production.js
  - config1
  - config2

And these framework will first gather config1 and config2, and do a overriding with the specified environment config. Yet this framework WON'T do this.

All environment related config should be controlled by .env file

add logging

This framework use log4js wrapped in a plugin logger. Things can be configured in config/logger.js.

Please config your config/app.js

{
    plugins: ['logger'];
}

There is no magic for configuring the Logger. Please visit: https://www.npmjs.com/package/log4js

Most of the cases, you just need to add categories like 'broadcast', 'queueHandling'. It just bases on what feature you want to take log.

Besides, as we are using cloudwatch, we just append our logs to stdout at this moment.

more about logging practce

Log level

1. debug

2. trace

3. warn

4. error

5. info

6. trace: Most of the case we will add trace log every where as we should be able to investigate problems in a black-box system in production

7. warn: some error that are not exactly exceptional but you want to keep track of these kind of weird behaviour

8. error: Every exceptional should be logged with error log, no matter it breaks the process or not

9. info: System-wise log will be assigned to info log, like 'connected mongo'.

Log structure

must-have:

1. logCategory
2. logLevel
3. obj

logCategory: a category to group logs. most of the case it is designed by feature like 'broadcast', 'notificationMessage'. logLevel: like the upper section obj: a obj to be JSON.stringify. WE HIGHLY RECOMMEND YOU ADD THE FOLLOWING: 1. action (string to describe the process), 2. traceId

Use cases with built-in Model / Service

using mongoose for models

There is a built-in model called MongooseModel. This model wants to:

1. make class declaration using class instead of using prototype
2. handle the way of mixing the mongoose schema and the class by using mongoose.model
3. provide a more user friendly way to use the mongoose pre and post hook.

class AwesomeModel extends MongooseModel {
    static schema() {
        /* you can always access the mongoose library with this.mongoose */
        return {
            ownerId: { type: String, required: true }
        };
    }

    static beforeSave(obj, next) {
        //do something
        return next();
    }
}

module.exports = AwesomeModel;

connecting to mongo

Mongo should be connected when MongooseModel is used. There is a static getter function in MongooseModel and App. The one in MongooseModel will return the mongoose lib. The one in App will return the mongoose in MongooseModel. They are actually, most of the time, the same.

There are built-in function for connection mongo. What you need to do is adding ENV to your .env file. Basically, we have a config/mongoose.js in the framework that mapping a mongo endpoint to ENVs so you just need to add ENV.

MONGODB_USER
MONGODB_PASS
MONGODB_HOST
MONGODB_PORT
MONGODB_DATABASE

Please config your config/app.js

{
    plugins: ['mongoose'];
}

Add the following to your docker-compose.yml

version: '3'

services:
    # your app build
    # ...

    mongo:
        image: 'mongo'
        ports:
            - '27017:27017' # configure your port
        volumes:
            - 'mongodb:/data/db'
    # ...
    # your other services (rabbitmq, redis)

volumes:
    mongodb:
        driver: local

using redis

By default we have a config file in framework mapping ENVs to the redis config

REDIS_USER
REDIS_PASS
REDIS_HOST
REDIS_PORT
REDIS_DATABASE
REDIS_TIMEOUT_MS

Please config your config/app.js

{
    plugins: ['redis'];
}

Add the following to your docker-compose.yml

version: '3'

services:
    # your app build
    # ...

    redis:
        image: 'redis'
        ports:
            - '6379:6379' # configure your port

    # ...
    # your other services (rabbitmq, mongo)

using messageQueue

By default we have a config file in framework mapping ENVs to the redis config

RABBITMQ_USER
RABBITMQ_PASS
RABBITMQ_HOST
RABBITMQ_PORT
RABBITMQ_PREFETCH_COUNT
RABBITMQ_QUEUE_PREFIX

Please config your config/app.js

{
    plugins: ['messageQueue'];
}

Add the following to your docker-compose.yml

version: '3'

services:
    # your app build
    # ...

    rabbitmq:
        image: 'rabbitmq:3-management'
        ports:
            - '5672:5672' # configure your port
            - '15672:15672'
    # ...
    # your other services (redis, mongo)

using QueueTask

You need to connect to both Redis and Rabbitmq for this feature. By default, we will store the payload of a task to Redis and only send the task id to the rabbitmq. This design will avoid sending too large payload to the rabbitmq. QueueTask, as a model, will handle all this for you.

Please refer to using redis section. Please refer to using messageQueue section.

To get it set up, you need to add the following code:

Add queueTask plugin to config/app.js

{
    plugins: ['queueTask', '...your other plugins'];
}

Add a config file config/queueTask.js

module.exports = [
    {
        type: 'TEST', // an identifier for you task
        queue: 'test_queue', // the CONSUMER_QUEUE_ID or consumerQueueId to handle the queue
        messageExpireTimeSec: 3600 * 24 // set a timeout on message, default to (3600 * 24)s. After the timeout has expired, the message will automatically be discarded.
        handler: 'Test.dequeue', // the handler of the tasks of this type
        description: 'any remarks you want to add'
    }
];

Add a Test.js file on api/models/Test.js to queue and dequeue handling

class Test {
    static async enqueue(test) {
        //do something to make a payload
        let payload = {
            firstName: test.firstName,
            sex: test.sex
        };

        await QueueTask.queue({
            taskType: 'TEST',
            payload: payload
        });
    }

    static async dequeue(queueTask) {
        let payload = queueTask.payload;

        // handle the payload
        console.log(payload.firstName);
    }
}

module.exports = Test;

To test the queueTask feature, you will have to start your app with a consumer and publisher role. To do so, you can use docker-compose to instantiate two containers of the app, one with a consumer role and the other with a publisher role.

Modify your docker-compose.yml

services:
    test-app-publisher:
        build: .
        ports:
            - '3000:3000'
        volumes:
            - .:/app
            - /app/node_modules
        environment:
            - APP_ROLE=PUBLISHER # to start the container using a publisher role
        command: bash -c "chmod +x ./wait-for-it.sh && ./wait-for-it.sh rabbitmq:5672 -- nodemon server.js"

    test-app-consumer:
        build: .
        volumes:
            - .:/app
            - /app/node_modules
        environment:
            - APP_ROLE=CONSUMER # to start the container using a consumer role
            - CONSUMER_QUEUE_ID=test_queue # set the id the same as the queue attr in config/queueTask.js
        command: bash -c "chmod +x ./wait-for-it.sh && ./wait-for-it.sh rabbitmq:5672 -- nodemon server.js"
    #
    #
    # your other services (redis, mongo, rabbitmq)

using Plugin

You can build any plugin you like using Plugin feature. SL-expres will

1. read the app.config.plugins
2. read /plugins of YOUR application folder and import the plugin ONLY the key exists in the config
3. if there are some keys in the config cannot be imported, it try to import them from sl-express. (overriding the default)

// config/app.js

module.exports = {
  plugins: [
    'helloWorld',
    'drinkTea',
  ]
}

the plugin must fulfill the directory structure

// plugins
- helloWorld
  - index.js
- drinkTea
  - index.js;

the export of the index.js must provide the following interfaces

1. prepare(app) { }
2. async connectDependencies(app) { }
3. async disconnectDependencies(app) { }
4. async willStartService(app) { }
5. async didStartService(app) { }

These interfaces are related to specific App phases. check the class App for details

app means the App instance. You can get properties through this app instance. Most of the cases, you will need the app.config

in more advanced usages

// plugins/
- helloWorld
  - lib/
    - ModelA.js
    - ModelB.js
    - HelloWorldService.js
    - HelloWorldPlugin.js
  - index.js
  - README.md

// index.js

const HelloWorldPlugin = require('./HelloWorldPlugin')
module.exports = new HelloWorldPlugin()

// HelloWorldPlugin.js

class HelloWorldPlugin {
  prepare(app) {
    const service = new HelloWorldService()
  }

  async connectDependencies(app) { }
  async disconnectDependencies(app) { }
  async willStartService(app) { }
  async didStartService(app) { }
}

module.exports = HelloWorldPlugin

Layers of SL-express application

We can organise our code structure will five kinds of components:

1. Controller: its actions to receive API calls
2. AppService: services that provided by our sl-express application to complete use cases by making use of models.
3. Model: to control the queries, data structure, data change of itself
4. Service: services that are intaken into the application.
5. Plugin: a structure that compose of 2,3,4. it can abstruct the whole service you want to provide. It also conform the interface to integrate with the sl-express app

###More about plugin In a simple way, it's a connector between the app and the library / modules you write.

Sometimes, in your application, you may need to solve some use cases that involve a complicated logic flow. it may involve many models and intaken services which only concern this part of this logic but not the app. In this case, you would like to "hide" these models and services and wrap it into a modules. And you may also need to setup a bit before pluging it into the application. Using the integration feature of the plugin layer, your modules can simply focus on the logic.