Spike: Discover and define models at runtime #4235

bajtos · 2019-12-02T14:16:19Z

This pull request implements a controller allowing clients to define models & expose their REST APIs at runtime, with no model/repository/controller source files required.

Cross-posting from #2484:

Accept a MySQL connection string (host+port+database+credentials) and a list of table names.
Connect to the specified MySQL database.
Discover model schemas from the given tables
Define LB4 models for the given tables
Expose these models via REST API
Return a list of URL paths of the newly added models (e.g. /Products, /CoffeeShops)

We can use the public database from loopback-getting-started) when testing this new endpoint manually.

Please note that public database is no longer available. What I did instead:

Setup a local MySQL server (e.g. via Docker)
Run the following SQL script to create Coffeeshop table and seed it with some records: https://github.com/strongloop/loopback-getting-started/blob/master/bin/demo.sql

Try it out

Setup the MySQL server, create some tables
Checkout the feature branch spike/import-model-at-runtime
npm install && npm build
Open API Explorer - http://127.0.0.1:3000/explorer/#/ModelAdminController/ModelAdminController.discoverAndPublishModels

Post a valid MySQL connection string and a list of existing tables, for example:

{
  "connectionString": "mysql://root@localhost/test",
  "tableNames": [
    "Coffeeshop"
  ]
}

Reload the API Explorer to see the new endpoints for your discovered model(s)

Status

Proof of concept (implementation)
Spike document, including a list of GitHub issues - follow-up user stories covering the implementation.

raymondfeng · 2019-12-03T00:06:39Z

@bajtos Thank you for kicking off the topic.

I have been thinking about leveraging Kubernetes's Operator and Custom Resource Definition pattern/formats to achieve similar goals recently.

Each artifact will be described as CRD
Allow extensions for artifact controllers/operators, for example, we can add LB4 datasource/connector operators

See:

Clarification:

I'm NOT proposing to require Kubernetes. The intention is to use the similar design: CRD + Operator to extend LB4 for such resources.

We can have an extension point for Operators, each of them can handle a specific kind/version of CRDs.

raymondfeng · 2019-12-04T18:16:51Z

packages/rest/src/rest.application.ts

@@ -133,6 +133,11 @@ export class RestApplication extends Application implements HttpServerLike {
    this.restServer.basePath(path);
  }

+  controller<T>(controllerCtor: ControllerClass<T>, name?: string): Binding {
+    this.restServer.invalidateRoutingCache();


I would rather to use a ContextView to invalidate the routing cache.

+1, I would like to see a ContextView-based solution too. In this spike, I used the fastest path to get done what I need.

@raymondfeng Do you have bandwidth to implement ContextView-based version yourself? Here is the GitHub issue tracking this user story: #433

examples/todo/src/controllers/model-admin.controller.ts

bajtos · 2019-12-05T09:56:12Z

examples/todo/src/controllers/model-admin.controller.ts

+  modelClass: typeof Entity,
+  options: CrudRestControllerOptions,
+) {
+  const RepositoryClass = defineCrudRepositoryClass(modelClass);


I am thinking about moving defineCrudRepositoryClass to @loopback/repository. I see the function as very coupled to DefaultCrudRepository class, not adding anything opinionated or controversial on top of the named constructor.

examples/todo/src/controllers/model-admin.controller.ts

Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

A short-term workaround for the missing feature tracked by #433 Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

- Accept a MySQL connection string (host+port+database+credentials) and a list of table names. - Connect to the specified MySQL database. - Discover model schemas from the given tables - Define LB4 models for the given tables - Expose these models via REST API - Return a list of URL paths of the newly added models Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

bajtos · 2019-12-09T07:54:58Z

In a878d83, I have simplified the demo controller implementation using the recently released features loopbackio/loopback-datasource-juggler#1807 and #4266. I have also reorganized the code to make it more clear how a model can stay private (not exposed via REST API).

I consider the demo app as done and proposing the following follow-up stories:

Dynamic binding/rebinding of controllers after app start Dynamic binding/rebinding of controllers after app start #433 (already exists)
How to define models, repositories and controllers at runtime

The idea is create a new documentation page that will explain how to:
1. Define a model class from a ModelDefinition object
2. Define a repository class for the given model, how to configure dependencies (the datasource to use)
3. Define a controller class for the given model & repository, how to configure dependencies (the repository to use)
4. Discover model definition from a database, build ModelDefinition object from the discovered schema.
The page can use code snippets from my demo controller to illustrate the approach in practice.

In my demo, I have a comment mentioning that it would be nice to have a LB4-native method for discovering model schemas, a method that return LB4 ModelDefinition object instead of LB3-like data. Considering that we don't know how many people will actually implement model discovery at runtime, I am proposing to leave this feature out of scope of the initial work.

@strongloop/sq-lb-apex Please take a look and let me know if you have any comments or suggestions. I'd like to get at least one approval and then I'll move on to create the follow-up story/stories.

agnes512

👏 It's really great that I can use endpoints with just a datasource config JSON!

I only have one suggestion. If I were a user who uses this feature a lot, I'd like to know how I can modify/config the controller to match my requirements. It'd be good if we can make this part of documentation clear/detailed in the follow-up story.

jannyHou · 2019-12-09T16:17:13Z

packages/rest/src/rest.server.ts

@@ -703,6 +713,9 @@ export class RestServer extends Context implements Server, HttpServerLike {
    let spec = this.getSync<OpenApiSpec>(RestBindings.API_SPEC);
    const defs = this.httpHandler.getApiDefinitions();

+    // Apply shallow-clone to prevent modification of user-provided API_SPEC


@bajtos could you explain more about the shallow clone here?
Wouldn't it ALLOW modification of user-provided API_SPEC?

The line below it(L#706) applies deep clone which prevents the modification of the original spec.

This is just a temporary workaround to get my demo working. The actual implementation is likely to be different, see the discussion in Dynamic binding/rebinding of controllers after app start #433.

jannyHou

Tested the /discover endpoint with a working url, everything is good 👍

(maybe another story)From UX's perspective...shall we create another endpoint that prints out the generated model def, like GET /showModelDefByName/{modelname}.
When testing the dynamically generated APIs, people can learn the signature of endpoints from /Todos, but It's not very easy to figure out the model properties by only having the path of the model.

jannyHou · 2019-12-09T17:53:04Z

examples/todo/src/controllers/model-admin.controller.ts

+    console.log('Defined repository %s', RepositoryClass.name);
+
+    // Step 4: Optionally, expose the new model via REST API
+    const basePath = '/' + modelDef.name;


nitpick: to be consistent with Todo model, maybe use the plural of the name as base path?

Sure, plural names would be better. There are different ways how to build plurals, I am leaving it up to our users to decide.

bajtos · 2019-12-10T07:19:24Z

Thank you @jannyHou and @agnes512 for thoughtful comments!

I only have one suggestion. If I were a user who uses this feature a lot, I'd like to know how I can modify/config the controller to match my requirements. It'd be good if we can make this part of documentation clear/detailed in the follow-up story.

Good idea 👍

In my mind, this is related to the epic From model definition to REST API #2036, where we already have a story to document how can extension authors build custom controller factories - see #2740.

Let's include #2740 in the follow-up stories for this spike.

(maybe another story)From UX's perspective...shall we create another endpoint that prints out the generated model def, like GET /showModelDefByName/{modelname}.
When testing the dynamically generated APIs, people can learn the signature of endpoints from /Todos, but It's not very easy to figure out the model properties by only having the path of the model.

I see your point. At the moment, you can find model properties in OpenAPI schema.

It is possible to build an endpoint you have described, or perhaps add the model definition to the response returned by POST /discover endpoint. I feel those changes are out of scope of this spike/demo app though. The purpose of this work is to discover missing building blocks and ensure everything fits together.

IMO, real world applications will never follow exactly our demo version, they will always combine these blocks in unique ways.

bajtos · 2019-12-12T13:34:35Z

How to define models, repositories and controllers at runtime

Created #4296

bajtos · 2019-12-12T13:37:04Z

Follow-up stories:

Dynamic binding/rebinding of controllers after app start Dynamic binding/rebinding of controllers after app start #433.
How to build models, repositories and controllers dynamically at runtime How to build models, repositories and controllers dynamically at runtime #4296
Docs for exposing REST API of a Model with no custom classes Docs for exposing REST API of a Model with no custom classes #2740

I am closing this spike as done.

bajtos added the spike label Dec 2, 2019

bajtos added this to the Dec 2019 milestone Dec 2, 2019

bajtos requested review from raymondfeng and a team December 2, 2019 14:16

bajtos self-assigned this Dec 2, 2019

bajtos mentioned this pull request Dec 2, 2019

Spike: Discover and define models at runtime #2484

Closed

4 tasks

raymondfeng reviewed Dec 4, 2019

View reviewed changes

bajtos mentioned this pull request Dec 5, 2019

Fix Promise/Callback variants in datasource types loopbackio/loopback-datasource-juggler#1807

Merged

3 tasks

bajtos commented Dec 5, 2019

View reviewed changes

examples/todo/src/controllers/model-admin.controller.ts Outdated Show resolved Hide resolved

bajtos commented Dec 5, 2019

View reviewed changes

examples/todo/src/controllers/model-admin.controller.ts Outdated Show resolved Hide resolved

bajtos commented Dec 5, 2019

View reviewed changes

bajtos mentioned this pull request Dec 6, 2019

repository: Add defineModelClass helper and fix DeepPartial<AnyObject> #4266

Merged

4 tasks

bajtos commented Dec 6, 2019

View reviewed changes

examples/todo/src/controllers/model-admin.controller.ts Outdated Show resolved Hide resolved

bajtos added 4 commits December 9, 2019 07:57

feat: add loopback-connector-mysql

38d4601

Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

feat(rest): support controllers added after start

6ce2228

A short-term workaround for the missing feature tracked by #433 Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

fixup! simplify demo implementation

a878d83

Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

bajtos force-pushed the spike/import-model-at-runtime branch from 5f43345 to a878d83 Compare December 9, 2019 07:45

bajtos changed the title ~~Spike: Discover and define models at runtime [WIP]~~ Spike: Discover and define models at runtime Dec 9, 2019

This was referenced Dec 9, 2019

Dynamic binding/rebinding of controllers after app start #433

Closed

Feature Request - API for dynamic model creation #1580

Closed

agnes512 approved these changes Dec 9, 2019

View reviewed changes

jannyHou reviewed Dec 9, 2019

View reviewed changes

bajtos mentioned this pull request Dec 12, 2019

How to build models, repositories and controllers dynamically at runtime #4296

Closed

7 tasks

bajtos closed this Dec 12, 2019

bajtos mentioned this pull request Dec 12, 2019

feat(docs): add 2020Q1 roadmap #4116

Merged

7 tasks

raymondfeng mentioned this pull request Mar 5, 2020

feat: move functions to define repository classes from @loopback/rest-crud to @loopback/repository #4792

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spike: Discover and define models at runtime #4235

Spike: Discover and define models at runtime #4235

bajtos commented Dec 2, 2019 •

edited

Loading

raymondfeng commented Dec 3, 2019 •

edited

Loading

raymondfeng Dec 4, 2019

bajtos Dec 5, 2019

bajtos Dec 5, 2019

bajtos commented Dec 9, 2019

agnes512 left a comment

jannyHou Dec 9, 2019

bajtos Dec 10, 2019

jannyHou left a comment

jannyHou Dec 9, 2019 •

edited

Loading

bajtos Dec 10, 2019

bajtos commented Dec 10, 2019

bajtos commented Dec 12, 2019

bajtos commented Dec 12, 2019

Spike: Discover and define models at runtime #4235

Spike: Discover and define models at runtime #4235

Conversation

bajtos commented Dec 2, 2019 • edited Loading

Try it out

Status

raymondfeng commented Dec 3, 2019 • edited Loading

raymondfeng Dec 4, 2019

Choose a reason for hiding this comment

bajtos Dec 5, 2019

Choose a reason for hiding this comment

bajtos Dec 5, 2019

Choose a reason for hiding this comment

bajtos commented Dec 9, 2019

agnes512 left a comment

Choose a reason for hiding this comment

jannyHou Dec 9, 2019

Choose a reason for hiding this comment

bajtos Dec 10, 2019

Choose a reason for hiding this comment

jannyHou left a comment

Choose a reason for hiding this comment

jannyHou Dec 9, 2019 • edited Loading

Choose a reason for hiding this comment

bajtos Dec 10, 2019

Choose a reason for hiding this comment

bajtos commented Dec 10, 2019

bajtos commented Dec 12, 2019

bajtos commented Dec 12, 2019

bajtos commented Dec 2, 2019 •

edited

Loading

raymondfeng commented Dec 3, 2019 •

edited

Loading

jannyHou Dec 9, 2019 •

edited

Loading