Issue 254 | Gateway service

[IceKhan13]

Summary

Gateway service initial works.
Service for managing programs execution, setting limits, etc.

TODOs:

• POC gateway
• client integration
• UML diagram
• tests
• docs
• linter
• update tutorials
• commands to create user, compute resource, site, etc.
• dockerfile
• CI tests
• job results

Details and comments

from quantum_serverless import QuantumServerless, Program, GatewayProvider, Provider
from quantum_serverless.core import ComputeResource

gateway_provider = GatewayProvider(
    username="john",
    password="password123",
    # token="<TOKEN>"  # alternatevely can provide token instead of user/password
    host="http://localhost:8000",
)

serverless = QuantumServerless([gateway_provider])

program = Program(
    name="awesome_program",
    entrypoint="program.py",
    working_dir="./",
    description="description of a program",
    version="0.0.1"
)

job = serverless.run_program(program)

job.logs()
job.result()

Requests flow

General overview

[IceKhan13]

Number of things is still need to be polished and finished, but it will be separate PRs. Majority of work is done here.

Remaining things:

• CI for nightly builds
• more tests coverage
• improve code quality (serialization, verification, ids of models in DB)
• helm chart

[Tansito]

This job is awesome @IceKhan13 . I couldn't review correctly all the code but I started with the django part. I left some comments with doubts and another ones to discuss. Something in what we can work too is to sync a little bit the projects from repository and this one.

[akihikokuroda]

This COPY overwrites the edited entrypoint.sh, doesn't it?

[IceKhan13]

good point!

[psschwei]

Just to clarify, when creating the gateway provider, the host (host="http://localhost:8000") is the repository service? (that's the only one I see on k8s using port 8000)

[IceKhan13]

Just to clarify, when creating the gateway provider, the host (host="http://localhost:8000") is the repository service? (that's the only one I see on k8s using port 8000)

No, it is gateway endpoint. And gateway is basically program runner middleware and that is it. Repository service will be living in some other host/port. At the end of the day it is just a matter of configuration.

[psschwei]

Whoops, I should've looked closer at the PR and not tried using the helm install here... will pop back over to docker-compose to poke around.

[IceKhan13]

@psschwei btw, docker compose will be updated with #301 to latest Keycloak image

[pacomf]

@psschwei the idea for the near future is use the gateway like entrypoint (to connect with repository, keycloak, etc), so we will have just one interface (API Gateway) to connect from Middleware client

[akihikokuroda]

When is the Job in the gateway deleted? I haven't looked though changes yet so I might miss it.

[IceKhan13]

it remains there for references, until user decides to delete it @akihikokuroda . We can put ttl or something like this later on

[Tansito]

As we talked this is a good first approach. From my part is approved although it will require some improvements that we will cover in future issues / pull requests:

• Sync repository and gateway applications to use same language #307
• Implement versioning for the end-points #308
• Implement validations for NestedProgram fields #310
• Improve Dockerfile configuration for gateway #311
• Verify dj-rest configuration and usage #312
• Add gateway image build into CI #313
• Migrate NestedProgram fields to JSONFields #314
• Configure API projects to use Postgres #315
• Improve test coverage for Gateway end-points #316
• Improve helm configuration introduced with gateway #318

I will create them tomorrow to start handle them.

[pacomf]

LGTM!

[akihikokuroda]

Is "0.1.0" better?

[akihikokuroda]

Is serverless-cluster your helm install release name? Would you replace it with HELM-RELEASE

[akihikokuroda]

Is this used?

[akihikokuroda]

LGTM. Great work! There are some minor comments but I approve this.