Strawman CLI Overhaul

[saterus]

This is partially a braindump from what we discussed at the offsite, and partially some extrapolation I'm making up. It's mostly here as a strawman for us to prompt discussion. Hopefully it gives us an idea of breadth as well a place to dive into depth.

Terminology

I'm inventing specific terms here to try and be clear. I'm not attached to these, so they can die here too.

• newcli: The placeholder binary name, used here to distinguish it from today's flowctl command.
• Model: Top-level, independent API resource managed by the user. May generate zero or more Specs during a flow-build.
• Catalog Model: The primary concepts within the system:
  • Collection
  • Capture
  • Materialization
  • Derivation
  • StorageMapping
• API Model: The secondary concepts necessary to facilitate running the service, but not currently a first-class part of Flow's data model. These may become Catalog Models over time, or they may be consumed by Flow in more primitive forms.
  • Secret
  • Tenant/Organization
  • User
  • AccessGrant
• MODEL_PATH: / delimited identifier for a single Model or a group of Models

CLI (thin client)

Daily Driver Commands

newcli build source.flow.yaml

Kicks off a catalog build within the control plane.

Requires a source file which is valid .flow.yaml containing one or more Catalog Models. Any content-addressable resources referenced will be fetched immediately. Any name-addressable resources will be provided by the Control Plane during the build.

1. Load source files
2. Resolve references
3. Create amalgamated catalog.yaml from the source and resolved references.
4. POST catalog.yaml to /builds
5. [Optional] Wait for build to finish (Q: --wait vs --no-wait?)

newcli list MODEL_PATH

Shows brief metadata about all the Models matching the provided path.

Output: MODEL_PATH, Model type, build_id?, status

Ex.

• newcli list /acmeCo/build/227
• newcli list /acmeCo/accounting
• newcli list /acmeCo/accounting/sales_*

newcli get MODEL_PATH

Show metadata information about a specific catalog Model.

Output: MODEL_PATH, Model type, build_id?, source_yaml, compiled_spec?, status

Ex.

• newcli get /acmeCo/build/227 # a Build
• newcli get /acmeCo/accounting/import_sales_figures # a Capture
• newcli get /acmeCo/accounting/sales_figures # a Collection
• newcli get /acmeCo/accounting/sale/0xabcd # a Schema

newcli inspect MODEL_PATH

Show the detailed contents of a catalog Model.

Ex.

• newcli inspect /acmeCo/accounting/sales_figures
  • => shows a jsonl preview of the data inside the collection
• newcli inspect /acmeCo/accounting/sale/0xabcd
  • => shows the full, resolved json schema contents

newcli logs MODEL_PATH

Reads the Flow logs emitted by a particular Model.

Ex.

• newcli logs /acmeCo/builds/227
• newcli logs --tail capture /acmeCo/accounting/sales_figures

Less Frequent Commands

newcli shards

Manages sharding of catalog tasks.

Ex.

• newcli shards split MODEL_PATH

newcli access

Manages access controls to a given Model path.

• Actor: Users & Models represented as an MODEL_PATH
• Role: Named sets of privileges given to an Actor that encapsulate one or more actions. TBD.

Ex.

• newcli access grant /acmeCo/alex admin /acmeCo/
• newcli access modify /acmeCo/alex collaborator /acmeCo/
• newcli access revoke ACTOR ROLE MODEL_PATH

Administration Commands

newcli services

Launch control plane + data plane components for local development.

Pass an optional specific component of the system to scope the command.

• Components: etcd, gazette_brokers, flow_consumers, gateway, control_plane

Ex.

• newcli services start [component]
• newcli services restart [component]
• newcli services stop [component]
• newcli services status [component]
• newcli services logs component
• newcli services install [component]
• newcli services update [component]

newcli context

Uses a configuration file to manage which control plane the CLI is talking to. Defaults to cloud.estuary.io.

Ex.

• newcli context add NAME CONTROL_PLANE_URL
• newcli context remove NAME
• newcli context current

newcli auth

Associates actions taken by this CLI with a specific user

Ex.

• newcli auth login
• newcli auth logout
• newcli auth whoami

Today's flowctl Commands

• apply: part of build
• check: ?
• develop: part of services
• discover: replaced by UI?
• journals: ?
• json-schema: ?
• print-config: ?
• serve: part of services
• shards: included
• test: part of build

Questions that I had while working on this:

• Naming / MODEL_PATHs:
  • Should there be a root prefix under which all tenant models are anchored?
    • We already have /ops and /recovery as "special" toplevel names that cannot be used a tenant names.
  • Should there be a distinction in naming/terminology between namespaces /acmeCo/accounting/ and singularly identified/resolved Models /acmeCo/account/sales_figures?
  • Should all Models have a path_name, eg. /acmeCo/accounting/sales_figures
    • Collections and Tasks have names currently
  • How should implicitly named API Models be included in the tenant-prefixed path scheme?
    • /acmeCo/builds/277 or
    • /builds/acmeCo/277 or something else?
  • How to reference content-addressed Models as an MODEL_PATH? (schemas, js, etc.)
• Is there a distinction between supplied schemas and inferred schemas that's useful to present in the API?