Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for command line linker, syntax highligher for GTO #128

Merged
merged 6 commits into from
Nov 17, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions packages/example/content/docs/gto/command-reference/annotate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# annotate

Update artifact metadata annotations.

## Synopsis

```usage
usage: gto annotate [-r <text>] [--type <text>] [--path <text>]
[-e] [--label <text>] [-d <text>] [--commit]
[--push] [-h]
name

arguments:
name Artifact name
```

## Description

The command adds the artifact annotation to `artifacts.yaml` file. Feel free to
modify the file directly!

For example,

```cli
$ gto annotate awesome-model \
--type model \
--path models/neural_network.h5 \
--label ml
--label cool
--description "This model is very cool"
```

will create

```yaml
awesome-model:
description: This model is very cool
labels:
- ml
- cool
path: models/neural_network.h5
type: model
```

This information can be later retrieved by running `gto describe` command.

> Don't forget to commit `artifacts.yaml` with Git to associate it with the
> latest artifact version and stage in any copy of the repo.

By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
GTO the artifact file is committed to Git.

<details>

### Virtual vs. Physical artifacts

- Physical files/directories are committed to the repo. When you create a new
version or assign a stage to it, Git guarantees that it's immutable -- you can
return a year later and get the same artifact by providing a version.

- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
local path to a metafile representing an externally stored artifact file (as
[with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
can't pin versions to a physical state of the artifact and guarantee its
immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
know it, nor have a way to recover the original file.

> In future versions, we will support additional enrichments: useful information
> that other tools like [DVC](https://dvc.org/) and [MLEM](https://mlem.ai/) can
> provide about the artifacts. This will allow treating DVC repo outputs as
> usual artifacts instead of `virtual` ones.

</details>

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--type <text>` - Artifact type
- `--path <text>` - Artifact path
- `-e`, `--must-exist` - Verify artifact is committed to Git
- `--label <text>` - Labels to add to artifact
- `-d <text>`, `--description <text>` - Artifact description
- `--commit` - Automatically commit changes due to this command (experimental)
- `--push` - Push created commit automatically (experimental) - will set
commit=True
- `-h`, `--help` - Show this message and exit.
61 changes: 61 additions & 0 deletions packages/example/content/docs/gto/command-reference/assign.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# assign

Assign stage to specific artifact version.

## Synopsis

```usage
usage: gto assign [-r <text>] [--version <text>]
[--stage <text>] [-m <text>]
[--simple <text>] [--force] [--push] [--sr]
[-h]
name [ref]

arguments:
name Artifact name
[ref] Git reference to use
```

## Description

To assign an actionable stage for a specific artifact version use the same
`gto assign` command. Stages can mark the artifact readiness for a specific
consumer. You can plug in a real downsteam system via CI/CD or web hooks, e.g.
to redeploy an ML model.

```cli
$ gto assign awesome-model --version v0.0.1 --stage prod
Created git tag 'awesome-model#prod#1' that assigns stage to 'v0.0.1'
```

GTO creates a special Git tag in
[the standard format](/doc/gto/user-guide/git-tags).

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--version <text>` - If you provide REF, this will be used to name new version
- `--stage <text>` - Stage to assign
- `-m <text>`, `--message <text>` - Message to annotate the Git tag with
- `--simple <text>` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
[supported values: auto, true, false] [default: auto]
- `--force` - Create the Git tag even if it already exists and is in effect
- `--push` - Push created tag automatically (experimental)
- `--sr`, `--skip-registration` - Don't register a version at specified commit
- `-h`, `--help` - Show this message and exit.

## Examples

Assign artifact "nn" to "prod" at specific Git ref instead of supplying artifact
version (note that this will also register a version if it doesn't exist):

```cli
$ gto assign nn abcd123 --stage prod
```

Assign stage at specific Git ref and name the version explicitly (this assumes
that version was not registered yet):

```cli
$ gto assign nn abcd123 --version v1.0.0 --stage prod
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# check-ref

Find out the artifact version registered/assigned with ref.

## Synopsis

```usage
usage: gto check-ref [-r <text>] [--json] [--name] [--version]
[--event] [--stage] [-h]
ref

arguments:
ref Git reference to analyze
```

## Description

You can use `gto check-ref` to interpret a Git tag:

```cli
$ gto check-ref -r build/example-gto churn#prod#3
✅ Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
```

For machine-consumable format, use `--json` flag or output specific pieces of
information with `--name`, `--version`, `--stage` or `--event`.

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--json` - Print output in json format
- `--name` - Show artifact name
- `--version` - Output artifact version
- `--event` - Show event
- `--stage` - Show artifact stage
- `-h`, `--help` - Show this message and exit.
141 changes: 141 additions & 0 deletions packages/example/content/docs/gto/command-reference/deprecate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# deprecate

Deprecate artifact, deregister a version, or unassign a stage.

## Synopsis

```usage
usage: gto deprecate [-r <text>] [-m <text>] [--simple <text>]
[--force] [-d] [--push] [-h]
name [version] [stage]

arguments:
name Artifact name
[version] Artifact version
[stage] Stage to unassign
```

## Description

The command supports three use cases:

```cli
# deprecate an artifact:
$ gto deprecate nn

# deprecate a version:
$ gto deprecate nn v0.0.1

# unassign a stage:
$ gto deprecate nn v0.0.1 prod
```

### Unassigning a stage

Sometimes you need to mark an artifact version no longer ready for a specific
consumer, and maybe signal a downstream system about this. You can use
`gto deprecate` for that:

```cli
$ gto deprecate awesome-model v0.0.1 prod
Created git tag 'awesome-model#prod!#2' that unassigns a stage from 'v0.0.1'
```

<details>

### Unassigning a stage: some details and options

GTO creates a special Git tag in
[the standard format](/doc/gto/user-guide/git-tags).

Note, that later you can create this stage again, if you need to, by calling
`$ gto assign` again.

You also may want to delete the git tag instead of creating a new one. This is
useful if you don't want to keep extra tags in you Git repo, don't need history
and don't want to trigger a CI/CD or another downstream system. For that, you
can use:

```cli
$ gto deprecate awesome-model v0.0.1 prod --delete
Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'
To push the changes upstream, run:
git push origin awesome-model#prod#1 --delete
```

</details>

### Deregister a version

Sometimes you need mark a specific artifact version as a no longer ready for
usage. You could just delete a git tag, but if you want to preserve a history of
the actions, you may again use `gto deprecate`.

```cli
$ gto deprecate awesome-model v0.0.1
Created git tag 'awesome-model@v0.0.1!' that deregistered a version.
```

<details>

### Deregister a version: some details and options

If you want to deregister the version by deleting the Git tags itself, you could
use

```cli
$ gto deprecate awesome-model v0.0.1 --delete
Deleted git tag 'awesome-model@v0.0.1' that registered a version.
Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
To push the changes upstream, run:
git push origin awesome-model@v0.0.1 awesome-model#prod#1 awesome-model#prod!#2 --delete
```

This includes all Git tags related to the version: a tag that registered it and
all tags that assigned stages to it.

</details>

### Deprecating an artifact

Sometimes you need to need to mark the artifact as "deprecated", usually meaning
it's outdated and will no longer be developed. To do this, you could run:

```cli
$ gto deprecate awesome-model
Created Git tag 'awesome-model@deprecated' that deprecates an artifact.
```

<details>

### Deprecating an artifact: some details and options

With `awesome-model@deprecated` Git tag the artifact will be considered
deprecated until you register a new version or assign a new stage to it after
the deprecation.

If you want to deprecate an artifact by deleting git tags, you'll need to delete
all of them for the artifact. You could do that with

```cli
$ gto deprecate awesome-model --delete
Deleted git tag 'awesome-model@v0.0.1' that registered a version.
Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
To push the changes upstream, run:
git push origin awesome-model@v0.0.1 awesome-model#prod#1 awesome-model#prod!#2 --delete
```

</details>

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `-m <text>`, `--message <text>` - Message to annotate the Git tag with
- `--simple <text>` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
[supported values: auto, true, false] [default: auto]
- `--force` - Create the Git tag even if it already exists and is in effect
- `-d`, `--delete` - Delete the git tag(s) instead of creating the new one
- `--push` - Push created tag automatically (experimental)
- `-h`, `--help` - Show this message and exit.
38 changes: 38 additions & 0 deletions packages/example/content/docs/gto/command-reference/describe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# describe

Display enrichments for an artifact.

## Synopsis

```usage
usage: gto describe [-r <text>] [--rev <text>] [--type] [--path]
[--description] [-h]
name

arguments:
name Artifact name
```

## Description

To get details about an artifact (from `artifacts.yaml`) use `gto describe`:

```cli
$ gto describe churn -r https://github.com/iterative/example-gto
{
"type": "model",
"path": "models/churn.pkl",
"virtual": false
}
```

The output is in JSON format for ease of parsing programatically.

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--rev <text>` - Repo revision to use
- `--type` - Show type
- `--path` - Show path
- `--description` - Show description
- `-h`, `--help` - Show this message and exit.
Loading