Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jan/copy 2 #47

Merged
merged 3 commits into from
Sep 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 26 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,23 +225,32 @@ Actors' results.

The Actor results are typically fully available only after the Actor run finishes,
but the consumers of the results might want to access partial results during the run.
Therefore, the Actors don't generate the output object directly, but rather
Therefore, Actors don't generate the output object in their code, but they
define an [Output schema file](#output-schema-file), which contains
instruction how to generate such output object automatically.

You can define how the Actor output looks like using the [Output schema file](#output-schema-file).
The system uses this information to automatically generate an immutable JSON file,
which tells users where to find the results produced by the Actor.
The output object is stored by the system
to the Actor run object under the `output` property, and returned via API immediately when
the Actor is started, without the need to wait for it to finish or generate the actual results.
This is useful to automatically generate UI previews of the results, API examples,
and integrations.

The output object is similar to input object, as it contains properties and values.
For example, for the `bob/screenshot-taker` Actor the output object can look like this:
For example, for the `bob/screenshot-taker` Actor, the output object can look like this:

```json
{
"screenshotUrl": "https://api.apify.com/key-value-stores/FkspwWd8dFknjkxx/screenshot.png"
"screenshotUrl": "https://api.apify.com/v2/key-value-stores/skgGkFLQpax59AsFD/records/screenshot.jpg",
"productImages": "https://api.apify.com/v2/key-value-stores/skgGkFLQpax59AsFD/records/product*.jpg",
"productDetails": "https://api.apify.com/datasets/9dFknjkxxGkspwWd/records?fields=url,name",
"productExplorer": "https://bob--screenshot.apify.actor/product-explorer",
// or this with live view
"productExplorer": "https://13413434.runs.apify.net/product-explorer"
}

```

### Storage
Expand Down Expand Up @@ -1563,9 +1572,21 @@ This is an example of the output schema file for the `bob/screenshot-taker` Acto
"title": "Output schema for Screenshot Taker Actor",
"description": "The URL to the resulting screenshot",
"properties": {

"currentProducts": {
"type": "$defaultDataset",
"views": ["productVariants"]
},

"screenshotUrl": {
"type": "$defaultKeyValueStore.file",
"title": "Webpage screenshot"
"type": "$defaultKeyValueStore",
"keys": ["screenshot.png"],
"title": "Product page screenshot"
},

"productExplorer": {
"type": "$defaultWebServer",
"title": "API server"
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions pages/KEY_VALUE_STORE_SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This JSON file should contain schema for files stored in the key-value store,
defining their name, format, or content type.

**BEWARE: This is currently not implemented yet and work in progress.**
**BEWARE: This is currently not implemented yet and subject to change.**

## Basic properties

Expand All @@ -19,7 +19,7 @@ UI and API.
"collections": {
"screenshots": {
"name": "Post images",
"keyPrefix": "images-",
"keyPrefixes": ["images-"],
"contentTypes": ["image/jpeg", "image/png"]
}
}
Expand Down
113 changes: 45 additions & 68 deletions pages/OUTPUT_SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ to see format of its results as it's predefined by the output schema.

The output schema is also used by the system to generate the user interface, API examples, integrations, etc.

The file format is a JSON Schema with our extensions.

## Structure

```jsonc
Expand All @@ -22,47 +24,61 @@ The output schema is also used by the system to generate the user interface, API
"title": "Some title", // optional
"description": "Text that is shown in the Output UI", // optional
"type": "object",
// TODO: This should be JSON schema with extensions, for consistency with input schema and dataset schema
"properties": {
// Default dataset contains all the scraped products
// In the "output" object, the field should be a link to dataset with the right view

// Properties in output reference Actor run's default storages,
// or live view / Standby mode server.

"currentProducts": {
// We extend JSON Schema "type" with the new ones, all prefixed with '$':
"type": "$dataset",
// You can reference views how to render the output, using "views" defined by the Dataset schema file
"views": ["productVariants"]
// Optionally, the output can reference an existing dataset to provide its results from,
// if it was passed via "type": "dataset" property.
"target": "$input.myProductsDatasetId"
// or specific storage JSON schema file
"type": "./dataset_schema.json",

// Specify where the value will eventually be produced
"link": "$actor.dataset",
// you can also reference a property from input object,
// the linkage will be checked for type compatibility
"link": "$input.myProductsDatasetId",

// Select views how to render the output, using "views" defined by the Dataset schema file
"views": ["productVariants"],
},

// Selects a specific group of records with a certain prefix. In UI, this can be shown
// as a list of images. In the output object, this will be a link to a API with "prefix" param.
"productImages": {
"type": "$keyValueStore",
"title": "Product images", //optional
"description": "Yaddada", //optional
"collections": ["screenshots"] // optional, default means all collections in key-value-store
},

// If the users want to reference a single file, they do it via selecting a collection
// that displays only a single file. In the output object, the result should be a link to file.
"summaryReportView": {
"type": "$keyValueStore",
"title": "API server", // optional
"collections": ["monitoringReport"],
"target": "default" //optional
// or specific storage JSON schema file
"type": "./key_value_store_schema.json",

"link": "$actor.keyValueStore",

"title": "Product images",
"description": "Yaddada", // optional

// optionally, you can specify which files to display in UI for key-value stores
"keyPrefixes": ["images-"],
"collection": "screenshots",
},

// Live view
// In the "output" object, the result should be a link to live view URL
"apiServer": {
"type": "$webServer",
"title": "API server", // optional
// Live view web server for to the Actor, or Standby mode fixed URL
// In the "output" view, this page is rendered in an IFRAME
"productExplorer": {
// a generic web server
"type": "$defaultWebServer",

// Reference an OpenAPI schema of the web server
"type": "./web_server_openapi.json"

"link": "$actor.webServer",

"title": "API server",

"description": "API documentation is available in swagger.com/api/xxxx", // optional
"path": "/nice-report?query=123",
// IDEA: In the future, we could perhaps work with the API/swagger schema on more advanced level,
// but it might be an overkill
"schema": "TODO"

// specify a path to open?
"viewPath": "/nice-report?query=123",
}
}
}
Expand Down Expand Up @@ -126,42 +142,3 @@ This tab will be at the first position and displayed by default. Tab will show t
- Default setup, i.e., what output components should be displayed at the default run tab
- Optionally, the setup for different states
- Be able to pragmatically changes this using API by Actor itself


## OUTPUT.json

```jsonc
{
"title": "Some title",
"description": "Some description",
"properties": {
"currentProducts": {
"id": "lkspwWd8dFknjkxx",
"type": "dataset",
"url": "https://api.apify.com/datasets/lkspwWd8dFknjkxx",
"views": {
"productVariants": {
"title": "Product variants",
"description": "yadayada",
"url": "https://api.apify.com/datasets/lkspwWd8dFknjkxx/?view=productVariants"
}
}
},
"productImages": {
"id": "FkspwWd8dFknjkxx",
"type": "key-value-store",
"title": "Product images",
"description": "Yaddada",
"url": "https://api.apify.com/key-value-stores/FkspwWd8dFknjkxx",
"collections": {
"screenshots": {
"title": "...",
"description": "...",
"url": "https://api.apify.com/key-value-stores//lkspwWd8dFknjkxx/?collection=screenshots"
}
}
}
}
}
```

Loading