Skip to content

Commit

Permalink
feat: support cohere and claude 3 in rag example
Browse files Browse the repository at this point in the history
  • Loading branch information
HQarroum committed Sep 16, 2024
1 parent d8aea3b commit 4fd86c9
Show file tree
Hide file tree
Showing 124 changed files with 1,337 additions and 784 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: TAR Inflate
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/tar-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Zip
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/zip-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Unzip
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/zip-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Audio Metadata
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/audio-metadata-extractor">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/audio-processing/bark-synthesizer.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Bark
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/bark-synthesizer">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Polly
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/polly-synthesizer">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Transcribe
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/transcribe-audio-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Whisper
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/whisper-transcriber">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Firehose Connector
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/firehose-storage-connector">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/data-stores/s3-storage-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: S3 Connector
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/s3-storage-connector">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/data-stores/sqs-storage-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: SQS Connector
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/sqs-storage-connector">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: CLIP
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/clip-image-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: PANN
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/panns-embedding-processor">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/flow-control/condition.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Condition
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/condition">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/flow-control/delay.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Delay
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/delay">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/flow-control/passthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Passthrough
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/passthrough">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/flow-control/reducer.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Reducer
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/reducer">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Extractor
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/structured-entity-extractor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Drawing
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/image-layer-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Image Metadata
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/image-metadata-extractor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: BERT Summarizer
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/bert-extractive-summarizer">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Email
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/email-text-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: JMESPath
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/jmespath-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: KeyBERT
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/keybert-text-processor">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/text-processing/newspaper3k.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Newspaper3k
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/newspaper3k">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: NLP
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/nlp-text-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Pandoc
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/pandoc-text-converter">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: PDF
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/pdf-text-converter">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Subtitle
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/subtitle-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: RSS Feeds
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/syndication-feed-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Text Transform
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/text-transform-processor">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/text-processing/trafilatura.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Trafilatura
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/trafilatura">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Translate
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/translate-text-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Regexp Splitter
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/regexp-text-splitter">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Sentence Splitter
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/sentence-text-splitter">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Tiling Splitter
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/tiling-text-splitter">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/triggers/s3-event-trigger.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: S3 Trigger
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/s3-event-trigger">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/triggers/scheduler-event-trigger.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Scheduler
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/scheduler-event-trigger">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/triggers/sqs-event-trigger.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: SQS Trigger
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/sqs-event-trigger">
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/video-processing/ffmpeg-processor.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: FFMPEG
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/ffmpeg-processor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Video Metadata
Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
0.8.0
0.9.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
<a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/video-metadata-extractor">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Before moving towards the retrieval step of interrogating an LLM using the RAG C

Since the OpenSearch domain deployed in this example resides in a new isolated VPC that's not accessible from the public Internet, you will need to use a bastion host in that VPC to access the OpenSearch domain from your local machine.

> 💁 You can for example use an EC2 bastion host to establish a soft VPN connection to your VPC (see [`sshuttle`](https://github.com/sshuttle/sshuttle)). Alternatively, you can also run this example within an [AWS Cloud9](https://docs.aws.amazon.com/cloud9/latest/user-guide/welcome.html) instance located in the same VPC as the OpenSearch endpoint.
> 💁 You can for example use an EC2 bastion host to establish a soft VPN connection to your VPC using [`sshuttle`](https://github.com/sshuttle/sshuttle). Alternatively, you can also run this example within an [AWS Cloud9](https://docs.aws.amazon.com/cloud9/latest/user-guide/welcome.html) instance located in the same VPC as the OpenSearch endpoint.
The URL of the OpenSearch domain is displayed in the CDK outputs as `OpenSearchEndpoint`, and you can check whether you have access using the following command.

Expand Down Expand Up @@ -139,8 +139,8 @@ npm install
You can then invoke the CLI and ask a question to your documents. For example, if your deployment region is `eu-west-1` and you would like to use Amazon Bedrock in the `us-east-1` region, you can run the following command:

```bash
node dist/index.js \
--opensearch-endpoint 'https://domain.eu-west-1.es.amazonaws.com' \
npx tsx --tsconfig tsconfig.json src/index.ts \
--opensearch-endpoint 'https://vpc-opensearch-vectors-example.eu-west-1.es.amazonaws.com' \
--opensearch-region eu-west-1 \
--bedrock-region us-east-1
```
Expand Down
Loading

0 comments on commit 4fd86c9

Please sign in to comment.