Error: Failed to Ingest

[mayooear]

If you run the ingest script and run into errors calling upsert for the PineconeClient and have followed every single step in the README.md, here are some major culprits to troubleshoot:

• In the config folder, replace the PINECONE_INDEX_NAME to match your index name in pinecone.
• Make sure to set Dimensions in the Pinecone dashboard to 1536. (These are OpenAI embeddings dimensions).
• Switch your Environment in pinecone to another environment, if the other environment is causing issues (requires paid plan).
• Ensure you have a .env file in the root that contains valid API keys from the pinecone dashboard. Test the api key separately from the repo.
• Pinecone indexes of users on the Starter(free) plan are deleted after 7 days of inactivity. To prevent this, send an API request to Pinecone to reset the counter.
• Check Pinecone status for uptime issues.

[luomin]

Hi @mayooear

How can i Switch my Environment in pinecone to us-east1-gcp, as my default is us-central1-gcp.
Do I need to upgrade my account to Standard or Enterprise?

Because I am currently getting error as below when running "npm run ingest", thank you for an advice

us-central1-gcp
PineconeClient: Error getting project name: FetchError: request to https://controller.us-central1-gcp.pinecone.io/actions/whoami failed, reason: unable to get local issuer certificate
...
creating vector store...
error [Error: PineconeClient: Project name not set. Call init() first.]
(node:51504) UnhandledPromiseRejectionWarning

[mayooear]

Yes you are correct, it requires an upgrade to switch environments. Do you think yours is causing the issue?

[luomin]

Hi @mayooear

Thanks for the reply.

I am not sure because i am not able to change the hosting location.
But i am getting the invalid API key error as below

curl -i https://controller.us-central1-gcp.pinecone.io/actions/whoami -H 'my pinecone API key'

HTTP/1.1 401 Unauthorized
www-authenticate: API key is missing or invalid for the environment "us-central1-gcp". Check that the correct environment is specified.
content-length: 117
date: Wed, 22 Mar 2023 02:40:17 GMT
server: envoy
Proxy-Support: Session-Based-Authentication

API key is missing or invalid for the environment "us-central1-gcp". Check that the correct environment is specified.

I believe i correctly copied the API key as in my pipecone API Keys dashboard, do you think this may also caused by the hosting location?

[yudidina]

Do I need to upgrade my account to Standard or Enterprise?

Hello, did you solve this problem? I have the question

[mayooear]

If you run the ingest script and run into errors calling upsert for the PineconeClient and have followed every single step in the README.md, here are some major culprits to troubleshoot:

• In the config folder, replace the PINECONE_INDEX_NAME to match your index name in pinecone.
• Make sure to set Dimensions in the Pinecone dashboard to 1536. (These are OpenAI embeddings dimensions).
• Switch your Environment in pinecone to us-east1-gcp if the other environment is causing issues.
• Ensure you have a .env file in the root that contains valid API keys from the pinecone dashboard.
• Pinecone has limits for each upsert operation, you can read them here and see some below. If you are uploading massive PDF files, you just need to write a loop to ensure upserts don't exceed 100 chunks per request. I will make a PR to the LangChain repo to integrate this.
  • Max size for an upsert request is 2MB. Recommended upsert limit is 100 vectors per request.
  • Max metadata size per vector is 40 KB.
• Pinecone indexes of users on the Starter(free) plan are deleted after 7 days of inactivity. To prevent this, send an API request to Pinecone to reset the counter.

I merged a PR to fix the large size issue causing upsert errors.

[tamvm]

Hi @mayooear,

Pinecone doesnt allow to create index in another environment (see attached)

Also they announced that there's some issue creation in us-east1-gcp as well https://status.pinecone.io/?utm_source=embed and advised to create in another enviroment.

I created an index in default environment with Demensions set to 1536 (and upgraded my account in Pinecone as well) but still got this error when running ingest:

error [TypeError: t.replaceAll is not a function]
(node:71384) UnhandledPromiseRejectionWarning
(Use `node --trace-warnings ...` to show where the warning was created)
(node:71384) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 27)
(node:71384) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Please advise.

[mayooear]

"This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). "

Have you resolved issue?

[naticio]

I was able to fix my issue....

I created a new project with a new index in pinecone, and apparently each project has a different api so I was using the previous project api...

[Alvaroinfantee]

Hey. Sorry if this is a dumb question but does anyone know how to fix this error: Error: Failed to ingest your data
at run (c:\Users\alvar\OneDrive\Escritorio\gpt4-pdf-chatbot-langchain-main\scripts\ingest-data.ts:44:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at (c:\Users\alvar\OneDrive\Escritorio\gpt4-pdf-chatbot-langchain-main\scripts\ingest-data.ts:49:3)

I have tried starting fresh with a new project and index. Im on the latest node version. I made sure my keys were correct in the .env file, etc.

What can I do?

Thanks for your time

[jontestfastai]

same error message for me

[Q22AN]

I am on Mac with the latest npm version and did all the steps laid out in the readme.
I could not run the ingest on either Pinecone nor chroma.

[nickmortensen]

I spent the better part of a day with a similar issue that resolved itself once I put the env variables inside of quotation marks. As in:

• OPENAI_API_KEY="sk-WcCpC000000000000Gj"
• PINECONE_API_KEY="100000-efef-408020-00000xxx"
• PINECONE_ENVIRONMENT="asia-northeast1-gcp"
• PINECONE_INDEX_NAME="testindex"

Deeply ashamed? Absolutely.

[Alvaroinfantee]

I spent the better part of a day with a similar issue that resolved itself once I put the env variables inside of quotation marks. As in:

• OPENAI_API_KEY="sk-WcCpC000000000000Gj"
• PINECONE_API_KEY="100000-efef-408020-00000xxx"
• PINECONE_ENVIRONMENT="asia-northeast1-gcp"
• PINECONE_INDEX_NAME="testindex"

Deeply ashamed? Absolutely.

I had tried it a few times, tried it again just in case and still the same error.
Thanks tho

[sagerock]

I am happy to report that I overcame my own "Failed to ingest" brick wall.

Here are a couple things that helped me.

@Alvaroinfantee is right. You definitely want to put your env variables in quotes. I hadn't done that. (But that didn't cause my "failed to ingest" issue.

My Node was out of date. So I downloaded a new version of that. But that didn't fix my problem.

On line 10 of "ingest-data.ts" I worked on pointing to my document. That didn't do it.

This is what worked for me:

1. I deleted my index at Pincone and started over. I'm using the free Environment: us-west4-gcp. It has 1536 Dimensions. My .env file looks like this:
   OPENAI_API_KEY="sk-5ulsKRudqG1kljdgfa;jigdjgiojg;mg;mdf"
   PINECONE_API_KEY="7ffe6akjfgkjgakgjkafm;asfma"
   PINECONE_ENVIRONMENT="us-west4-gcp"
   PINECONE_INDEX_NAME="thursdaynight"

2. I pulled down the repository: https://github.com/mayooear/gpt4-pdf-chatbot-langchain/tree/main and cloned it in a new location on my computer.

I'm using the Terminal in Visual Studio Code. While I was in the folder of my new local repository in Visual Studio Code I ran this:
npm install yarn -g
Then I ran this:
yarn install

I set up new .env file like I showed above.
Then I ran this: npm run ingest
Then I ran this: npm run dev

And it worked.

I think I had probably screwed something up the first time I tried installing this. And it did something that made it so the entire thing wouldn't work.

So just started completely over was the answer for me.

This is an AMAZING script. I'm so excited to see the different ways I can use it.

Thank you!
Sage

[wasabikev]

Many thanks to all the comments above. It helped me clear this hurdle.
In the end, mine was the result of a mismatch between on the name of the index listed in Pinecone and what was in my .env file. Newbie mistake.