diff --git a/.env.example b/.env.example
index 7c1bb4ec4..f86255e5a 100644
--- a/.env.example
+++ b/.env.example
@@ -1,2 +1,3 @@
OPENAI_API_KEY=
-TAVILY_API_KEY=
\ No newline at end of file
+TAVILY_API_KEY=
+DOC_PATH=./my-docs
\ No newline at end of file
diff --git a/README-ko_KR.md b/README-ko_KR.md
index f6c24a929..418436958 100644
--- a/README-ko_KR.md
+++ b/README-ko_KR.md
@@ -128,7 +128,7 @@ python -m uvicorn main:app --reload
-**[Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started#poetry) 또는 [가상 환경](https://docs.gptr.dev/docs/gpt-researcher/getting-started#virtual-environment)에 대해 배우고 싶다면, [문서](https://docs.gptr.dev/docs/gpt-researcher/getting-started)를 참조하세요.**
+**[Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#poetry) 또는 [가상 환경](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#virtual-environment)에 대해 배우고 싶다면, [문서](https://docs.gptr.dev/docs/gpt-researcher/getting-started)를 참조하세요.**
### PIP 패키지로 실행하기
```bash
@@ -148,11 +148,11 @@ report = await researcher.write_report()
...
```
-**더 많은 예제와 구성 옵션은 [PIP 문서](https://docs.gptr.dev/docs/gpt-researcher/pip-package)를 참조하세요.**
+**더 많은 예제와 구성 옵션은 [PIP 문서](https://docs.gptr.dev/docs/gpt-researcher/gptr/pip-package)를 참조하세요.**
## Docker로 실행
-> **1단계** - [Docker 설치](https://docs.gptr.dev/docs/gpt-researcher/getting-started#try-it-with-docker)
+> **1단계** - [Docker 설치](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
> **2단계** - `.env.example` 파일을 복사하고 API 키를 추가한 후, 파일을 `.env`로 저장하세요.
@@ -180,7 +180,7 @@ export DOC_PATH="./my-docs"
2단계:
- 프론트엔드 앱을 localhost:8000에서 실행 중이라면, "Report Source" 드롭다운 옵션에서 "My Documents"를 선택하세요.
- - GPT Researcher를 [PIP 패키지](https://docs.tavily.com/docs/gpt-researcher/pip-package)로 실행 중이라면, `report_source` 인수를 "documents"로 설정하여 `GPTResearcher` 클래스를 인스턴스화하세요. [코드 예제](https://docs.tavily.com/docs/gpt-researcher/tailored-research)를 참조하세요.
+ - GPT Researcher를 [PIP 패키지](https://docs.tavily.com/docs/gpt-researcher/pip-package)로 실행 중이라면, `report_source` 인수를 "local"로 설정하여 `GPTResearcher` 클래스를 인스턴스화하세요. [코드 예제](https://docs.gptr.dev/docs/gpt-researcher/context/tailored-research)를 참조하세요.
## 👪 다중 에이전트 어시스턴트
@@ -190,7 +190,7 @@ LangGraph를 사용하면 여러 에이전트의 전문 기술을 활용하여
평균 실행은 5-6 페이지 분량의 연구 보고서를 PDF, Docx, Markdown 형식으로 생성합니다.
-[여기](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents)에서 확인하거나 [문서](https://docs.gptr.dev/docs/gpt-researcher/langgraph)에서 자세한 내용을 참조하세요.
+[여기](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents)에서 확인하거나 [문서](https://docs.gptr.dev/docs/gpt-researcher/multi_agents)에서 자세한 내용을 참조하세요.
## 🖥️ 프론트엔드 애플리케이션
diff --git a/README.md b/README.md
index a42bcf868..69e7ff9d9 100644
--- a/README.md
+++ b/README.md
@@ -17,7 +17,7 @@
[](https://badge.fury.io/py/gpt-researcher)
-[](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/examples/pip-run.ipynb)
+[](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/docs/docs/examples/pip-run.ipynb)
[](https://hub.docker.com/r/gptresearcher/gpt-researcher)
[](https://twitter.com/assaf_elovic)
@@ -108,7 +108,7 @@ export TAVILY_API_KEY={Your Tavily API Key here}
For a more permanent setup, create a `.env` file in the current `gpt-researcher` directory and input the env vars (without `export`).
- The default LLM is [GPT](https://platform.openai.com/docs/guides/gpt), but you can use other LLMs such as `claude`, `ollama3`, `gemini`, `mistral` and more. To learn how to change the LLM provider, see the [LLMs documentation](https://docs.gptr.dev/docs/gpt-researcher/llms) page. Please note: this project is optimized for OpenAI GPT models.
-- The default retriever is [Tavily](https://app.tavily.com), but you can refer to other retrievers such as `duckduckgo`, `google`, `bing`, `serper`, `searx`, `arxiv`, `exa` and more. To learn how to change the search provider, see the [retrievers documentation](https://docs.gptr.dev/docs/gpt-researcher/retrievers) page.
+- The default retriever is [Tavily](https://app.tavily.com), but you can refer to other retrievers such as `duckduckgo`, `google`, `bing`, `serper`, `searx`, `arxiv`, `exa` and more. To learn how to change the search provider, see the [retrievers documentation](https://docs.gptr.dev/docs/gpt-researcher/search-engines/retrievers) page.
### Quickstart
@@ -128,7 +128,7 @@ python -m uvicorn main:app --reload
-**To learn how to get started with [Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started#poetry) or a [virtual environment](https://docs.gptr.dev/docs/gpt-researcher/getting-started#virtual-environment) check out the [documentation](https://docs.gptr.dev/docs/gpt-researcher/getting-started) page.**
+**To learn how to get started with [Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#poetry) or a [virtual environment](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#virtual-environment) check out the [documentation](https://docs.gptr.dev/docs/gpt-researcher/getting-started) page.**
### Run as PIP package
```bash
@@ -148,21 +148,27 @@ report = await researcher.write_report()
...
```
-**For more examples and configurations, please refer to the [PIP documentation](https://docs.gptr.dev/docs/gpt-researcher/pip-package) page.**
+**For more examples and configurations, please refer to the [PIP documentation](https://docs.gptr.dev/docs/gpt-researcher/gptr/pip-package) page.**
## Run with Docker
-> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started#try-it-with-docker)
+> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
> **Step 2** - Clone the '.env.example' file, add your API Keys to the cloned file and save the file as '.env'
> **Step 3** - Within the docker-compose file comment out services that you don't want to run with Docker.
```bash
-$ docker-compose up --build
+docker-compose up --build
```
+If that doesn't work, try running it without the dash:
+```bash
+docker compose up --build
+```
+
+
> **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
- the Python server running on localhost:8000
- the React app running on localhost:3000
@@ -183,7 +189,8 @@ export DOC_PATH="./my-docs"
Step 2:
- If you're running the frontend app on localhost:8000, simply select "My Documents" from the the "Report Source" Dropdown Options.
- - If you're running GPT Researcher with the [PIP package](https://docs.tavily.com/docs/gpt-researcher/pip-package), pass the `report_source` argument as "documents" when you instantiate the `GPTResearcher` class [code sample here](https://docs.tavily.com/docs/gpt-researcher/tailored-research).
+ - If you're running GPT Researcher with the [PIP package](https://docs.tavily.com/docs/gpt-researcher/pip-package), pass the `report_source` argument as "local" when you instantiate the `GPTResearcher` class [code sample here](https://docs.gptr.dev/docs/gpt-researcher/context/tailored-research).
+
## 👪 Multi-Agent Assistant
As AI evolves from prompt engineering and RAG to multi-agent systems, we're excited to introduce our new multi-agent assistant built with [LangGraph](https://python.langchain.com/v0.1/docs/langgraph/).
@@ -192,7 +199,7 @@ By using LangGraph, the research process can be significantly improved in depth
An average run generates a 5-6 page research report in multiple formats such as PDF, Docx and Markdown.
-Check it out [here](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents) or head over to our [documentation](https://docs.gptr.dev/docs/gpt-researcher/langgraph) for more information.
+Check it out [here](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents) or head over to our [documentation](https://docs.gptr.dev/docs/gpt-researcher/multi_agents/langgraph) for more information.
## 🖥️ Frontend Applications
diff --git a/examples/pip-run.ipynb b/docs/docs/examples/pip-run.ipynb
similarity index 100%
rename from examples/pip-run.ipynb
rename to docs/docs/examples/pip-run.ipynb
diff --git a/examples/sample_report.py b/docs/docs/examples/sample_report.py
similarity index 100%
rename from examples/sample_report.py
rename to docs/docs/examples/sample_report.py
diff --git a/examples/sample_sources_only.py b/docs/docs/examples/sample_sources_only.py
similarity index 100%
rename from examples/sample_sources_only.py
rename to docs/docs/examples/sample_sources_only.py
diff --git a/docs/docs/faq.md b/docs/docs/faq.md
index 9d8e54902..cf4d36337 100644
--- a/docs/docs/faq.md
+++ b/docs/docs/faq.md
@@ -4,22 +4,29 @@
It really depends on what you're aiming for.
If you're looking to connect your AI application to the internet with Tavily tailored API, check out the [Tavily API](https://docs.tavily.com/docs/tavily-api/introductionn) documentation.
-If you're looking to build and deploy our open source autonomous research agent GPT Researcher, please see [GPT Researcher](/docs/gpt-researcher/introduction) documentation.
+If you're looking to build and deploy our open source autonomous research agent GPT Researcher, please see [GPT Researcher](/docs/gpt-researcher/getting-started/introduction) documentation.
You can also check out demos and examples for inspiration [here](/docs/examples/examples).
+
### What is GPT Researcher?
+
GPT Researcher is a popular open source autonomous research agent that takes care of the tedious task of research for you, by scraping, filtering and aggregating over 20+ web sources per a single research task.
GPT Researcher is built with best practices for leveraging LLMs (prompt engineering, RAG, chains, embeddings, etc), and is optimized for quick and efficient research. It is also fully customizable and can be tailored to your specific needs.
-To learn more about GPT Researcher, check out the [documentation page](/docs/gpt-researcher/introduction).
+To learn more about GPT Researcher, check out the [documentation page](/docs/gpt-researcher/getting-started/introduction).
+
### How much does each research run cost?
+
A research task using GPT Researcher costs around $0.01 per a single run (for GPT-4 usage). We're constantly optimizing LLM calls to reduce costs and improve performance.
+
### How do you ensure the report is factual and accurate?
+
we do our best to ensure that the information we provide is factual and accurate. We do this by using multiple sources, and by using proprietary AI to score and rank the most relevant and accurate information. We also use proprietary AI to filter out irrelevant information and sources.
Lastly, by using RAG and other techniques, we ensure that the information is relevant to the context of the research task, leading to more accurate generative AI content and reduced hallucinations.
### What are your plans for the future?
+
We're constantly working on improving our products and services. We're currently working on improving our search API together with design partners, and adding more data sources to our search engine. We're also working on improving our research agent GPT Researcher, and adding more features to it while growing our amazing open source community.
If you're interested in our roadmap or looking to collaborate, check out our [roadmap page](https://trello.com/b/3O7KBePw/gpt-researcher-roadmap).
diff --git a/docs/docs/gpt-researcher/context/filtering-by-domain.md b/docs/docs/gpt-researcher/context/filtering-by-domain.md
new file mode 100644
index 000000000..c413c915a
--- /dev/null
+++ b/docs/docs/gpt-researcher/context/filtering-by-domain.md
@@ -0,0 +1,22 @@
+# Filtering by Domain
+
+If you set Google as a Retriever, you can filter web results by site.
+
+For example, set in the query param you pass the GPTResearcher class instance: `query="site:linkedin.com a python web developer to implement my custom gpt-researcher flow"` will limit the results to linkedin.com
+
+> **Step 1** - Set these environment variables with a .env file in the root folder
+
+TAVILY_API_KEY=
+LANGCHAIN_TRACING_V2=true
+LANGCHAIN_API_KEY=
+OPENAI_API_KEY=
+DOC_PATH=./my-docs
+RETRIEVER=google
+GOOGLE_API_KEY=
+GOOGLE_CX_KEY=
+
+> **Step 2** - from the root project run:
+
+docker-compose up -- build
+
+> **Step 3** - from the frontend input box in localhost:3000, you can append any google search filter (such as filtering by domain names)
\ No newline at end of file
diff --git a/docs/docs/gpt-researcher/context/local-docs.md b/docs/docs/gpt-researcher/context/local-docs.md
new file mode 100644
index 000000000..46c0cf38f
--- /dev/null
+++ b/docs/docs/gpt-researcher/context/local-docs.md
@@ -0,0 +1,22 @@
+# 📄 Research on Local Documents
+
+## Just Local Docs
+
+You can instruct the GPT Researcher to run research tasks based on your local documents. Currently supported file formats are: PDF, plain text, CSV, Excel, Markdown, PowerPoint, and Word documents.
+
+Step 1: Add the env variable `DOC_PATH` pointing to the folder where your documents are located.
+
+```bash
+export DOC_PATH="./my-docs"
+```
+
+Step 2:
+ - If you're running the frontend app on localhost:8000, simply select "My Documents" from the the "Report Source" Dropdown Options.
+ - If you're running GPT Researcher with the [PIP package](https://docs.tavily.com/docs/gpt-researcher/gptr/pip-package), pass the `report_source` argument as "local" when you instantiate the `GPTResearcher` class [code sample here](https://docs.gptr.dev/docs/gpt-researcher/context/tailored-research).
+
+## Local Docs + Web (Hybrid)
+
+
+
+Check out the blog post on [Hybrid Research](https://docs.gptr.dev/blog/gptr-hybrid) to learn more about how to combine local documents with web research.
+```
\ No newline at end of file
diff --git a/docs/docs/gpt-researcher/tailored-research.md b/docs/docs/gpt-researcher/context/tailored-research.md
similarity index 98%
rename from docs/docs/gpt-researcher/tailored-research.md
rename to docs/docs/gpt-researcher/context/tailored-research.md
index 71f4ec5e7..2ff32faed 100644
--- a/docs/docs/gpt-researcher/tailored-research.md
+++ b/docs/docs/gpt-researcher/context/tailored-research.md
@@ -1,4 +1,5 @@
# Tailored Research
+
The GPT Researcher package allows you to tailor the research to your needs such as researching on specific sources or local documents, and even specify the agent prompt instruction upon which the research is conducted.
### Research on Specific Sources 📚
@@ -88,7 +89,7 @@ You can combine the above methods to conduct hybrid research. For example, you c
Simply provide the sources and set the `report_source` argument as `"hybrid"` and watch the magic happen.
Please note! You should set the proper retrievers for the web sources and doc path for local documents for this to work.
-To lean more about retrievers check out the [Retrievers](https://docs.gptr.dev/docs/gpt-researcher/retrievers) documentation.
+To lean more about retrievers check out the [Retrievers](https://docs.gptr.dev/docs/gpt-researcher/search-engines/retrievers) documentation.
### Research on LangChain Documents 🦜️🔗
diff --git a/docs/docs/gpt-researcher/vector-stores.md b/docs/docs/gpt-researcher/context/vector-stores.md
similarity index 99%
rename from docs/docs/gpt-researcher/vector-stores.md
rename to docs/docs/gpt-researcher/context/vector-stores.md
index 5a9fc9496..bd6100109 100644
--- a/docs/docs/gpt-researcher/vector-stores.md
+++ b/docs/docs/gpt-researcher/context/vector-stores.md
@@ -1,4 +1,5 @@
# Vector Stores
+
The GPT Researcher package allows you to integrate with existing langchain vector stores that have been populated.
For a complete list of supported langchain vector stores, please refer to this [link](https://python.langchain.com/v0.2/docs/integrations/vectorstores/).
diff --git a/docs/docs/gpt-researcher/frontend.md b/docs/docs/gpt-researcher/frontend/frontend.md
similarity index 59%
rename from docs/docs/gpt-researcher/frontend.md
rename to docs/docs/gpt-researcher/frontend/frontend.md
index b7df49cdd..a68fc284a 100644
--- a/docs/docs/gpt-researcher/frontend.md
+++ b/docs/docs/gpt-researcher/frontend/frontend.md
@@ -2,7 +2,45 @@
This frontend project aims to enhance the user experience of GPT-Researcher, providing an intuitive and efficient interface for automated research. It offers two deployment options to suit different needs and environments.
-## Option 1: Static Frontend (FastAPI)
+View a Product Tutorial here: [GPT-Researcher Frontend Tutorial](https://www.youtube.com/watch?v=hIZqA6lPusk)
+
+
+## NextJS Frontend App
+
+The React app (located in the `frontend` directory) is our Frontend 2.0 which we hope will enable us to display the robustness of the backend on the frontend, as well.
+
+It comes with loads of added features, such as:
+ - a drag-n-drop user interface for uploading and deleting files to be used as local documents by GPTResearcher.
+ - a GUI for setting your GPTR environment variables.
+ - the ability to trigger the multi_agents flow via the Backend Module or Langgraph Cloud Host (currently in closed beta).
+ - stability fixes
+ - and more coming soon!
+
+### Run the NextJS React App with Docker
+
+> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
+
+> **Step 2** - Clone the '.env.example' file, add your API Keys to the cloned file and save the file as '.env'
+
+> **Step 3** - Within the docker-compose file comment out services that you don't want to run with Docker.
+
+```bash
+docker compose up --build
+```
+
+If that doesn't work, try running it without the dash:
+```bash
+docker compose up --build
+```
+
+> **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
+ - the Python server running on localhost:8000
+ - the React app running on localhost:3000
+
+Visit localhost:3000 on any browser and enjoy researching!
+
+
+## Other Options: 1: Static Frontend (FastAPI)
A lightweight solution using FastAPI to serve static files.
@@ -28,7 +66,7 @@ A lightweight solution using FastAPI to serve static files.
-## Option 2: NextJS Frontend
+## Yet Another Option: Running NextJS Frontend via CLI
A more robust solution with enhanced features and performance.
diff --git a/docs/docs/gpt-researcher/frontend/playing-with-webhooks.md b/docs/docs/gpt-researcher/frontend/playing-with-webhooks.md
new file mode 100644
index 000000000..310613ccb
--- /dev/null
+++ b/docs/docs/gpt-researcher/frontend/playing-with-webhooks.md
@@ -0,0 +1,23 @@
+# Playing with Webhooks
+
+The GPTR Frontend is powered by Webhooks streaming back from the Backend. This allows for real-time updates on the status of your research tasks, as well as the ability to interact with the Backend directly from the Frontend.
+
+
+## Inspecting Webhooks
+
+When running reports via the frontend, you can inspect the websocket messages in the Network Tab.
+
+Here's how:
+
+
+
+
+### Am I polling the right URL?
+
+If you're concerned that your frontend isn't hitting the right API Endpoint, you can check the URL in the Network Tab.
+
+Click into the WS request & go to the "Headers" tab
+
+
+
+For debugging, have a look at the getHost function.
\ No newline at end of file
diff --git a/docs/docs/gpt-researcher/getting-started/getting-started-with-docker.md b/docs/docs/gpt-researcher/getting-started/getting-started-with-docker.md
new file mode 100644
index 000000000..e2928c856
--- /dev/null
+++ b/docs/docs/gpt-researcher/getting-started/getting-started-with-docker.md
@@ -0,0 +1,28 @@
+# Docker: Quickstart
+
+> **Step 1** - Install & Open Docker Desktop
+
+Follow instructions at https://www.docker.com/products/docker-desktop/
+
+
+> **Step 2** - [Follow this flow](https://www.youtube.com/watch?v=x1gKFt_6Us4)
+
+This mainly includes cloning the '.env.example' file, adding your API Keys to the cloned file and saving the file as '.env'
+
+> **Step 3** - Within root, run with Docker.
+
+```bash
+docker-compose up --build
+```
+
+If that doesn't work, try running it without the dash:
+```bash
+docker compose up --build
+```
+
+> **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
+ - the Python server running on localhost:8000
+ - the React app running on localhost:3000
+
+Visit localhost:3000 on any browser and enjoy researching!
+
diff --git a/docs/docs/gpt-researcher/getting-started.md b/docs/docs/gpt-researcher/getting-started/getting-started.md
similarity index 91%
rename from docs/docs/gpt-researcher/getting-started.md
rename to docs/docs/gpt-researcher/getting-started/getting-started.md
index f1803084c..57e47c12c 100644
--- a/docs/docs/gpt-researcher/getting-started.md
+++ b/docs/docs/gpt-researcher/getting-started/getting-started.md
@@ -1,4 +1,5 @@
# Getting Started
+
> **Step 0** - Install Python 3.11 or later. [See here](https://www.tutorialsteacher.com/python/install-python) for a step-by-step guide.
> **Step 1** - Download the project and navigate to its directory
@@ -101,22 +102,3 @@ python -m uvicorn main:app --reload
-## Try it with Docker
-
-> **Step 1** - Install Docker
-
-Follow instructions at https://docs.docker.com/engine/install/
-
-> **Step 2** - Create .env file with your OpenAI Key or simply export it
-
-```bash
-$ export OPENAI_API_KEY={Your API Key here}
-```
-
-> **Step 3** - Run the application
-
-```bash
-$ docker-compose up
-```
-
-> **Step 4** - Go to http://localhost:8000 on any browser and enjoy researching!
diff --git a/docs/docs/gpt-researcher/introduction.md b/docs/docs/gpt-researcher/getting-started/introduction.md
similarity index 98%
rename from docs/docs/gpt-researcher/introduction.md
rename to docs/docs/gpt-researcher/getting-started/introduction.md
index 4cdb193c1..0d19b9b4e 100644
--- a/docs/docs/gpt-researcher/introduction.md
+++ b/docs/docs/gpt-researcher/getting-started/introduction.md
@@ -6,7 +6,7 @@
[](https://github.com/assafelovic/gpt-researcher)
[](https://twitter.com/assaf_elovic)
[](https://badge.fury.io/py/gpt-researcher)
-[](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/examples/pip-run.ipynb)
+[](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/docs/docs/examples/pip-run.ipynb)
**[GPT Researcher](https://gptr.dev) is an autonomous agent designed for comprehensive online research on a variety of tasks.**
diff --git a/docs/docs/gpt-researcher/getting-started/linux-deployment.md b/docs/docs/gpt-researcher/getting-started/linux-deployment.md
new file mode 100644
index 000000000..d830a0b8a
--- /dev/null
+++ b/docs/docs/gpt-researcher/getting-started/linux-deployment.md
@@ -0,0 +1,122 @@
+# Running on Linux
+
+This guide will walk you through the process of deploying GPT Researcher on a Linux server.
+
+## Server Requirements
+
+The default Ubuntu droplet option on [DigitalOcean](https://m.do.co/c/1a2af257efba) works well, but this setup should work on any hosting service with similar specifications:
+
+- 2 GB RAM
+- 1 vCPU
+- 50 GB SSD Storage
+
+Here's a screenshot of the recommended Ubuntu machine specifications:
+
+
+
+## Deployment Steps
+
+After setting up your server, follow these steps to install Docker, Docker Compose, and Nginx.
+
+
+Some more commands to achieve that:
+
+### Step 1: Update the System
+### First, ensure your package index is up-to-date:
+
+```bash
+sudo apt update
+### Step 2: Install Git
+### Git is a version control system. Install it using:
+
+sudo apt install git -y
+
+### Verify the installation by checking the Git version:
+git --version
+### Step 3: Install Docker
+### Docker is a platform for developing, shipping, and running applications inside containers.
+
+### Install prerequisites:
+
+sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
+### Add Docker’s official GPG key:
+
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+### Set up the stable repository:
+
+echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+### Update the package index again and install Docker:
+
+sudo apt update
+sudo apt install docker-ce -y
+### Verify Docker installation:
+
+sudo systemctl status docker
+### Optionally, add your user to the docker group to run Docker without sudo:
+
+sudo usermod -aG docker ${USER}
+### Log out and back in for the group change to take effect.
+
+Step 4: Install Nginx
+### Nginx is a high-performance web server.
+
+### Install Nginx:
+
+sudo apt install nginx -y
+### Start and enable Nginx:
+
+sudo systemctl start nginx
+sudo systemctl enable nginx
+### Verify Nginx installation:
+
+sudo systemctl status nginx
+```
+
+Here's your nginx config file:
+
+```bash
+events {}
+
+http {
+ server {
+ listen 80;
+ server_name name.example;
+
+ location / {
+ proxy_pass http://localhost:3000;
+ proxy_http_version 1.1;
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection 'upgrade';
+ proxy_set_header Host $host;
+ proxy_cache_bypass $http_upgrade;
+ }
+
+ location ~ ^/(ws|upload|files|outputs) {
+ proxy_pass http://localhost:8000;
+ proxy_http_version 1.1;
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection "Upgrade";
+ proxy_set_header Host $host;
+ }
+ }
+}
+```
+
+And the relevant commands:
+
+
+```bash
+vim /etc/nginx/nginx.conf
+### Edit it to reflect above. Then verify all is good with:
+
+sudo nginx -t
+# If there are no errors:
+
+sudo systemctl restart nginx
+
+# Clone .env.example as .env
+# Run from root:
+
+docker-compose up --build
+
+```
\ No newline at end of file
diff --git a/docs/docs/gpt-researcher/gptr/automated-tests.md b/docs/docs/gpt-researcher/gptr/automated-tests.md
new file mode 100644
index 000000000..334e993d5
--- /dev/null
+++ b/docs/docs/gpt-researcher/gptr/automated-tests.md
@@ -0,0 +1,43 @@
+# Automated Tests
+
+## Automated Testing with Github Actions
+
+This repository contains the code for the automated testing of the GPT-Researcher Repo using Github Actions.
+
+The tests are triggered in a docker container which runs the tests via the `pytest` module.
+
+## Running the Tests
+
+You can run the tests:
+
+### Via a docker command
+
+```bash
+docker-compose --profile test run --rm gpt-researcher-tests
+```
+
+### Via a Github Action
+
+
+
+Attaching here the required settings & screenshots on the github repo level:
+
+Step 1: Within the repo, press the "Settings" tab
+
+Step 2: Create a new environment named "tests" (all lowercase)
+
+Step 3: Click into the "tests" environment & add environment secrets of ```OPENAI_API_KEY``` & ```TAVILY_API_KEY```
+
+Get the keys from here:
+
+https://app.tavily.com/sign-in
+
+https://platform.openai.com/api-keys
+
+
+
+
+
+If configured correctly, here's what the Github action should look like when opening a new PR or committing to an open PR:
+
+
\ No newline at end of file
diff --git a/docs/docs/gpt-researcher/config.md b/docs/docs/gpt-researcher/gptr/config.md
similarity index 100%
rename from docs/docs/gpt-researcher/config.md
rename to docs/docs/gpt-researcher/gptr/config.md
diff --git a/docs/docs/gpt-researcher/example.md b/docs/docs/gpt-researcher/gptr/example.md
similarity index 100%
rename from docs/docs/gpt-researcher/example.md
rename to docs/docs/gpt-researcher/gptr/example.md
diff --git a/docs/docs/gpt-researcher/pip-package.md b/docs/docs/gpt-researcher/gptr/pip-package.md
similarity index 98%
rename from docs/docs/gpt-researcher/pip-package.md
rename to docs/docs/gpt-researcher/gptr/pip-package.md
index ef3fbea1d..0e2093ae0 100644
--- a/docs/docs/gpt-researcher/pip-package.md
+++ b/docs/docs/gpt-researcher/gptr/pip-package.md
@@ -1,6 +1,6 @@
# PIP Package
[](https://badge.fury.io/py/gpt-researcher)
-[](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/examples/pip-run.ipynb)
+[](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/docs/docs/examples/pip-run.ipynb)
🌟 **Exciting News!** Now, you can integrate `gpt-researcher` with your apps seamlessly!
diff --git a/docs/docs/gpt-researcher/gptr/querying-the-backend.md b/docs/docs/gpt-researcher/gptr/querying-the-backend.md
new file mode 100644
index 000000000..b499ef6cb
--- /dev/null
+++ b/docs/docs/gpt-researcher/gptr/querying-the-backend.md
@@ -0,0 +1,106 @@
+# Querying the Backend
+
+## Introduction
+
+In this section, we will discuss how to query the GPTR backend server. The GPTR backend server is a Python server that runs the GPTR Python package. The server listens for WebSocket connections and processes incoming messages to generate reports, streaming back logs and results to the client.
+
+An example WebSocket client is implemented in the `gptr-webhook.js` file below.
+
+This function sends a Webhook Message to the GPTR Python backend running on localhost:8000, but this example can also be modified to query a [GPTR Server hosted on Linux](https://docs.gptr.dev/docs/gpt-researcher/getting-started/linux-deployment).
+
+// gptr-webhook.js
+
+```javascript
+
+const WebSocket = require('ws');
+
+let socket = null;
+let responseCallback = null;
+
+async function initializeWebSocket() {
+ if (!socket) {
+ const host = 'localhost:8000';
+ const ws_uri = `ws://${host}/ws`;
+
+ socket = new WebSocket(ws_uri);
+
+ socket.onopen = () => {
+ console.log('WebSocket connection established');
+ };
+
+ socket.onmessage = (event) => {
+ const data = JSON.parse(event.data);
+ console.log('WebSocket data received:', data);
+
+ if (data.content === 'dev_team_result'
+ && data.output.rubber_ducker_thoughts != undefined
+ && data.output.tech_lead_review != undefined) {
+ if (responseCallback) {
+ responseCallback(data.output);
+ responseCallback = null; // Clear callback after use
+ }
+ } else {
+ console.log('Received data:', data);
+ }
+ };
+
+ socket.onclose = () => {
+ console.log('WebSocket connection closed');
+ socket = null;
+ };
+
+ socket.onerror = (error) => {
+ console.error('WebSocket error:', error);
+ };
+ }
+}
+
+async function sendWebhookMessage(message) {
+ return new Promise((resolve, reject) => {
+ if (!socket || socket.readyState !== WebSocket.OPEN) {
+ initializeWebSocket();
+ }
+
+ const data = {
+ task: message,
+ report_type: 'dev_team',
+ report_source: 'web',
+ tone: 'Objective',
+ headers: {},
+ repo_name: 'elishakay/gpt-researcher'
+ };
+
+ const payload = "start " + JSON.stringify(data);
+
+ responseCallback = (response) => {
+ resolve(response); // Resolve the promise with the WebSocket response
+ };
+
+ if (socket.readyState === WebSocket.OPEN) {
+ socket.send(payload);
+ console.log('Message sent:', payload);
+ } else {
+ socket.onopen = () => {
+ socket.send(payload);
+ console.log('Message sent after connection:', payload);
+ };
+ }
+ });
+}
+
+module.exports = {
+ sendWebhookMessage
+};
+```
+
+And here's how you can leverage this helper function:
+
+```javascript
+const { sendWebhookMessage } = require('./gptr-webhook');
+
+async function main() {
+ const message = 'How do I get started with GPT-Researcher Websockets?';
+ const response = await sendWebhookMessage(message);
+ console.log('Response:', response);
+}
+```
\ No newline at end of file
diff --git a/docs/docs/gpt-researcher/troubleshooting.md b/docs/docs/gpt-researcher/gptr/troubleshooting.md
similarity index 99%
rename from docs/docs/gpt-researcher/troubleshooting.md
rename to docs/docs/gpt-researcher/gptr/troubleshooting.md
index d02092c3f..86a50e1aa 100644
--- a/docs/docs/gpt-researcher/troubleshooting.md
+++ b/docs/docs/gpt-researcher/gptr/troubleshooting.md
@@ -1,4 +1,5 @@
# Troubleshooting
+
We're constantly working to provide a more stable version. If you're running into any issues, please first check out the resolved issues or ask us via our [Discord community](https://discord.gg/QgZXvJAccX).
### model: gpt-4 does not exist
diff --git a/docs/docs/gpt-researcher/llms.md b/docs/docs/gpt-researcher/llms/llms.md
similarity index 97%
rename from docs/docs/gpt-researcher/llms.md
rename to docs/docs/gpt-researcher/llms/llms.md
index 13fcd7b15..99829403b 100644
--- a/docs/docs/gpt-researcher/llms.md
+++ b/docs/docs/gpt-researcher/llms/llms.md
@@ -1,5 +1,6 @@
# Configure LLM
-As described in the [introduction](/docs/gpt-researcher/config), the default LLM is OpenAI due to its superior performance and speed.
+
+As described in the [introduction](/docs/gpt-researcher/gptr/config), the default LLM is OpenAI due to its superior performance and speed.
With that said, GPT Researcher supports various open/closed source LLMs, and you can easily switch between them by adding the `LLM_PROVIDER` env variable and corresponding configuration params.
Current supported LLMs are `openai`, `google` (gemini), `azure_openai`, `ollama`, `anthropic`, `mistral`, `huggingface` and `groq`.
diff --git a/docs/docs/gpt-researcher/llms/running-with-ollama.md b/docs/docs/gpt-researcher/llms/running-with-ollama.md
new file mode 100644
index 000000000..cbbe5509b
--- /dev/null
+++ b/docs/docs/gpt-researcher/llms/running-with-ollama.md
@@ -0,0 +1,120 @@
+# Running with Ollama
+
+Ollama is a platform that allows you to deploy and manage custom language models. This guide will walk you through deploying a custom language model on Ollama.
+
+Read on to understand how to install a Custom LLM with the Ollama WebUI, and how to query it with GPT-Researcher.
+
+
+## Fetching the Desired LLM Models
+
+After deploying Ollama WebUI, you'll want to enter the [Open WebUI Admin App](https://github.com/open-webui/open-webui/tree/main) & download a custom LLM.
+
+Choose a model from [Ollama's Library of LLM's](https://ollama.com/library?sort=popular)
+
+Paste the model name & size into the Web UI:
+
+
+
+For our example, let's choose to download the `qwen2:1.5b` model.
+
+This model now automatically becomes available via your Server's out-of-the-box API - we'll leverage it within our GPT-Researcher .env file in the next step.
+
+
+## Querying your Custom LLM with GPT-Researcher
+
+If you deploy ollama locally, a .env like so, should enable powering GPT-Researcher with Ollama:
+
+```bash
+OPENAI_API_KEY="123"
+OPENAI_API_BASE="http://127.0.0.1:11434/v1"
+OLLAMA_BASE_URL="http://127.0.0.1:11434/"
+FAST_LLM_MODEL=qwen2:1.5b
+SMART_LLM_MODEL=qwen2:1.5b
+OLLAMA_EMBEDDING_MODEL=all-minilm:22m
+LLM_PROVIDER=openai
+EMBEDDING_PROVIDER=ollama
+```
+
+Replace FAST_LLM_MODEL & SMART_LLM_MODEL with the model you downloaded from the Elestio Web UI in the previous step.
+
+
+## Run LLM Test Script for GPTR
+
+And here's a custom python script you can use to query your custom LLM:
+
+```python
+
+import os
+import asyncio
+import logging
+logging.basicConfig(level=logging.DEBUG)
+from gpt_researcher.llm_provider.generic import GenericLLMProvider
+from gpt_researcher.utils.llm import get_llm
+
+# Set up environment variables
+os.environ["LLM_PROVIDER"] = "ollama"
+os.environ["OLLAMA_BASE_URL"] = "https://ollama-ug3qr-u21899.vm.elestio.app:57987"
+os.environ["FAST_LLM_MODEL"] = "llama3.1"
+
+# Create the GenericLLMProvider instance
+llm_provider = get_llm(
+ "ollama",
+ base_url=os.environ["OLLAMA_BASE_URL"],
+ model=os.environ["FAST_LLM_MODEL"],
+ temperature=0.7,
+ max_tokens=2000,
+ verify_ssl=False # Add this line
+)
+
+# Test the connection with a simple query
+messages = [{"role": "user", "content": "sup?"}]
+
+async def test_ollama():
+ try:
+ response = await llm_provider.get_chat_response(messages, stream=False)
+ print("Ollama response:", response)
+ except Exception as e:
+ print(f"Error: {e}")
+
+# Run the async function
+asyncio.run(test_ollama())
+
+```
+
+Replace `OLLAMA_BASE_URL` with the URL of your Ollama instance, and `FAST_LLM_MODEL` with the model you downloaded from the Ollama Web UI.
+
+Run the script to test the connection with your custom LLM.
+
+
+## Deploy Ollama on Elestio
+
+Elestio is a platform that allows you to deploy and manage custom language models. This guide will walk you through deploying a custom language model on Elestio.
+
+You can deploy an [Open WebUI](https://github.com/open-webui/open-webui/tree/main) server with [Elestio](https://elest.io/open-source/ollama)
+
+Here's an example .env file that will enable powering GPT-Researcher with Elestio:
+
+```bash
+OPENAI_API_KEY="123"
+OPENAI_API_BASE="https://.vm.elestio.app:57987/v1"
+OLLAMA_BASE_URL="https://.vm.elestio.app:57987/"
+FAST_LLM_MODEL=qwen2:1.5b
+SMART_LLM_MODEL=qwen2:1.5b
+OLLAMA_EMBEDDING_MODEL=all-minilm:22m
+LLM_PROVIDER=openai
+EMBEDDING_PROVIDER=ollama
+```
+
+#### Disable Elestio Authentication or Add Auth Headers
+
+To remove the basic auth you have to follow the below steps:
+Go to your service -> Security -> at last Nginx -> in that find the below code:
+
+```bash
+auth_basic "Authentication";
+
+auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
+```
+
+Comment these both these lines out and click the button "Update & Restart" to reflect the changes.
+
diff --git a/docs/docs/gpt-researcher/langgraph.md b/docs/docs/gpt-researcher/multi_agents/langgraph.md
similarity index 99%
rename from docs/docs/gpt-researcher/langgraph.md
rename to docs/docs/gpt-researcher/multi_agents/langgraph.md
index a3b6f5e7b..f450931ce 100644
--- a/docs/docs/gpt-researcher/langgraph.md
+++ b/docs/docs/gpt-researcher/multi_agents/langgraph.md
@@ -1,4 +1,5 @@
# LangGraph
+
[LangGraph](https://python.langchain.com/docs/langgraph) is a library for building stateful, multi-actor applications with LLMs.
This example uses Langgraph to automate the process of an in depth research on any given topic.
@@ -119,7 +120,7 @@ It comes with loads of added features, such as:
### Run the NextJS React App with Docker
-> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started#try-it-with-docker)
+> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
> **Step 2** - Clone the '.env.example' file, add your API Keys to the cloned file and save the file as '.env'
diff --git a/docs/docs/gpt-researcher/retrievers.md b/docs/docs/gpt-researcher/search-engines/retrievers.md
similarity index 100%
rename from docs/docs/gpt-researcher/retrievers.md
rename to docs/docs/gpt-researcher/search-engines/retrievers.md
diff --git a/docs/docs/gpt-researcher/roadmap.md b/docs/docs/roadmap.md
similarity index 100%
rename from docs/docs/gpt-researcher/roadmap.md
rename to docs/docs/roadmap.md
diff --git a/docs/docs/welcome.md b/docs/docs/welcome.md
index e53121762..3831b2fe6 100644
--- a/docs/docs/welcome.md
+++ b/docs/docs/welcome.md
@@ -10,4 +10,4 @@ Quickly accessing relevant and trustworthy information is more crucial than ever
This is why we've built the trending open source **[GPT Researcher](https://github.com/assafelovic/gpt-researcher)**. GPT Researcher is an autonomous agent that takes care of the tedious task of research for you, by scraping, filtering and aggregating over 20+ web sources per a single research task.
-To learn more about GPT Researcher, check out the [documentation page](/docs/gpt-researcher/introduction).
+To learn more about GPT Researcher, check out the [documentation page](/docs/gpt-researcher/getting-started/introduction).
diff --git a/docs/sidebars.js b/docs/sidebars.js
index f638e0132..448b8a3a7 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -12,31 +12,71 @@
module.exports = {
docsSidebar: [
'welcome',
+ {
+ type: 'category',
+ label: 'Getting Started',
+ collapsible: true,
+ collapsed: false,
+ items: [
+ 'gpt-researcher/getting-started/introduction',
+ 'gpt-researcher/getting-started/getting-started-with-docker',
+ 'gpt-researcher/getting-started/getting-started',
+ 'gpt-researcher/getting-started/linux-deployment',
+ ]
+ },
{
type: 'category',
label: 'GPT Researcher',
collapsible: true,
collapsed: false,
items: [
- 'gpt-researcher/introduction',
- 'gpt-researcher/getting-started',
- 'gpt-researcher/pip-package',
- 'gpt-researcher/frontend',
- 'gpt-researcher/example',
- 'gpt-researcher/troubleshooting',
+ 'gpt-researcher/gptr/pip-package',
+ 'gpt-researcher/gptr/example',
+ 'gpt-researcher/gptr/querying-the-backend',
+ 'gpt-researcher/gptr/config',
+ 'gpt-researcher/gptr/automated-tests',
+ 'gpt-researcher/gptr/troubleshooting',
+ ],
+ },
+ {
+ type: 'category',
+ label: 'Frontend',
+ collapsible: true,
+ collapsed: false,
+ items: [
+ 'gpt-researcher/frontend/frontend',
+ 'gpt-researcher/frontend/playing-with-webhooks',
],
},
{
type: 'category',
- label: 'Customization',
+ label: 'Custom Context',
+ collapsible: true,
+ collapsed: false,
+ items: [
+ 'gpt-researcher/context/tailored-research',
+ 'gpt-researcher/context/local-docs',
+ 'gpt-researcher/context/filtering-by-domain',
+ 'gpt-researcher/context/vector-stores',
+ ]
+ },
+ {
+ type: 'category',
+ label: 'Large Language Models',
+ collapsible: true,
+ collapsed: false,
+ items: [
+ 'gpt-researcher/llms/llms',
+ 'gpt-researcher/llms/running-with-ollama'
+ ]
+ },
+ {
+ type: 'category',
+ label: 'Search Engines',
collapsible: true,
collapsed: true,
items: [
- 'gpt-researcher/config',
- 'gpt-researcher/tailored-research',
- 'gpt-researcher/retrievers',
- 'gpt-researcher/llms',
- 'gpt-researcher/vector-stores',
+ 'gpt-researcher/search-engines/retrievers',
]
},
{
@@ -45,11 +85,13 @@
collapsible: true,
collapsed: true,
items: [
- 'gpt-researcher/langgraph',
+ 'gpt-researcher/multi_agents/langgraph',
]
},
{'Examples': [{type: 'autogenerated', dirName: 'examples'}]},
'contribute',
+ 'roadmap',
+ 'faq',
],
// pydoc-markdown auto-generated markdowns from docstrings
referenceSideBar: [require("./docs/reference/sidebar.json")]
diff --git a/docs/src/components/HomepageFeatures.js b/docs/src/components/HomepageFeatures.js
index 91281997e..ffd2a6b4f 100644
--- a/docs/src/components/HomepageFeatures.js
+++ b/docs/src/components/HomepageFeatures.js
@@ -27,7 +27,7 @@ const FeatureList = [
{
title: 'Multi-Agent Assistant',
Svg: require('../../static/img/multi-agent.png').default,
- docLink: './docs/gpt-researcher/langgraph',
+ docLink: './docs/gpt-researcher/multi_agents',
description: (
<>
Learn how a team of AI agents can work together to conduct research on a given topic, from planning to publication.
diff --git a/multi_agents/README.md b/multi_agents/README.md
index 7a942e30e..3004c6beb 100644
--- a/multi_agents/README.md
+++ b/multi_agents/README.md
@@ -105,43 +105,3 @@ langgraph up
```
From there, see documentation [here](https://github.com/langchain-ai/langgraph-example) on how to use the streaming and async endpoints, as well as the playground.
-
-## NextJS Frontend App
-
-The React app (located in `frontend` directory) is our Frontend 2.0 which we hope will enable us to display the robustness of the backend on the frontend, as well.
-
-It comes with loads of added features, such as:
- - a drag-n-drop user interface for uploading and deleting files to be used as local documents by GPTResearcher.
- - a GUI for setting your GPTR environment variables.
- - the ability to trigger the multi_agents flow via the Backend Module or Langgraph Cloud Host (currently in closed beta).
- - stability fixes
- - and more coming soon!
-
-### Run the NextJS React App with Docker
-
-> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started#try-it-with-docker)
-
-> **Step 2** - Clone the '.env.example' file, add your API Keys to the cloned file and save the file as '.env'
-
-> **Step 3** - Within the docker-compose file comment out services that you don't want to run with Docker.
-
-```bash
-$ docker compose up --build
-```
-
-> **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
- - the Python server running on localhost:8000
- - the React app running on localhost:3000
-
-Visit localhost:3000 on any browser and enjoy researching!
-
-
-### Run the NextJS React App with NPM
-
-```bash
-cd frontend
-nvm install 18.17.0
-nvm use v18.17.0
-npm install --legacy-peer-deps
-npm run dev
-```
diff --git a/multi_agents/frontend/next-env.d.ts b/multi_agents/frontend/next-env.d.ts
new file mode 100644
index 000000000..4f11a03dc
--- /dev/null
+++ b/multi_agents/frontend/next-env.d.ts
@@ -0,0 +1,5 @@
+///
+///
+
+// NOTE: This file should not be edited
+// see https://nextjs.org/docs/basic-features/typescript for more information.