assafelovic · assafelovic · Sep 13, 2024 · Aug 27, 2024 · Aug 27, 2024 · Aug 27, 2024
diff --git a/.env.example b/.env.example
@@ -1,2 +1,3 @@
 OPENAI_API_KEY=
-TAVILY_API_KEY=
+TAVILY_API_KEY=
+DOC_PATH=./my-docs
diff --git a/README-ko_KR.md b/README-ko_KR.md
@@ -128,7 +128,7 @@ python -m uvicorn main:app --reload
 
 <br />
 
-**[Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started#poetry) 또는 [가상 환경](https://docs.gptr.dev/docs/gpt-researcher/getting-started#virtual-environment)에 대해 배우고 싶다면, [문서](https://docs.gptr.dev/docs/gpt-researcher/getting-started)를 참조하세요.**
+**[Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#poetry) 또는 [가상 환경](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#virtual-environment)에 대해 배우고 싶다면, [문서](https://docs.gptr.dev/docs/gpt-researcher/getting-started)를 참조하세요.**
 
 ### PIP 패키지로 실행하기
 ```bash
@@ -148,11 +148,11 @@ report = await researcher.write_report()
 ...
 ```
 
-**더 많은 예제와 구성 옵션은 [PIP 문서](https://docs.gptr.dev/docs/gpt-researcher/pip-package)를 참조하세요.**
+**더 많은 예제와 구성 옵션은 [PIP 문서](https://docs.gptr.dev/docs/gpt-researcher/gptr/pip-package)를 참조하세요.**
 
 ## Docker로 실행
 
-> **1단계** - [Docker 설치](https://docs.gptr.dev/docs/gpt-researcher/getting-started#try-it-with-docker)
+> **1단계** - [Docker 설치](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
 
 > **2단계** - `.env.example` 파일을 복사하고 API 키를 추가한 후, 파일을 `.env`로 저장하세요.
 
@@ -180,7 +180,7 @@ export DOC_PATH="./my-docs"
 
 2단계:
  - 프론트엔드 앱을 localhost:8000에서 실행 중이라면, "Report Source" 드롭다운 옵션에서 "My Documents"를 선택하세요.
- - GPT Researcher를 [PIP 패키지](https://docs.tavily.com/docs/gpt-researcher/pip-package)로 실행 중이라면, `report_source` 인수를 "documents"로 설정하여 `GPTResearcher` 클래스를 인스턴스화하세요. [코드 예제](https://docs.tavily.com/docs/gpt-researcher/tailored-research)를 참조하세요.
+ - GPT Researcher를 [PIP 패키지](https://docs.tavily.com/docs/gpt-researcher/pip-package)로 실행 중이라면, `report_source` 인수를 "local"로 설정하여 `GPTResearcher` 클래스를 인스턴스화하세요. [코드 예제](https://docs.gptr.dev/docs/gpt-researcher/context/tailored-research)를 참조하세요.
 
 ## 👪 다중 에이전트 어시스턴트
 
@@ -190,7 +190,7 @@ LangGraph를 사용하면 여러 에이전트의 전문 기술을 활용하여
 
 평균 실행은 5-6 페이지 분량의 연구 보고서를 PDF, Docx, Markdown 형식으로 생성합니다.
 
-[여기](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents)에서 확인하거나 [문서](https://docs.gptr.dev/docs/gpt-researcher/langgraph)에서 자세한 내용을 참조하세요.
+[여기](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents)에서 확인하거나 [문서](https://docs.gptr.dev/docs/gpt-researcher/multi_agents)에서 자세한 내용을 참조하세요.
 
 ## 🖥️ 프론트엔드 애플리케이션
 

diff --git a/README.md b/README.md
@@ -17,7 +17,7 @@
 <!--[![License](https://img.shields.io/github/license/assafelovic/gpt-researcher?color=FF5A5F)](https://github.com/assafelovic/gpt-researcher/blob/main/LICENSE)-->
 [![PyPI version](https://img.shields.io/pypi/v/gpt-researcher?logo=pypi&logoColor=white&style=flat)](https://badge.fury.io/py/gpt-researcher)
 ![GitHub Release](https://img.shields.io/github/v/release/assafelovic/gpt-researcher?style=flat&logo=github)
-[![Open In Colab](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=grey&color=yellow&label=%20&style=flat&logoSize=40)](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/examples/pip-run.ipynb)
+[![Open In Colab](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=grey&color=yellow&label=%20&style=flat&logoSize=40)](https://colab.research.google.com/github/assafelovic/gpt-researcher/blob/master/docs/docs/examples/pip-run.ipynb)
 [![Docker Image Version](https://img.shields.io/docker/v/elestio/gpt-researcher/latest?arch=amd64&style=flat&logo=docker&logoColor=white&color=1D63ED)](https://hub.docker.com/r/gptresearcher/gpt-researcher)
 [![Twitter Follow](https://img.shields.io/twitter/follow/assaf_elovic?style=social)](https://twitter.com/assaf_elovic)
 <!--[![Twitter](https://img.shields.io/badge/Twitter-000000?style=flat-square&logo=x&logoColor=white)](https://twitter.com/assaf_elovic)-->
@@ -108,7 +108,7 @@ export TAVILY_API_KEY={Your Tavily API Key here}
 For a more permanent setup, create a `.env` file in the current `gpt-researcher` directory and input the env vars (without `export`).
 
 - The default LLM is [GPT](https://platform.openai.com/docs/guides/gpt), but you can use other LLMs such as `claude`, `ollama3`, `gemini`, `mistral` and more. To learn how to change the LLM provider, see the [LLMs documentation](https://docs.gptr.dev/docs/gpt-researcher/llms) page. Please note: this project is optimized for OpenAI GPT models.
-- The default retriever is [Tavily](https://app.tavily.com), but you can refer to other retrievers such as `duckduckgo`, `google`, `bing`, `serper`, `searx`, `arxiv`, `exa` and more. To learn how to change the search provider, see the [retrievers documentation](https://docs.gptr.dev/docs/gpt-researcher/retrievers) page.
+- The default retriever is [Tavily](https://app.tavily.com), but you can refer to other retrievers such as `duckduckgo`, `google`, `bing`, `serper`, `searx`, `arxiv`, `exa` and more. To learn how to change the search provider, see the [retrievers documentation](https://docs.gptr.dev/docs/gpt-researcher/search-engines/retrievers) page.
 
 ### Quickstart
 
@@ -128,7 +128,7 @@ python -m uvicorn main:app --reload
 
 <br />
 
-**To learn how to get started with [Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started#poetry) or a [virtual environment](https://docs.gptr.dev/docs/gpt-researcher/getting-started#virtual-environment) check out the [documentation](https://docs.gptr.dev/docs/gpt-researcher/getting-started) page.**
+**To learn how to get started with [Poetry](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#poetry) or a [virtual environment](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started#virtual-environment) check out the [documentation](https://docs.gptr.dev/docs/gpt-researcher/getting-started) page.**
 
 ### Run as PIP package
 ```bash
@@ -148,21 +148,27 @@ report = await researcher.write_report()
 ...
 ```
 
-**For more examples and configurations, please refer to the [PIP documentation](https://docs.gptr.dev/docs/gpt-researcher/pip-package) page.**
+**For more examples and configurations, please refer to the [PIP documentation](https://docs.gptr.dev/docs/gpt-researcher/gptr/pip-package) page.**
 
 
 ## Run with Docker
 
-> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started#try-it-with-docker)
+> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
 
 > **Step 2** - Clone the '.env.example' file, add your API Keys to the cloned file and save the file as '.env'
 
 > **Step 3** - Within the docker-compose file comment out services that you don't want to run with Docker.
 
 ```bash
-$ docker-compose up --build
+docker-compose up --build
 ```
 
+If that doesn't work, try running it without the dash:
+```bash
+docker compose up --build
+```
+
+
 > **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
  - the Python server running on localhost:8000<br>
  - the React app running on localhost:3000<br>
@@ -183,7 +189,8 @@ export DOC_PATH="./my-docs"
 
 Step 2: 
  - If you're running the frontend app on localhost:8000, simply select "My Documents" from the the "Report Source" Dropdown Options.
- - If you're running GPT Researcher with the [PIP package](https://docs.tavily.com/docs/gpt-researcher/pip-package), pass the `report_source` argument as "documents" when you instantiate the `GPTResearcher` class [code sample here](https://docs.tavily.com/docs/gpt-researcher/tailored-research).
+ - If you're running GPT Researcher with the [PIP package](https://docs.tavily.com/docs/gpt-researcher/pip-package), pass the `report_source` argument as "local" when you instantiate the `GPTResearcher` class [code sample here](https://docs.gptr.dev/docs/gpt-researcher/context/tailored-research).
+
 
 ## 👪 Multi-Agent Assistant
 As AI evolves from prompt engineering and RAG to multi-agent systems, we're excited to introduce our new multi-agent assistant built with [LangGraph](https://python.langchain.com/v0.1/docs/langgraph/).
@@ -192,7 +199,7 @@ By using LangGraph, the research process can be significantly improved in depth
 
 An average run generates a 5-6 page research report in multiple formats such as PDF, Docx and Markdown.
 
-Check it out [here](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents) or head over to our [documentation](https://docs.gptr.dev/docs/gpt-researcher/langgraph) for more information.
+Check it out [here](https://github.com/assafelovic/gpt-researcher/tree/master/multi_agents) or head over to our [documentation](https://docs.gptr.dev/docs/gpt-researcher/multi_agents/langgraph) for more information.
 
 ## 🖥️ Frontend Applications
 

diff --git a/examples/pip-run.ipynb → docs/docs/examples/pip-run.ipynb b/examples/pip-run.ipynb → docs/docs/examples/pip-run.ipynb
diff --git a/examples/sample_report.py → docs/docs/examples/sample_report.py b/examples/sample_report.py → docs/docs/examples/sample_report.py
diff --git a/examples/sample_sources_only.py → docs/docs/examples/sample_sources_only.py b/examples/sample_sources_only.py → docs/docs/examples/sample_sources_only.py
diff --git a/docs/docs/faq.md b/docs/docs/faq.md
@@ -4,22 +4,29 @@
 It really depends on what you're aiming for. 
 
 If you're looking to connect your AI application to the internet with Tavily tailored API, check out the [Tavily API](https://docs.tavily.com/docs/tavily-api/introductionn) documentation. 
-If you're looking to build and deploy our open source autonomous research agent GPT Researcher, please see [GPT Researcher](/docs/gpt-researcher/introduction) documentation.
+If you're looking to build and deploy our open source autonomous research agent GPT Researcher, please see [GPT Researcher](/docs/gpt-researcher/getting-started/introduction) documentation.
 You can also check out demos and examples for inspiration [here](/docs/examples/examples).
+
 ### What is GPT Researcher?
+
 GPT Researcher is a popular open source autonomous research agent that takes care of the tedious task of research for you, by scraping, filtering and aggregating over 20+ web sources per a single research task.
 
 GPT Researcher is built with best practices for leveraging LLMs (prompt engineering, RAG, chains, embeddings, etc), and is optimized for quick and efficient research. It is also fully customizable and can be tailored to your specific needs.
 
-To learn more about GPT Researcher, check out the [documentation page](/docs/gpt-researcher/introduction).
+To learn more about GPT Researcher, check out the [documentation page](/docs/gpt-researcher/getting-started/introduction).
+
 ### How much does each research run cost?
+
 A research task using GPT Researcher costs around $0.01 per a single run (for GPT-4 usage). We're constantly optimizing LLM calls to reduce costs and improve performance. 
+
 ### How do you ensure the report is factual and accurate?
+
 we do our best to ensure that the information we provide is factual and accurate. We do this by using multiple sources, and by using proprietary AI to score and rank the most relevant and accurate information. We also use proprietary AI to filter out irrelevant information and sources.
 
 Lastly, by using RAG and other techniques, we ensure that the information is relevant to the context of the research task, leading to more accurate generative AI content and reduced hallucinations.
 
 ### What are your plans for the future?
+
 We're constantly working on improving our products and services. We're currently working on improving our search API together with design partners, and adding more data sources to our search engine. We're also working on improving our research agent GPT Researcher, and adding more features to it while growing our amazing open source community.
 
 If you're interested in our roadmap or looking to collaborate, check out our [roadmap page](https://trello.com/b/3O7KBePw/gpt-researcher-roadmap). 

diff --git a/docs/docs/gpt-researcher/context/filtering-by-domain.md b/docs/docs/gpt-researcher/context/filtering-by-domain.md
@@ -0,0 +1,22 @@
+# Filtering by Domain
+
+If you set Google as a Retriever, you can filter web results by site.
+
+For example, set in the query param you pass the GPTResearcher class instance: `query="site:linkedin.com a python web developer to implement my custom gpt-researcher flow"` will limit the results to linkedin.com
+
+> **Step 1** -  Set these environment variables with a .env file in the root folder
+
+TAVILY_API_KEY=
+LANGCHAIN_TRACING_V2=true
+LANGCHAIN_API_KEY=
+OPENAI_API_KEY=
+DOC_PATH=./my-docs
+RETRIEVER=google
+GOOGLE_API_KEY=
+GOOGLE_CX_KEY=
+
+> **Step 2** -  from the root project run:
+
+docker-compose up -- build
+
+> **Step 3** -  from the frontend input box in localhost:3000, you can append any google search filter (such as filtering by domain names)
diff --git a/docs/docs/gpt-researcher/context/local-docs.md b/docs/docs/gpt-researcher/context/local-docs.md
@@ -0,0 +1,22 @@
+# 📄 Research on Local Documents
+
+## Just Local Docs
+
+You can instruct the GPT Researcher to run research tasks based on your local documents. Currently supported file formats are: PDF, plain text, CSV, Excel, Markdown, PowerPoint, and Word documents.
+
+Step 1: Add the env variable `DOC_PATH` pointing to the folder where your documents are located.
+
+```bash
+export DOC_PATH="./my-docs"
+```
+
+Step 2: 
+ - If you're running the frontend app on localhost:8000, simply select "My Documents" from the the "Report Source" Dropdown Options.
+ - If you're running GPT Researcher with the [PIP package](https://docs.tavily.com/docs/gpt-researcher/gptr/pip-package), pass the `report_source` argument as "local" when you instantiate the `GPTResearcher` class [code sample here](https://docs.gptr.dev/docs/gpt-researcher/context/tailored-research).
+
+## Local Docs + Web (Hybrid)
+
+![GPT Researcher hybrid research](./gptr-hybrid.png)
+
+Check out the blog post on [Hybrid Research](https://docs.gptr.dev/blog/gptr-hybrid) to learn more about how to combine local documents with web research.
+```
diff --git a/.../docs/gpt-researcher/tailored-research.md → ...t-researcher/context/tailored-research.md b/.../docs/gpt-researcher/tailored-research.md → ...t-researcher/context/tailored-research.md
@@ -1,4 +1,5 @@
 # Tailored Research
+
 The GPT Researcher package allows you to tailor the research to your needs such as researching on specific sources or local documents, and even specify the agent prompt instruction upon which the research is conducted.
 
 ### Research on Specific Sources 📚
@@ -88,7 +89,7 @@ You can combine the above methods to conduct hybrid research. For example, you c
 Simply provide the sources and set the `report_source` argument as `"hybrid"` and watch the magic happen.
 
 Please note! You should set the proper retrievers for the web sources and doc path for local documents for this to work.
-To lean more about retrievers check out the [Retrievers](https://docs.gptr.dev/docs/gpt-researcher/retrievers) documentation.
+To lean more about retrievers check out the [Retrievers](https://docs.gptr.dev/docs/gpt-researcher/search-engines/retrievers) documentation.
 
 
 ### Research on LangChain Documents 🦜️🔗

diff --git a/docs/docs/gpt-researcher/vector-stores.md → ...s/gpt-researcher/context/vector-stores.md b/docs/docs/gpt-researcher/vector-stores.md → ...s/gpt-researcher/context/vector-stores.md
@@ -1,4 +1,5 @@
 # Vector Stores
+
 The GPT Researcher package allows you to integrate with existing langchain vector stores that have been populated.
 For a complete list of supported langchain vector stores, please refer to this [link](https://python.langchain.com/v0.2/docs/integrations/vectorstores/).
 

diff --git a/docs/docs/gpt-researcher/frontend.md → .../docs/gpt-researcher/frontend/frontend.md b/docs/docs/gpt-researcher/frontend.md → .../docs/gpt-researcher/frontend/frontend.md
@@ -2,7 +2,45 @@
 
 This frontend project aims to enhance the user experience of GPT-Researcher, providing an intuitive and efficient interface for automated research. It offers two deployment options to suit different needs and environments.
 
-## Option 1: Static Frontend (FastAPI)
+View a Product Tutorial here: [GPT-Researcher Frontend Tutorial](https://www.youtube.com/watch?v=hIZqA6lPusk)
+
+
+## NextJS Frontend App
+
+The React app (located in the `frontend` directory) is our Frontend 2.0 which we hope will enable us to display the robustness of the backend on the frontend, as well.
+
+It comes with loads of added features, such as: 
+ - a drag-n-drop user interface for uploading and deleting files to be used as local documents by GPTResearcher.
+ - a GUI for setting your GPTR environment variables.
+ - the ability to trigger the multi_agents flow via the Backend Module or Langgraph Cloud Host (currently in closed beta).
+ - stability fixes
+ - and more coming soon!
+
+### Run the NextJS React App with Docker
+
+> **Step 1** - [Install Docker](https://docs.gptr.dev/docs/gpt-researcher/getting-started/getting-started-with-docker)
+
+> **Step 2** - Clone the '.env.example' file, add your API Keys to the cloned file and save the file as '.env'
+
+> **Step 3** - Within the docker-compose file comment out services that you don't want to run with Docker.
+
+```bash
+docker compose up --build
+```
+
+If that doesn't work, try running it without the dash:
+```bash
+docker compose up --build
+```
+
+> **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
+ - the Python server running on localhost:8000<br>
+ - the React app running on localhost:3000<br>
+
+Visit localhost:3000 on any browser and enjoy researching!
+
+
+## Other Options: 1: Static Frontend (FastAPI)
 
 A lightweight solution using FastAPI to serve static files.
 
@@ -28,7 +66,7 @@ A lightweight solution using FastAPI to serve static files.
 <iframe height="400" width="700" src="https://github.com/assafelovic/gpt-researcher/assets/13554167/dd6cf08f-b31e-40c6-9907-1915f52a7110" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
 
 
-## Option 2: NextJS Frontend
+## Yet Another Option: Running NextJS Frontend via CLI
 
 A more robust solution with enhanced features and performance.
 

diff --git a/docs/docs/gpt-researcher/frontend/playing-with-webhooks.md b/docs/docs/gpt-researcher/frontend/playing-with-webhooks.md
@@ -0,0 +1,23 @@
+# Playing with Webhooks
+
+The GPTR Frontend is powered by Webhooks streaming back from the Backend. This allows for real-time updates on the status of your research tasks, as well as the ability to interact with the Backend directly from the Frontend.
+
+
+## Inspecting Webhooks
+
+When running reports via the frontend, you can inspect the websocket messages in the Network Tab.
+
+Here's how: 
+
+![image](https://github.com/user-attachments/assets/15fcb5a4-77ea-4b3b-87d7-55d4b6f80095)
+
+
+### Am I polling the right URL?
+
+If you're concerned that your frontend isn't hitting the right API Endpoint, you can check the URL in the Network Tab.
+
+Click into the WS request & go to the "Headers" tab
+
+![image](https://github.com/user-attachments/assets/dbd58c1d-3506-411a-852b-e1b133b6f5c8)
+
+For debugging, have a look at the <a href="https://github.com/assafelovic/gpt-researcher/blob/master/frontend/nextjs/helpers/getHost.js">getHost function.</a>
diff --git a/docs/docs/gpt-researcher/getting-started/getting-started-with-docker.md b/docs/docs/gpt-researcher/getting-started/getting-started-with-docker.md
@@ -0,0 +1,28 @@
+# Docker: Quickstart
+
+> **Step 1** - Install & Open Docker Desktop
+
+Follow instructions at https://www.docker.com/products/docker-desktop/
+
+
+> **Step 2** - [Follow this flow](https://www.youtube.com/watch?v=x1gKFt_6Us4)
+
+This mainly includes cloning the '.env.example' file, adding your API Keys to the cloned file and saving the file as '.env'
+
+> **Step 3** - Within root, run with Docker.
+
+```bash
+docker-compose up --build
+```
+
+If that doesn't work, try running it without the dash:
+```bash
+docker compose up --build
+```
+
+> **Step 4** - By default, if you haven't uncommented anything in your docker-compose file, this flow will start 2 processes:
+ - the Python server running on localhost:8000<br>
+ - the React app running on localhost:3000<br>
+
+Visit localhost:3000 on any browser and enjoy researching!
+