crash because of missing bs4 dependency in version 2.12

[hansvdam]

System Info

Langchain: 2.12
commit: #6455
version 2.11 does niet have this

Who can help?

@rlancemartin, @eyurtsev

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

workaround: install bs4 manually (pip install bs4)

from langchain.agents import initialize_agent, AgentType
leads to:

  File "//main.py", line 17, in <module>
    from langchain.agents import initialize_agent, AgentType
  File "/usr/local/lib/python3.11/site-packages/langchain/__init__.py", line 6, in <module>
    from langchain.agents import MRKLChain, ReActChain, SelfAskWithSearchChain
  File "/usr/local/lib/python3.11/site-packages/langchain/agents/__init__.py", line 2, in <module>
    from langchain.agents.agent import (
  File "/usr/local/lib/python3.11/site-packages/langchain/agents/agent.py", line 16, in <module>
    from langchain.agents.tools import InvalidTool
  File "/usr/local/lib/python3.11/site-packages/langchain/agents/tools.py", line 8, in <module>
    from langchain.tools.base import BaseTool, Tool, tool
  File "/usr/local/lib/python3.11/site-packages/langchain/tools/__init__.py", line 3, in <module>
    from langchain.tools.arxiv.tool import ArxivQueryRun
  File "/usr/local/lib/python3.11/site-packages/langchain/tools/arxiv/tool.py", line 12, in <module>
    from langchain.utilities.arxiv import ArxivAPIWrapper
  File "/usr/local/lib/python3.11/site-packages/langchain/utilities/__init__.py", line 3, in <module>
    from langchain.utilities.apify import ApifyWrapper
  File "/usr/local/lib/python3.11/site-packages/langchain/utilities/apify.py", line 5, in <module>
    from langchain.document_loaders import ApifyDatasetLoader
  File "/usr/local/lib/python3.11/site-packages/langchain/document_loaders/__init__.py", line 97, in <module>
    from langchain.document_loaders.recursive_url_loader import RecusiveUrlLoader
  File "/usr/local/lib/python3.11/site-packages/langchain/document_loaders/recursive_url_loader.py", line 5, in <module>
    from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'

requirements.txt:

openai==0.27.8
fastapi==0.97.0
websockets==11.0.3
pydantic==1.10.9
langchain==0.0.212
uvicorn[standard]
jinja2
lancedb==0.1.8
itsdangerous
tiktoken==0.4.0

Expected behavior

I think
from bs4 import BeautifulSoup
in recursive_url_loader.py
should have been a local import

[dosubot]

Answer generated by a 🤖

Answer

The crash is caused by a missing bs4 dependency in LangChain version 2.12. To resolve this issue, you can add the missing dependency to the requirements.txt file:

openai==0.27.8
fastapi==0.97.0
websockets==11.0.3
pydantic==1.10.9
langchain==0.0.212
uvicorn[standard]
jinja2
lancedb==0.1.8
itsdangerous
tiktoken==0.4.0
bs4

Then, run pip install -r requirements.txt to install the missing dependency. This should resolve the ModuleNotFoundError for bs4 (Source: langchain/document_loaders/parsers/html/init.py).

Sources

• langchain/document_loaders/web_base.py
• docs/snippets/modules/chains/popular/sqlite.mdx
• langchain/document_loaders/parsers/html/init.py
• langchain/agents/load_tools.py
• docs/extras/modules/agents/tools/integrations/bash.ipynb
• langchain/agents/agent.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[rlancemartin]

Yes, it should be a local import.

I can push a fix shortly.

[rlancemartin]

Fix is up: #6693