-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
Describe the bug
I set search_engine as duckduckgo. But it still uses Google Search engine and it seems that Google search cannot be scraped.
`
graph_config = {
"llm_model": "ollama/llama3",
"llm": {
"model": "ollama/llama3",
"format": "json",
"temperature": 0,
"base_url": "http://localhost:11434",
},
"verbose": True,
"headless": False,
"search_engine": "duckduckgo",
"max_results": 5,
}
prompt = {
"description": "Finding description about company {company}.",
"history": "Finding history about company {company}.",
"events": "Finding recent events about company{company}.",
}
def retrieve_basic_info_by_search(company_name: str) -> Dict[str, str]:
smart_scraper_graph = SearchGraph(
prompt=prompt["description"].format(company=company_name),
config=graph_config,
)
result = smart_scraper_graph.run()
return json.dumps(result, indent=2)
if name == "main":
company = "LinkedIn"
company_info = retrieve_basic_info_by_search(company)
print(f"\nSummary for {company}:")
print(company_info)
`
--- Executing MergeAnswers Node ---
Summary for LinkedIn:
{
"company_description": [
{
"source": "Website 1",
"description": "LinkedIn is a social networking platform designed for professionals and job seekers."
},
{
"source": "Website 2",
"description": "LinkedIn is a business and employment-oriented online service that operates through websites and mobile apps."
}
],
"sources": [
"http://www.google.com/search?hl=en&q={"description+about+LinkedIn"+:+"company+LinkedIn"}"
]
}