AttributeError: 'FetchNode' object has no attribute 'update_state'

[MahdiSepiRashidi]

Describe the bug
in File "scrapegraphai\nodes\fetch_node.py" , clas FetchNode nor in BaseNode which is the parent of the class FerchNode there is no method defined as update_state(), although it has been used in both FetchNode.handle_local_source() and FetchNode.handle_web_source()

full error: File "scrapegraphai\nodes\fetch_node.py", line 233, in handle_local_source
return self.update_state(state, compressed_document)
^^^^^^^^^^^^^^^^^
AttributeError: 'FetchNode' object has no attribute 'update_state'

Desktop (please complete the following information):

• OS: Windows server 2022
• python library scrapegraphai
• Version 1.26.7

[VinciGit00]

Please share me the code

[MahdiSepiRashidi]

Please share me the code

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
"llm": {
"model": "ollama/llama3.1",
"temperature": 0,
"format": "json", # Ollama needs the format to be specified explicitly
"base_url": "http://localhost:11434", # set Ollama URL
},
"embeddings": {
"model": "ollama/nomic-embed-text",
"base_url": "http://localhost:11434", # set Ollama URL
},
"verbose": True,
}

string = "any str"

user_prompt ="get all products in this"

Create a SmartScraperGraph object

smart_scraper_graph = SmartScraperGraph(
prompt=user_prompt,
source=string,
config=graph_config
)

result = smart_scraper_graph.run()
print(result)

[VinciGit00]

ok you should try with the new scripts.
It is outdated

[jrc]

This is the relevant example code: https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/examples/local_models/smart_scraper_ollama.py

[MahdiSepiRashidi]

ok you should try with the new scripts. It is outdated

Thanks for the answer.
But I am not getting what you mean by "new scripts", since the last git version (https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/scrapegraphai/nodes/fetch_node.py) also seems not having any method as FetchNode.update_state().
I managed to fix the error by changing the method FetchNode.handle_local_source() in https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/scrapegraphai/nodes/fetch_node.py a bit.
I assumed the handle_web_source is working properly and mimicked its way of updating the "state".
So instead of
return self.update_state(state, compressed_document)
I used
state.update({self.output[0]: compressed_document,})
and the problem fixed.

[VinciGit00]

Ok please cann you make the pr?

[MahdiSepiRashidi]

Ok please cann you make the pr?

My pleasure.
I made the pr ready and I only need the permission to publish on the repository.

[YohannBlack]

Hi,
Has any solution been found for this issue ? I am having the same problem when trying to scrape certain website.

[VinciGit00]

@MahdiSepiRashidi you have to do the pull request on the ScrapegraphAI org, I have not seen it

[matheus-rossi]

scrapegraphai = "v1.30.0-beta.4"

config = {
        "llm": {
            "api_key": settings.ANTHROPIC_API_KEY,
            "model": "anthropic/claude-3-5-haiku-20241022", 
            "timeout": 60,
            "temperature": 0,
            "max_tokens": 8192
        },
        "verbose": verbose
    }

  response = SmartScraperGraph(
      prompt="MY_PROMPT",
      source=cleaned_html, -- MY HTML GOES HERE
      config=config,
  ).run()

gives me the following error

ERROR | src.utils.aiutils:_scrape_content:284 - 'FetchNode' object has no attribute 'update_state'