Does BrowserGym supports image_som?

[leoozy]

I do not find image_som in the current obs:
obs = {
"chat_messages": copy.deepcopy(self.chat.messages),
"goal": goal_msg, # TODO: redundant with chat messages, to be removed?
"goal_image_urls": goal_image_urls, # TODO: redundant with chat messages, to be removed?
"open_pages_urls": [page.url for page in self.context.pages],
"active_page_index": np.asarray([self.context.pages.index(self.page)]),
"url": self.page.url,
"screenshot": extract_screenshot(self.page),
"dom_object": dom,
"axtree_object": axtree,
"extra_element_properties": extra_properties,
"focused_element_bid": focused_element_bid,
"last_action": self.last_action,
"last_action_error": self.last_action_error,
"elapsed_time": np.asarray([time.time() - self.start_time]),
}
Does BrowerGym supports image som ?

[ThibaultLSDC]

Hi @leoozy, BGym does have a function to generate the SOM representation.

BrowserGym/browsergym/core/src/browsergym/utils/obs.py

Lines 425 to 431 in 87c45ad

	def overlay_som(
	screenshot: np.typing.ArrayLike,
	extra_properties: dict,
	fontsize: int = 12,
	linewidth: int = 2,
	tag_margin: int = 2,
	):

It is not computed by default to not slow down text only users.
Hope this helps :)

[gasse]

See here for an example
https://github.com/ServiceNow/AgentLab/blob/f385533ce85b89ad9456f676ebd019221d968f85/src/agentlab/agents/dynamic_prompting.py#L819