How can I return a web element using Tab's .evaluate() method?

[hamzaaitbrik]

Hello, I was building a project on top of ultrafunkamsterdam/nodriver. But, I decided to switch to Zendriver because it has better support. I am now stuck trying to locate a few web elements that are nested deeply into shadowRoots.

I used to locate all elements that are nested inside shadowRoots using this JavaScript code snippet I wrote:

// !! code snippet made by hamzaaitbrik @ GitHub !!



(() => {

    const matchedElements = [];

    function traverse(node) {

        // Check if the node is an element node and has the specified href
        if (node.nodeType === Node.ELEMENT_NODE && node.tagName === 'TAGNAME') {

            if (node.getAttribute('ATTRIBUTE') === 'VALUE') {

                matchedElements.push(node);

            }

        }

        // If the node is a shadow host, traverse its shadow root
        if (node.shadowRoot) {

            traverse(node.shadowRoot);

        }

        // Recursively traverse child nodes
        node.childNodes.forEach(child => traverse(child));

    }

    // Start traversing from the document body
    traverse(document.body);

    const targetElement = matchedElements[0];

    return targetElement;

})();

But when I tried to run that with .evaluate(), I get None while I get my target web element when I run that on the browser's console. I did some further debugging and I figured out that .evaluate() only returns serializable data, which is a huge problem for me because I need to run that snippet to ensure consistency in locating my targeted web elements. I've been debugging this for hours with no luck.

I am writing this issue as a way to reach out to the owner of this repository to maybe clarify to me a way on how I can return a web element using .evaluate(), or at least is there any workaround to accomplish what I'm trying to accomplish..?

Thanks.

[stephanlensky]

Hi @hamzaaitbrik, unfortunately there is no workaround for the requirement that your data is serializable. This is because in order to transfer the data between the browser and your Python program, it must be encoded as JSON and sent over a websocket connection.

Can you use outerHTML (MDN docs) to serialize the element as a string? Then, on the Python side, you could re-parse it using something like BeautifulSoup if you need to do any further processing.

[hamzaaitbrik]

@stephanlensky thanks for the reply. I was stuck for two days I had to think of something...I ended up changing the source code of zendriver, I added a function that can locate an element based on its tagname, attribute/value. I also added another function that can return a list of elements based on the same attributes.

Those two functions search recursively in nested shadowRoots on a web page, selecting only elements with our targeted tagname, then narrowing down with an attribute and its value. So if we're aiming for a button that has the attribute type equal to submit, we would call the function with those values as parameters. That way, you're guaranteed your targeted element(s) every single time.

Do you think that's something the project would benefit from? I personally believe that the functions I wrote are far better at identifying elements in very complex web pages than the already existing .find() and .select().

If you think it will add some value into the project, I will make a pull request for further examination.

Thanks.

[stephanlensky]

@hamzaaitbrik sure, I'd be happy to take a look. I think that much of the code for finding elements etc. is going to need to be substantially rewritten at some point, so I'd be happy for contributions which start moving it in the right direction.

[stephanlensky]

@pythonlw It's pretty hard to read your comments, I'd suggest reading the GitHub markdown docs to see how to properly format code in comments.

I'm also not sure that the library you recommended is a great suggestion for most people reading this, as it appears targeted mainly at Chinese language speakers and the documentation does not have an English translation.

I understand that you are trying to help, but with both of these things in mind I am going to hide your comment for now and ask that you exercise more care with your contributions in the future. Thanks.