NoSuchElementException when selecting Elements

[justinleinbach-wk]

Hi,

My team and I have encountered NoSuchElementException when parsing and selecting elements.

Our caller uses the parser to parse HTML into a document, selects one of the elements and attempts to access sub-elements through a select call with a passed in evaluator. All interactions with the document, including parsing, element selection, and sub-element access, are confined to a solitary thread.

The stack trace from the caller is as follows:

 java.util.NoSuchElementException: null
 at org.jsoup.nodes.NodeIterator.next(NodeIterator.java:69)
 at org.jsoup.select.StructuralEvaluator$Has.matches(StructuralEvaluator.java:73)
 at org.jsoup.select.CombiningEvaluator$And.matches(CombiningEvaluator.java:84)
 at org.jsoup.select.StructuralEvaluator$ImmediateParentRun.matches(StructuralEvaluator.java:218)
 at org.jsoup.select.CombiningEvaluator$Or.matches(CombiningEvaluator.java:125)
 at org.jsoup.select.Evaluator.lambda$asPredicate$0(Evaluator.java:38)
 at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
 at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
 at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
 at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
 at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
 at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
 at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
 at org.jsoup.select.Collector.collect(Collector.java:29)
 at org.jsoup.select.Selector.select(Selector.java:111)
 at org.jsoup.nodes.Element.select(Element.java:463)

Tracing that, it appears to be throwing here, which shouldn't be possible because of the null check in the caller here.

We haven't been able to reproduce from a standalone test case, but have seen it occur intermittently twice now in a long-running process.

Our service currently uses jsoup 1.17.1 on Amazon Corretto 17 images. Additional details below

openjdk version "17.0.9" 2023-10-17 LTS
OpenJDK Runtime Environment Corretto-17.0.9.8.1 (build 17.0.9+8-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.9.8.1 (build 17.0.9+8-LTS, mixed mode, sharing)

Are there any ideas or explanation for what might be wrong?
If there's any additional information we can provide please let me know.

[jhy]

Hi, thanks for the report.

To confirm -- your entry point is:

    public Elements select(Evaluator evaluator) {
        return Selector.select(evaluator, this);
    }

Right?

I am wondering if you are using that Evaluator object across multiple concurrent threads?

I think that would explain it - the Iterator is created when the Evaluator is created and then is getting hit concurrently. And then in the .hasNext() / .next() calls there is a race on the next object.

The evaluator is supposed to be threadsafe and other instance fields are ThreadLocals. So I missed making the Iterator a threadlocal.

The iteration method in the :has evaluator changed in 1.17.1 which introduced this local Iterator, so you wouldn't have seen the issue in previous versions.

[justinleinbach-wk]

Hi, thanks for the report.

To confirm -- your entry point is:

    public Elements select(Evaluator evaluator) {
        return Selector.select(evaluator, this);
    }

Right?

I am wondering if you are using that Evaluator object across multiple concurrent threads?

I think that would explain it - the Iterator is created when the Evaluator is created and then is getting hit concurrently. And then in the .hasNext() / .next() calls there is a race on the next object.

The evaluator is supposed to be threadsafe and other instance fields are ThreadLocals. So I missed making the Iterator a threadlocal.

The iteration method in the :has evaluator changed in 1.17.1 which introduced this local Iterator, so you wouldn't have seen the issue in previous versions.

That's correct. We are sharing the evaluator between threads so that seems to us like the correct solution to the issue we were seeing. Thanks!