StackOverflow while trying to lookup entities having reference to no-longer-existing entity

[h-arlt]

Describe the bug

Attempts to query for entities using an equality filter on a field annotated with @Reference(lazy=true, idOnly=true) causes StackOverflow errors in case the referenced entity has gone.

java.lang.StackOverflowError: null
	at java.base/java.util.Arrays.hashCode(Arrays.java:4493)
	at java.base/java.util.Objects.hash(Objects.java:133)
	at org.bson.internal.CodecCache$CodecCacheKey.hashCode(CodecCache.java:55)
	at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at org.bson.internal.CodecCache.get(CodecCache.java:78)
	at org.bson.internal.ProvidersCodecRegistry.get(ProvidersCodecRegistry.java:74)
	at org.bson.internal.ProvidersCodecRegistry.get(ProvidersCodecRegistry.java:48)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:57)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)
	at dev.morphia.mapping.codec.pojo.EntityEncoder.encode(EntityEncoder.java:58)
	at dev.morphia.mapping.codec.pojo.MorphiaCodec.encode(MorphiaCodec.java:81)

To Reproduce
Steps to reproduce the behavior:

1. Store two entities in DB where one references the other by having the field - say r - annotated as described above.
2. Lookup the root entity.
3. Store the value of the field r into a local variable (should be a proxy instance).
4. Delete the referenced entity.
5. Lookup single or all entities of root entity's type using an equality filter for field r and value of variable from step 3.

Expected behavior
Query succeeds w/o any issues.

** Please complete the following information: **

• Server Version: 5.0
• Driver Version: 4.8.1
• Morphia Version: 2.3.2

Additional context

package dev.morphia.test.mapping.lazy;

import java.util.List;
import java.util.Objects;

import dev.morphia.Datastore;
import dev.morphia.annotations.Reference;
import dev.morphia.test.mapping.ProxyTestBase;
import dev.morphia.test.models.TestEntity;

import org.testng.annotations.Test;

import static dev.morphia.query.filters.Filters.eq;
import static org.testng.Assert.assertNotNull;

@Test(groups = "references")
public class TestLazyIdOnly extends ProxyTestBase {

    @Test
    public void testQueryAfterReferentIsGone() {
        checkForProxyTypes();

        RootEntity root = new RootEntity();
        root.setBar("foo");
        final ReferencedEntity reference = new ReferencedEntity();
        reference.setFoo("bar");

        root.setR(reference);

        final Datastore datastore = getDs();
        datastore.save(List.of(reference, root));

        root = datastore.find(RootEntity.class).filter(eq("_id", root.getId())).first();

        ReferencedEntity p = root.r;

        assertIsProxy(p);
        assertNotFetched(p);

        datastore.delete(reference);

        root = datastore.find(RootEntity.class).filter(eq("r", p)).first();

        assertNotNull(root);

        p = root.r;

        assertIsProxy(p);
        assertNotFetched(p);
    }

    public static class ReferencedEntity extends TestEntity {
        private String foo;

        public String getFoo() {
            return foo;
        }

        public void setFoo(String string) {
            foo = string;
        }

        @Override
        public int hashCode() {
            return Objects.hash(id, foo);
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) {
                return true;
            }
            if (!(o instanceof ReferencedEntity)) {
                return false;
            }
            ReferencedEntity that = (ReferencedEntity) o;
            return Objects.equals(id, that.id) && Objects.equals(foo, that.foo);
        }
    }

    public static class RootEntity extends TestEntity {
        @Reference(idOnly = true, lazy = true)
        private ReferencedEntity r;

        private String bar;

        public String getBar() {
            return bar;
        }

        public void setBar(String bar) {
            this.bar = bar;
        }

        public ReferencedEntity getR() {
            return r;
        }

        public void setR(ReferencedEntity r) {
            this.r = r;
        }
    }
}

[evanchooly]

@h-arlt if you could take a look at the updated test, i think this will get you what you need. I had to update the test because some assumptions didn't apply but this push should fix the problem for you. If you'd like to run it locally, you can test against 2.3.5-SNAPSHOT which should be online shortly in the central snapshots repo.

[h-arlt]

Updated test looks fine to me and works like a charm with latest 2.3.x 🎉

PS: To satisfy my tester paranoia, I'd change line 53 to

root = datastore.find(RootEntity.class).filter(eq("ignoreMissing", p)).first();
assertNotNull(root);

[evanchooly]

done. :) though it should be assertNull because that query won't find that relation.

[evanchooly]

2.3.5 is now live

[h-arlt]

done. :) though it should be assertNull because that query won't find that relation.

Just out of curiosity, why won't the query find a matching document? Is there any existence check on the referenced document involved? I'd rather expect a simple equality check on property ignoreMissing and the ID of the passed document p 🤔

[evanchooly]

It wouldn't find it because the query is asking, "can you find me this A that references this B?" and the answer is no because that B doesn't exist any more.

[h-arlt]

It wouldn't find it because the query is asking, "can you find me this A that references this B?" and the answer is no because that B doesn't exist any more.

Exactly. That's what I'd expect when the reference wouldn't be marked with idOnly but it is in that case. Hence, I'd translate the query to "find me all documents in RootEntitys collection whose ignoreMissing field is equal to the ID of p - regardless of the existence of any document with that ID since the reference is also marked as being lazy".

Don't want to bother you; I'm just trying to understand the implications and limitations when working with Morphia references.

[evanchooly]

ignoreMissing applies to loading an entity with a missing reference not when trying to query against that reference's ID.

[h-arlt]

In my example above, ìgnoreMissing was the name of field that holds the reference, not the annotation's attribute.

What about idOnly? Question is: how do I query for documents that reference (by ID) a document hat has vanished from DB but whose ID is still known for whatever reasons?

My use case is as follows: I use a reference to model a weak relation between two types of entities. This reference is marked as lazy, ignoreMissing and idOnly. At some point in time, some of the referenced documents vanish from DB (e.g. deleted in course of user interaction with the the app). Now, in order to find all documents that have a dangling reference, I need to query for the reference's field having an ID of one of the deleted documents.

[evanchooly]

I see. With the way references are currently done internally, that will be tricky with a query. (I'm reworking how references are handled in 3.0 to clean (hopefully) all of that up.) Probably the most semantically reading way to do what you need here is with a $lookup aggregation and filter on those null values to find the dangling references.

[h-arlt]

Roger that. As long as there is a way I'm happy ;)

Thanks for clarification and the efforts you put in this project! Appreciate it 👍