Embedded objects migration: How to delete Orphaned objects after migration?

[vardansargsyan92]

Hi.
Another question related to the embedded objects migration. In the project, when migration of embedded objects completes I am getting this issue At least one object does not have a backlink (data would get lost).. I have googled it to understand what is going on and found the following discussion. I have tried to delete the orphaned objects but no luck yet. Here is the workaround that I have tried:

the initial state of classes were following:

public class MasterItemTruckExt : RealmObject
 {
     [PrimaryKey]
     [MapTo("_id")]
     public string Id { get; set; } = Guid.NewGuid().ToString();
     
      // Shuttle block
     public int ShuttleDefaultCapacity { get; set; }
     public int ShuttleMaxCapacity { get; set; }
     public int ShuttleDefaultLoadTime { get; set; }
     public int ShuttleMaxLoadTime { get; set; }
     public int ShuttleDefaultUnloadTime { get; set; }
     public int ShuttleMaxUnloadTime { get; set; }

     // LD block
     public int LDDefaultCapacity { get; set; }
     public int LDMaxCapacity { get; set; }
     public int LDDefaultLoadTime { get; set; }
     public int LDMaxLoadTime { get; set; }
     public int LDDefaultUnloadTime { get; set; }
     public int LDMaxUnloadTime { get; set; }

     public MasterItem TruckMasterItem { get; set; }
 }
 public class MasterItem : RealmObject
 {
     [PrimaryKey]
     [MapTo("_id")]
     public string Id { get; set; } = Guid.NewGuid().ToString();

     public string Name { get; set; }
     public string Description { get; set; }
 
     public MasterItem Group { get; set; }

     [Backlink(nameof(Group))]
     public IQueryable<MasterItem> GroupMasterItems { get; }
   
     public double Cubes { get; set; }

     public int TimePerPiece { get; set; }

     public Move.Move Move { get; set; }

     [Backlink(nameof(AutoIncludedItem.ParentItem))]
     public IQueryable<AutoIncludedItem> AutoIncludedItems { get; }

     public decimal DeclaredAmount { get; set; }
     public decimal DeductibleAmount { get; set; }

 }

Then I decided to make MasterItemTruckExt an embedded object and did the following change.

    public class MasterItemTruckExt : EmbeddedObject
    {
        // Shuttle block
        public int ShuttleDefaultCapacity { get; set; }
        public int ShuttleMaxCapacity { get; set; }
        public int ShuttleDefaultLoadTime { get; set; }
        public int ShuttleMaxLoadTime { get; set; }
        public int ShuttleDefaultUnloadTime { get; set; }
        public int ShuttleMaxUnloadTime { get; set; }

        // LD block
        public int LDDefaultCapacity { get; set; }
        public int LDMaxCapacity { get; set; }
        public int LDDefaultLoadTime { get; set; }
        public int LDMaxLoadTime { get; set; }
        public int LDDefaultUnloadTime { get; set; }
        public int LDMaxUnloadTime { get; set; }
    }
    public class MasterItem : RealmObject
    {
        [PrimaryKey]
        [MapTo("_id")]
        public string Id { get; set; } = Guid.NewGuid().ToString();

        public string Name { get; set; }
        public string Description { get; set; }
    
        public MasterItem Group { get; set; }

        [Backlink(nameof(Group))]
        public IQueryable<MasterItem> GroupMasterItems { get; }
      
        public double Cubes { get; set; }

        public int TimePerPiece { get; set; }

        public Move.Move Move { get; set; }

        [Backlink(nameof(AutoIncludedItem.ParentItem))]
        public IQueryable<AutoIncludedItem> AutoIncludedItems { get; }

        public decimal DeclaredAmount { get; set; }
        public decimal DeductibleAmount { get; set; }

        public MasterItemTruckExt MasterItemTruckExt { get; set; }

    }

This part is working fine. The next challenge is to write a migration function to consolidate old and new versions of realm databases. And I have the following migration function.

 public class MasterItemTruckExtMigration : IMigration
    {
        private const int IntroducedInSchemaVersion = 7;

        public void Migrate(Migration migration, ulong oldSchemaVersion)
        {
            if(oldSchemaVersion >= IntroducedInSchemaVersion)
            {
                return;
            }

            IQueryable<dynamic> masterItemTruckExtItems = migration.OldRealm.DynamicApi.All("MasterItemTruckExt");
            IQueryable<MasterItem> newMasterItems = migration.NewRealm.All<MasterItem>();

            for(var i = 0; i < masterItemTruckExtItems.Count(); i++)
            {
                dynamic masterItemTruckExt = masterItemTruckExtItems.GetElementAt(i);
                string truckMasterItemId = masterItemTruckExt.TruckMasterItem.Id;
                MasterItem truckMasterItem = newMasterItems.First(x => x.Id == truckMasterItemId);
                DuplicateTruckExtItemFields(masterItemTruckExt, truckMasterItem);
            }
        }

        private void DuplicateTruckExtItemFields(dynamic masterItemTruckExt, MasterItem masterItem)
        {
            masterItem.MasterItemTruckExt = new MasterItemTruckExt
                                            {
                                                // Shuttle block
                                                ShuttleDefaultCapacity = (int) masterItemTruckExt.ShuttleDefaultCapacity,
                                                ShuttleMaxCapacity = (int) masterItemTruckExt.ShuttleMaxCapacity,
                                                ShuttleDefaultLoadTime = (int) masterItemTruckExt.ShuttleDefaultLoadTime,
                                                ShuttleMaxLoadTime = (int) masterItemTruckExt.ShuttleMaxLoadTime,
                                                ShuttleDefaultUnloadTime = (int) masterItemTruckExt.ShuttleDefaultUnloadTime,
                                                ShuttleMaxUnloadTime = (int) masterItemTruckExt.ShuttleMaxUnloadTime,

                                                // LD block
                                                LDDefaultCapacity = (int) masterItemTruckExt.LDDefaultCapacity,
                                                LDMaxCapacity = (int) masterItemTruckExt.LDMaxCapacity,
                                                LDDefaultLoadTime = (int) masterItemTruckExt.LDDefaultLoadTime,
                                                LDMaxLoadTime = (int) masterItemTruckExt.LDMaxLoadTime,
                                                LDDefaultUnloadTime = (int) masterItemTruckExt.LDDefaultUnloadTime,
                                                LDMaxUnloadTime = (int) masterItemTruckExt.LDMaxUnloadTime
                                            };
        }
    }

I have tried to remove orphaned objects with following:

foreach(dynamic masterItemTruckExtItem in masterItemTruckExtItems)
            {
                migration.OldRealm.Remove(masterItemTruckExtItem);
            }

then

migration.OldRealm.DynamicApi.RemoveAll("MasterItemTruckExt");

I am getting this issue Cannot modify managed objects outside of a write transaction.

then I try this but again no luck.

migration.OldRealm.Write(()=>
                                         {
                                           migration.OldRealm.Remove(masterItemTruckExtItem);
                                         });

So I would very happy to find any workaround how to resolve this. Thank you.

[nirinchev]

You'll need to remove them in the NewRealm - OldRealm is readonly. What you could do is add the following block at the end of the migration function:

var toDelete = migration.NewRealm.DynamicApi.All("MasterItemTruckExt").Filter("@links.MasterItem.MasterItemTruckExt.@count == 0");
migration.NewRealm.RemoveRange(toDelete);

I haven't tested this, so likely won't work from the get-go, but might get you on the right path. We'll try to look into this issue at some point this week and come up with a more comprehensive docs about it. Sorry for the omission.

[vardansargsyan92]

hi @nirinchev

Thank you for your instant response. I really appreciate it. It's very interesting to deep into the realms of Realm ))). I will do that.

[softlion]

That does not work. Error "The class represents an embedded object and thus cannot be queried directly"

[DominicFrei]

I have added three new tests for this as part of #2394:
https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Tests/Realm.Tests/Database/MigrationTests.cs#L149-L343

@vardansargsyan92 As @nirinchev already mentioned, deleting objects in the oldRealm is not going to have any effect, since that's not going to be forwarded to the newRealm while the migration is already in progress. The oldRealm is basically just a reference to the database using the old schema and the newRealm is the view onto the database using the new schema.

You would have to iterate in the new realm therefore. The problem is now, looking at https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Tests/Realm.Tests/Database/MigrationTests.cs#L272 specifically (which is the iteration we're talking about here - the one mentioned by Nikola) we see a crash.

It originates in https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Realm/Realm/Realm.cs#L1736 where the Metadata is used which was set earlier in https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Realm/Realm/Realm.cs#L281.

We check against the Metadata and find this to be set to isEmbedded already. This appears to be a bug to me at the moment since the EmbeddedObject is actually not set to embedded while the migration happens but after.

I've fixed a bug in core a while ago where this was actually done pre migration:
https://github.com/realm/realm-core/pull/4414/files#diff-1d2d2dee75a4c6f88551eafd9474f4a6bab23c83377ed2a7696ee694b0d6a753R729
This is fixed now which is why I assume this might be a problem with us checking against our already updated metadata.

I'm going to investigate further.

[vardansargsyan92]

Hi @DominicFrei Thank you for response. I will try to somehow mitigate that migration. Perhaps I need to use a new class as embedded object instead of refactoring old one.

[DominicFrei]

@vardansargsyan92 I have seen some workarounds in Cocoa since we got the same question / problem over there.

Your idea seems to be the easiest way to do this as a workaround until we figured this out. 👍

The alternative would be to not transition to EmbeddedObject just now in case you don't necessarily need to and do a normal migration with the same class as soon as this is fixed.

Creating new classes that are EmbeddedObject right from the start works nicely but the migration to them needs some more work / documentation.

[vardansargsyan92]

@DominicFrei I hope everything will be fixed soon )))