Ignored fields list for copyToRealmOrUpdate

[beeender]

User case from SO:
http://stackoverflow.com/questions/35244823/realm-for-android-how-to-sync-server-data-to-realm-database

If one field is stored locally, it would be convenient to ignore the local field while updating the existing RealmObject.

API proposal:

public <E extends RealmObject> E copyToRealmOrUpdate(E object, String... ignoredFields)

[donnysim]

It also could be something like creating a separate class that contains only the fields needed for the update and maybe it could be annotated something like @RealmUpdate(City.class) to point out what class it updates. Though sometimes this would require morphing some objects like City to CityUpdate, etc.. Ignored fields would ease a lot of update logic.

[cmelchior]

IMO this could probably be solved more elegantly be a new annotation like @LocalField or similar.

public class Foo extends RealmObject {

  @PrimaryKey 
  private long id;

  private String name;

  @LocalField 
  private boolean isDirty;
}

This would enable our annotation processor to just remove the code from our proxy classes instead of having users specifying the field each time. The above of course depends on the assumption that you only use copyToRealm when working with the network layer. I suspect that assumption is OK, but it needs to be validated.

[donnysim]

Would the @LocalField be ignored from updates but still saved to database? it would help where data is not synced, like user favorites, etc.. but how would it help with explicit updates? e.g. the first time I download the whole Foo list and want everything to be created, then, I already have a full Foo list, give me only a list of id and enabled columns, so I can update it on my side. I would like to use copyToRealmOrUpdate on the list and have only the enabled column updated (let's not get into details how something like updateIfExists would be better :) ). I don't think this would let you include\exclude the field when needed.

[cmelchior]

@donnysim Your right. My proposal would not allow different behaviour in two different code paths, so if that is a use case then it won't work. Note we already have an issue for createIfNotExists: #900

It is slightly different than only updating some fields though.

[donnysim]

@cmelchior Though @LocalField would still be useful if it does save to database :D

[cmelchior]

Yes, it is a fine balance. Trying to make first-class support for all these ways of updating can increase the complexity of the API quite quickly.

[guillermomuntaner]

This is related to the issue #2179 I opened. I see 2 use-cases (both of them I am currently facing and solving via inconvenient helper methods):

1. To be able to create specific local fields that are never updated via copyToRealmOrUpdate. In this case, the proposed @LocalField can work.
2. To be able to partially update a model explicitly specifying fields. E.g. i have "compact" models coming from some endpoints and "extended" models coming from others (think of extended user model with tons of information vs a tiny user model with username and avatar to include in other objects like posts and replies). Problem i faced is that saving "compact" models using copyToRealmOrUpdate() will destroy all the information already saved in "extended" models. In this case @LocalField approach is not enough. Via annotations, i can thing of something like @DontOverwriteIfNull but as we discussed in other issues, null sometimes can be a desired value. I like the proposed public <E extends RealmObject> E copyToRealmOrUpdate(E object, String... ignoredFields)

[cmelchior]

Thanks for the reminder @guillermomuntaner . I will merge the two issues so we only have the discussion in one place.

Copy paste from #2179

---

Feature request:

Partial updates

How?

Related to issue #1853
I agree that only updating not-null values is not a correct approach, since null can be sometimes a desired value and also because of not nullable primitive types with default values.

But, what about a call to specify a list of the fields that one want to overwrite (or preserve)?
copyToRealmOrUpdateOverwritingFields("field1","field2")
copyToRealmOrUpdatePreservingFields("field1","field2")
copyToRealmOrUpdate(obj).preserve("field1","field2")

In case the object does not exist in realm, the "fields to be preserved" can be initialized with default values, which i think is quite solid from a design pov.

Thoughts on this?

My issue scenario:

Im doing a cache with Realm where objects from one class can be updated online or locally based on user interaction. This is a problem, because every single online update using copyToRealmOrUpdate() will overwrite all the local updates which i would like to keep. The option to use json would work for me, but since im dealing with complex json and doing custom deserialization plus other manipulations in order to get the POJO i am storing, having to serialize to Json just to update feels a bit excessive.

As reference found also other related issue with a scenario like mine #1540

[cmelchior]

Interesting use case with a REST API that provides both compact / extended models. It does make sense though.

Wouldn't inheritance solve this quite elegantly though?

public class ExtendedFoo extends SimpleFoo {
  private Date birthday;
}

public class SimpleFoo extends RealmObject {
  @PrimaryKey
  private long id;
  private String name;
  private String avatar;
}

// Option A) Automatically ignore any sub type properties
<E extends RealmObject> copyToRealmOrUpdate(E object);
realm.copyToRealmOrUpdate(simpleFoo); // Ignores all fields in ExtendedFoo

// Option B) Provide a boolean for controlling it 
<E extends RealmObject> copyToRealmOrUpdate(E object, boolean ignoreSubTypeFields);
realm.copyToRealmOrUpdate(simpleFoo, true); // Ignores all fields in ExtendedFoo

Somehow having to maintain a list of fields to ignore just seems very manual and very error prone to me, so brainstorming a bit for other ideas.

[guillermomuntaner]

@cmelchior At first the idea looked great and elegant to me, but after thinking a bit I don't get how inherited models would be stored. The optimal way would be to store the "expanded" model and cast to "compact" models, but this does not fit well with inheritance nature, where several "expanded" models can exist.

• Models can have several "compact" submodels. One model lead to several smaller and uncomplete models.
• Classes can have several inherited subclases. On class lead to several expanded classes.

Thus, the proper way to handle your inheritance proposal would be to store base class info in one table and extended models info into another table and be able to merge to produce final objects. Not sure if Realm is capable of it.

Actually im doing something similar to this idea manually, but it would be nice to access parent class properties directly by inheritance.

public class ExtendedFoo extends RealmObject {
  private Date birthday;
  private SimpleFoo simpleFoo;
}

public class SimpleFoo extends RealmObject {
  @PrimaryKey
  private long id;
  private String name;
  private String avatar;
}

[donnysim]

I am currently using 2 classes, one is a small compact model, and the other contains extended info with id to the compact one. I don't really like this way, you constantly have to think about two classes. If you add a link from compact to extended model, then updating compact will still remove that link and removing that compact entry will leave extended info floating around, so it still provides more work than is really necessary. I liked how in one SQL ORM I just create a new compact and extended/update class and just point to the same table.

[guillermomuntaner]

@donnysim Indeed I faced same linkage issues. I ended up doing 1 big class and helper method to partialy update. What i am doing now to update is just manually searching if the object already exists and update only specific fields:

Class1 object = customWayToGetANonRealmObject();
Class1 realmObject = instance.where(Class1.class).equalTo("id",object.getId()).findFirst();
if(realmObject!=null) {
    // Write possible updated fields
    realmObject.setField1(object.getField1());
    realmObject.setField2(object.getField2());
    .... 
}
else {
    //If object not found in realm, just copy it.
    realmObject = instance.copyToRealm(object);
}

And what i would like to do is something as proposed:
realmObject = instance. copyToRealmOrUpdate(object,"field1","field2");

@cmelchior Another advantage of field name lists vs using subclasses is that you can dynamically adjust them. Think of APIs with more than 1 "compact" version or with dynamic fields, like the Facebook Api where you can set the specific fields you want returned. I agree that using field names as strings is error prone, but currently this is how the query filtering and sorting works in Realm too.

[fuwaneko]

I also have similar issues with copyToRealmOrUpdate and relations. The idea is this: I fetch data from server which contains relationship information as a primary key:

{
"id": 1,
"relatedObject": 101
}

public class Item extends RealmObject {
    @PrimaryKey private int id;
    private RelatedObject relatedObject;
}

And I already have related objects stored locally. So when I call copyToRealmOrUpdate I get empty related objects (i.e. only PK is stored, all other fields become nulls).

As a workaround I use "raw" POJOs to decode for Retrofit and then manually process them, establishing relations and creating/updating RealmObjects from those POJOs. It's very annoying and introduces a lot of boilerplate code.

So, how about an option to turn off deep copying, i.e. copyToRealmOrUpdate(object, deep=false)?

I mean, we're on mobile, so sending full relationship objects with each data update via network is very unhealthy.

Note: it's different from just ignoring some fields, because I can obviously receive relationship update (i.e. different PK) which must be reflected locally.

[HsiangLeekwok]

It is very important to support partial update.

copyToRealmOrUpdate(T extends RealmObject item);// this function is exist in current realm
copyToRealmOrUpdate(T extends RealmObject item, String... ignoredFields);
copyToRealmOrUpdate(List<T extends RealmObject> list, String... ignoredFields);

example:
This object need fetch from remote server by JSON(it is not designed by me, so I don't know how many all fields is):

public class Dog {
    private String name;
    private int age;
    ... // other fields
    // ... Generated getters and setters ...
}

then, I create an object like this:

public class Dog extends RealmObject {
    @PrimaryKey
    private String name;
    private int age;
    private int localCage;// this is local field to store dog's cage number  :)
    // ... Generated getters and setters ...
}

fetching json objects and save to realm:

[{"name":"John","age":3},{"name":"Peter","age":2},{...}...]
// all localCage was set to 0(default value) by using Gson.fromJson(json....)
// copyToRealmOrUpdate(List<Dog> list)

user can rest dog's cage like this:

{"name":"John","age":3,"localCage":6}

The problem is: when user fetching dogs list again(ex. using SwipeRefreshLayout/RecyclerView), all local fields are reset to 0(by using copyToRealmOrUpdate(List list);). because that remote json has no field named "localCage", the local fields will be set to default value(integer type will set to 0, boolean type will set to false, string type will set to null.....)

// JSON is not include field "localCage", so all dogs field "localCage" will be set to 0 at this line.
List<Dog> dogs = gson.fromJson(jsonString, new TypeToken<List<Dog>>() {}.getType());
// realm.copyToRealmOrUpdate(dogs);// this place cannot use copyToRealmOrUpdate(dogs)
realm.copyToRealmOrUpdate(dogs, "localCage");// good, we will not change field "localCage" this time

in current realm, I doing this to avoid loss local fields value;

List<Dog> dogs = gson.fromJson(json, new TypeToken<List<Dog>>() {
}.getType());
for(Dog dog : dogs){
    Dog exist = getDogFromRealm(dog.getName());// realm.where(Dog.class).equal.......
    if (null != exist) {
        dog.setLocalCage(exist.getLocalCage());
        // ... other local fields
        // this place decided by your local fields, 
        // if local fields less than remote json object, use unmanaged object,
        // otherwise, use managed object. 
        // if it has dozens local fields and remote fields, GOD, it's hard to say
    }
}
realm.copyToRealmOrUpdate(dogs);

Other way to fix it?

public class Dog extends RealmObject {
    @PrimaryKey
    private String name;
    private int age;
    @Ignore // or @LocalField ???(@LocalField is not support yet)
    private int localCage;
}

[ArthurSav]

@HsiangLeekwok

Although i consider it a hack, along with any realm implementation at this point, i use the following method to 'update' an object.

public void updateUser(Realm realm, String userAsJson){
    realm.executeTransaction(realm1 -> realm1.createOrUpdateAllFromJson(User.class, userAsJson));
}

createOrUpdateAllFromJson will basically update the object while it ignores null values

Now what happens if you use custom serializers/deserializers while parsing.
For example
RealmList<RealmString> needs custom deserialization since realm can't handle a list of primitive values.

Well, in that case you do something like this:

public void updateUser(Realm realm, String userAsJson){
    User user = gson.fromJson(userAsJson, User.class); // gson that runs custom deserialization
    String deserializedUserAsJson = gson.toJson(user); // now we can insert user into realm without issues
    realm.executeTransaction(realm1 -> realm1.createOrUpdateAllFromJson(User.class, deserializedUserAsJson));
}

A nice solution from the realm team would be one or all of the following:

• Gson uses TypeAdapter & TypeAdapterFactory to customize serialization/deserialization behaviour, introduce something similar. This would basically allow us to write custom rules when inserting/retrieving data from realm.
• Allow realm to update objects, while ignoring null values from the updating source

[HsiangLeekwok]

Great! Thanks! Helpful! @ArthurSav

Sorry, I haven't read the realm documents carefully than you, I never use createOrUpdateAllFromJson yet :)

Check myself....

[eygraber]

Are there any plans to implement this, or is the official solution to use createOrUpdateAllFromJson?

[Zhuinden]

The official solution is only set the fields on the managed object what you want persisted, instead of just overwriting the currently existing one.

[eygraber]

@Zhuinden does that mean I'm not going to be able to use insertOrUpdate? That would be a pretty big performance hit.

Use case is I get a model from the network, and I automatically store it in realm. Now I would have to check to see if it's already in realm, and if it is, only update the fields I want. Otherwise, create a new object. Is that correct?

[Zhuinden]

MyObj obj = realm.where(MyObj.class).where("id", id).findFirst();
if(obj == null) {
    obj = realm.createObject(MyObj.class, id); // or realm.copyToRealmOrUpdate(unmanagedObj);
}
// ... set things up

You can't really tell the difference for if a field should be set to null, or if it is "not set". So Realm defaults to that if your object has null set as value, then it will be set as null.

[lordplagus02]

I don't think createOrUpdateAllFromJson is an acceptable solution, I feel this completely breaks any implementation of retrofit + converters (for example) in your network layer, as is my case. I can't use retrofit 2.0 and moshi to deserialize json objects into model classes i can work with, if I am forced to use createOrUpdateAllFromJson. Otherwise we have to hard code so many things.

[NinoDLC]

Any update on this ?

It's been almost a year I'm waiting for this improvement.

I always optimise my API calls for a quick and fast browsing in apps, but because of this limitation, the API has to fetch full data for every items called, otherwise the user would lose its "cache" every time he gets back on the main activity (which is querying the "lightweight" api).

[cmelchior]

We are not going to implement an ignore list of fields. Our JSON methods already support partial updates, so you can use those if you want. Also, there are a number of other features on our roadmap that would be better suited for this without inflating our API surface and complicate our existing copyToRealm methods considerably.

1. Inheritance ( Inheritance / Polymorphism #761 ) -> FullClass extends PartialClass
2. Support for @SerializableName or similar ( Support mapping from field name in model class to column name in database file. #2476 ) to allow our JSON methods to be used more easily.

I completely understand why you would want to Use Retrofit/GSON, but there is something fundamentally wrong if you first want to deserialize a "partial" object into the "full" Java definition and then expect other layers of your application to understand that the object isn't what it claims to be.

We do realize the experience can be frustrating right now as you are basically forced to implement the partial update yourself and we do hope to improve on the situation, but it will not be as an "ignore" list.

[Zhuinden]

If it is a partial object, then you're the only one who knows which fields are meant to exist and which do not exist in the partial object.

[NinoDLC]

@Zhuinden Not really, if the collection is null, it means it hasn't been set by retrofit (and thus, it shouldn't change anything in Realm). If the collection is empty is means it has been set by retrofit and then it should erase that in Realm.

Am I missing something that makes it so hard to implement ? I'm by no mean a DB expert

[Zhuinden]

@NinoDLC collection sure, other fields (string, date, etc.) definitely not. Even then whether null == empty collection is up for interpretation. The official answer was above my answer

[Zhuinden]

@cmelchior @beeender while watching this video I was kinda wondering - while you can't really use an ignore fields exclusion list because Proguard can eat your model class (unless you use the original field name during annotation processing, I guess) ~ I was wondering about an easier option

realm.insertOrUpdate(item); // persists nulls by default
realm.insertOrUpdate(item, true); // true being the default, "persist nulls"
realm.insertOrUpdate(item, false); // ignores null values

That way it's actually +1 method for realm.insertOrUpdate and I'm guessing +1 for each proxy, which is okay, and not that complex?

Is this a bad idea? It's 2 AM, I might be full of terrible ideas.

[beeender]

while you can't really use an ignore fields exclusion list because Proguard can eat your model class (unless you use the original field name during annotation processing, I guess)

Yes, proguard is a problem, we have it for the class name as well. The proxy will generate a map of Class to the realm name by annotation processor, and we access the map on runtime through https://github.com/realm/realm-java/blob/master/realm/realm-library/src/main/java/io/realm/internal/RealmProxyMediator.java#L78

Or maybe the API could be more generic to have even more flexibility:

interface RealmInsertionFilter<T extends RealmModel> {
    boolean shouldInsert(T objectToBeInserted, String fieldName);
}

RealmInsertionFilter insertionFilter = new RealmInsertionFilter() {
    boolean shouldInsert(Foo foo, String fieldName) {
        if (fieldName.equals("noPersist")) {
            return false;
        }
        return true;
    }
}

realm.insertOrUpdate(item, insertionFilter);

[Zhuinden]

I think the filter parameter is overkill, people seem to just want to be able to ignore null values, lol.

Although I guess

RealmInsertionFilter insertionFilter = new RealmInsertionFilter() {
    boolean shouldInsert(Foo foo, String fieldName) {
        if (foo == null) {
            return false;
        }
        return true;
    }
}

would work, too!

---

EDIT: wait a second, this would work only with reflection. Am I blind? a fieldValue parameter would be nice. Maybe oldValue, newValue

[asbadve]

Just to let you know I am migrating to Room due to this feature.So much boilerplate code I need to write for this implementation.

[Zhuinden]

@asbadve I'm pretty sure Room handles partial updates the same way if you use conflict resolution strategy REPLACE.

[asbadve]

@Zhuinden I am having a specific reason for partial updates not because of null values. I am having some extra field in pojo than the web API which I want to maintain at the client side. Every time I need to get those value from DB before updating of the whole pojo and then based on that then I updated the value in DB. Which is a lot of boilerplate code and plus use Gson and retrofit2 so there is no point in converting pojo to json and then update.

[Zhuinden]

@asbadve so how exactly will Room help with this? @UPDATE statement for each property?

[asbadve]

@Zhuinden not with @update statement but with @query . And I know that I need to mention all the fields in it but it only for once in Dao.
I know that can also be achieved using realm with repository pattern.

[ismdcf]

Any Update on this or was it decided to use createOrUpdateAllFromJson. If there are any other ways can anyone explain as I'm also using retrofit
Cheers

[Zhuinden]

@ismdcf findFirst()/createObject() and then set the expected fields

[im244]

5 Years afterwards and it's not implemented, any chance to get such feature?

[im12345dev]

Any news?