@SerializedName annotation for defining Class/Field names

[beeender]

We need something similar to GSON's SerializedName annotation which can map a different name field from JSON string to RealmObject's property.

eg:

{ "dog_name": "abc"}

class Dog {
    @SerializedName("dog_name")
    private String name;
}

[navjotbedi]

Any update on this feature ?

[taltstidl]

I'd really like to use Realms Json methods, but the way they currently work they are unfortunately too limited to be of any use to me. In addition to a @SerializedName annotation, I'd also need a @SerializedObjectId(s)Name annotation that allows linking of related objects using their ids. Most web APIs only return the id of the related object, which makes it impossible to use Realms Json methods (as far as I know).

So, for example, the following should work:

{"id": 1, "relatedId": 1, "relatedIds": [1, 2, 3]}

class TestObject extends RealmObject {
    // different serialized name in json representation
    @SerializedName("id") @PrimaryKey
    long mId;

    // one-to-one relationship in json representation
    @SerializedObjectIdName("relatedId")
    RelatedObject mRelatedObject;

    // one-to-many relationship in json representation
    @SerializedObjectIdsName("relatedIds")
    RealmList<RelatedObject> mRelatedObjects;
}

Using those annotations the Realm Json methods should be able to properly transfer the information encoded in the Json to the Realm object model, including its relationships. Having those annotations would solve quite a few issues for me with partially updating objects (see issue #3500) and would reduce the amount of code needed and make it more efficient.

[cmelchior]

@TR4Android I have browsed through the code and compiled a list of things that needs to be done (I probably forgot something though). If you decide to start on this, I would strongly encourage you to open a PR as soon as possible so we can follow progress along the way and adjust if needed. It would make sense for us to make a feature branch which you can make smaller PR's against instead of trying to cram everything into one big PR against master. It will be a lot easier for us to review.

Let me know. Cheers.

Overall design goal

Make it possible to use a Java specific name when instead of being forced to adopt the name being used by a JSON API or another platform.

USE CASE: My JSON input does not match my preferred Java class. This is useful if using our various createFromJson methods

USE CASE: I'm sharing a schema with iOS and would like my own names as the chosen names break normal Java conventions.

With this approach we combine these two use cases. Is that okay or will it bite us later so we don't want to mix JSON and Schema concerns?

TODO List:

• New annotation : @SerializedName (let the bikeshedding commence). It is an apt name, but already used by GSON, I don't really know if Jackson/Moshi has something similar and what their concepts are called. The annotation can be applied to both classes and fields.
• Our Proxy classes need to use the serialized name when reporting its name.
• Our Proxy classes need to use @SerializedName instead of the field name when creating the ColumnInfo. This will probably require extending ClassMetaData with this information somehow, so the RealmProxyClassGenerator can output the serialized name correctly. See https://github.com/realm/realm-java/blob/master/realm/realm-annotations-processor/src/main/java/io/realm/processor/ClassMetaData.java#L55 and https://github.com/realm/realm-java/blob/master/realm/realm-annotations-processor/src/test/resources/io/realm/AllTypesRealmProxy.java#L48
• DynamicRealmObject: We need to figure out if DynamicRealmObject uses the serialized name or the Java field name when accessing fields. I'm leaning towards using the SerializedName name since that is the underlying object being exposed + the Java object might not be present
• RealmQuery: Should use serialized name for DynamicRealm and Java names for Realm. This really irks me, but I think it is the sanest option. Thoughts?
• RealmObjectSchema methods should operate on the serialized name, always.
• RealmSchema methods should operate on the serialized name, always
• Migrations should only be triggered if the serialized name changes. If the Proxy class is changed correctly I believe this will happen automatically.
• Add unit tests for making sure that serialized name works for all JSON methods
• Add unit tests for migrations being triggered correctly if SerializedName changes for both class and fields
• Add unit tests for queries on models with serialized names defined (both Realm/DynamicRealm)

Conclussion

Am I missing something on this list @realm/java and do you have other points? I believe @zaki50 and @kneth were the last to touch these parts of the code.

[Zhuinden]

I don't think JSON concerns should be mixed with schema concerns.

[cmelchior]

I have the same feeling, but it gets a bit tricky, because cross-platform schemas are a big topic for us right now 😄

public class Foo extends RealmObject {

  // This is definitely eecky
  @SerializedName("my_stupid_json_name")
  @FieldName("internal_name")
  private String name;
} 

Thinking about this. Maybe this is actually the best time to actually split our JSON API away from Realm, because a @SerializedName for JSON would only make sense if you used our JSON API, if you use e.g GSON it would just be confusing that Realm also had a similar annotation.

Instead we should create adapters for the various JSON imports:

- `io.realm:android-json-adapter:1.0.0` : Basically our existing API just moved to a seperate dependency. This library could then include specialized annotations like `@SerializedName`

- `io.realm:gson-adapter:1.0.0` : GSON type adapters

- `io.realm:android-streaming-json-adapter:1.0.0` : Our API 11 streaming adapters

Alternatively they get configured through the new realm closure block introduced in 2.0

It would have the following implications:

• All createFromJson* and createOrUpdateFromJson* methods would be removed from our current public API.
• JSON Support is entirely opt-in. Right now they are a massive chunck of our proxy classes, we could strip a lot of code.
• We will have to find another way of calling JSON methods. GSON/Moshi/Jackson can continue to use our copyToRealm, but we need somewhere to hook "standard" JSON output in. Where isn't clear to me.

[taltstidl]

@cmelchior I tend to agree with @Zhuinden, I wouldn't want to mix the field and json names. Splitting this to its own library might make sense, as it would allow room for more superior support without making the core library more cluttered. I'd gladly add export APIs to this as well. Might make sense to name this realm-import or similiar and have it available for more platforms.

I'll probably need some guidance on the implementation then. Should the supplementary library alter the proxy classes? How should I split the library from the core one? How should I name the new classes for importing/exporting json?

I'll definitely create a PR as soon as possible, so you can track progress and intervene when I make mistakes on my side. I'm already exited to see this in a future release as it will make my app code that much simpler, and hopefully more performant!

One other thing, for the annotation I'd like to be able to build links to other objects using its primary key. That is currently the main reason I'm not using other json libraries, as I'm looking to build the relationships more natively. So, I'd like to see an option for that in the API design as well (see my previous comment).

[Zhuinden]

@cmelchior considering additional annotation processors can be executed on the same RealmObjects now that annotations aren't consumed, adding additional adapters (or adapter factories akin to Retrofit2) would make sense similarly to how "Rx support" is opt-in by adding the RealmObservableFactory.

[cmelchior]

@TR4Android I didn't consider the possibility to automatically hook up references, but you are right, it would also be a nice feature if done automatically. Especially since it is a lot of tedious code to do manually.

Basically by splitting the library we don't have to be as conservative with features, so export capabilities also make a lot more sense. I would however expect we want to share a lot of code between annotation processors, so we would need to find a way to do that.

We probably need to have an internal discussion first though as there are a lot of moving parts to this, but if you have some good ideas for how the public API could look like for this I am all ears.

@Zhuinden Yes, not consuming the annotations is definitely a major part of this as this allow us to mix any number of adapters. While the Retrofit factory method is very nice, keep in mind that right now we accept 3 different forms of JSON Input: InputStream, String and JSONObject/Array, so I don't think we can get away with just having a factory method somewhere as those 3 types are basically 3 different adapters.

[taltstidl]

@cmelchior Yes, you should probably have an internal discussion first. I'm not sure yet how the structure of the annotation processor should look, this is something I'd first have to investigate more. Any ideas would be appreciated. The public API would be pretty much what I have outlined in the other issue.

[cmelchior]

I'm mostly talking about the createFromJson* methods and friends. With the adapter approach I don't think it would make sense if they hang on the Realm object anymore, but they need to be somewhere (wishes for Kotlin extension methods)

[cmelchior]

Just throwing ideas at the the wall:

// Suggestion for the Realm closure
realm {
    gsonAdapters = true
    jsonStringAdapters = true
    jsonStreamAdapters = true 
    jsonObjectAdapters = true
}

// Version 1

Realm.init(context) // Automatically detect and register relevant adapters

// Combine all current methods so they accept `Object`
// This enable any adapter to "register" for a specific subclass and handle that
// Will reduce number of methods from 12 -> 4, and be backwards compatible with
// the proper adapter used. Will make the method signature less readable and 
// crash if called with no adapter installed.
Realm.createObjectFromJson(Class, Object);
Realm.createAllFromJson(Class, Object);
Realm.createOrUpdateObjectFromJson(Class, Object);
Realm.createOrUpdateAllFromJson(Class, Object);

// Version 2

// Each adapter exposes a init method + a controller class
// Having a really hard time convincing myself this is a good idea

// Init
RealmConfig config = getConfig()
RealmJson.init(realmConfig) // Controller class for each adapter (RealmJson/RealmJsonStream/RealmJsonObjects)

// Usage
realm.beginTransaction();
RealmJson.createOrUpdateObject(Realm, Class, Object);
realm.commitTransaction();

[beeender]

Split the json to another project would be great, especially we will have a chance to switch to javapoet from there. But the new project probably needs to be more generic to support other data converters than JSON, eg. flatbuffer.

[taltstidl]

Yes, I would agree that splitting this into its own project sounds like the best solution. This gives us the freedom to add additional import formats as well (XML, CSV, you name it...), though I'd focus on JSON as it's probably the most widely used format.

In order to prepare for splitting, a few design considerations need to be made first:

Where do the import/export methods go?

• For Java this would probably need to be an extra class, not sure though how it should be named (RealmJson, RealmData, RealmConverter, ...). @cmelchior Does it really require an init method to work properly? I would think not, but I might be wrong.
• For Swift and Objective-C this could be an extension to Realm or RLMRealm respectively, though an extra class would facilitate cross-platform consistency.

What are the import/export methods named?

• For Java I'd propose something similar to importDataFromJson or importDataFromXml.
• For Objective-C and Swift all sources could be consolidated into one method, called importData (note that import is not a good naming choice as it's a reserved keyword in Swift).

How are the import/export method realized?

• In Java this will probably require an annotation processor that runs after Realm's proxy generator and adds the required method for importing/exporting. Serialization attributes are specified using the @SerializedName annotation.
• In Objective-C and Swift there isn't a similar concept I know of, so I'd rely on key-value coding and serialization maps to realize the import/export methods. This would probably require an additional method serializedProperties for the objects which returns the mapping structure for the object (similar to how ObjectMapper does it).

Chances are that I missed something above, but that's the general direction I had in mind. First-class support for importing and exporting common formats can greatly increase the performance of the code and the speed of development using Realm!

[taltstidl]

@cmelchior Any update on that internal discussion? Thanks! 😉
By the way, do you have any statistics on how often the InputStream flavors are used? I'm not sure I have seen that approach very often, mostly only the JSONObject flavor is used.

[cmelchior]

Hi @TR4Android Sorry not yet. I'm writing up a proposal with use cases and solutions right now just so we are sure we got all present and future use cases covered.

We don't have any stats for when either is used. A gut feeling is that most people go through GSON and similar instead of using our methods for it. After that, it would assume that String/JSONObject are the most frequently used, yes.

[taltstidl]

@cmelchior No problem, I'm as interested in getting this right as you guys are! I appreciate the time you're investing in a proposal, keep up the good work 👍
If there's anything I can help with let me know!

[taltstidl]

@cmelchior Any update? 😉

[cmelchior]

Hi @TR4Android Sorry for the late turn-around time, we just came back from a company wide get-together. I posted some thoughts in #3758

[dhuma1981]

Any update on this issue? Eagerly waiting for it.

[abou7mied]

Until this issue is resolved I have made a workaround in my App, let's say I have
Chat.java

public class Chat extends RealmObject {
    @Index
    @SerializedName("c")
    private String code = "";

    @Index
    @SerializedName("t")
    private int type;
}

And RealmUtil.java

public class RealmUtil {


    private static HashMap<String, HashMap<String, String>> classesFieldsMap = new HashMap<>();

    static {
        initClasses(Chat.class);
    }

    private static void initClasses(Class cls) {
        if (!classesFieldsMap.containsKey(cls.toString())) {
            Field[] fields = cls.getDeclaredFields();
            HashMap<String, String> fieldsMap = new HashMap<>();
            for (Field a : fields) {
                SerializedName annotation = a.getAnnotation(SerializedName.class);
                if (annotation != null) {
                    fieldsMap.put(annotation.value(), a.getName());
                }
            }
            classesFieldsMap.put(cls.toString(), fieldsMap);
        }
    }

    public static JSONObject mapGsonObjectToRealm(Class cls, JSONObject object) throws JSONException {
        JSONObject newUserObj = new JSONObject();
        HashMap<String, String> fieldsMap = classesFieldsMap.get(cls.toString());
        for (String setKey : fieldsMap.keySet()) {
            String mappedKey = fieldsMap.get(setKey);
            if (object.has(setKey))
                newUserObj.put(mappedKey, object.get(setKey));
        }
        return newUserObj;
    }
}

By using reflection initClasses method will map each of the Gson SerializedName of the class to the corresponding names for Realm record.

And mapGsonObjectToRealm method will create a new object of the old one with new fields names.

Usage:

JSONObject newObject = RealmUtil.mapGsonObjectToRealm(Chat.class, new JSONObject().put("c", "this_is_code").put("t", "this_is_type"));
final JSONArray jsonArray = new JSONArray();
jsonArray.put(newObject);
Realm.getDefaultInstance().executeTransactionAsync(new Realm.Transaction() {
    @Override
     public void execute(Realm realm) {
        realm.createOrUpdateAllFromJson(Chat.class, jsonArray);
    }
});

[Zhuinden]

Realm.getDefaultInstance().executeTransactionAsync(

I wonder how many articles I need to write about how wrong this is before people stop doing it.

---

Btw, are you sure you need to do this reflection magic manually? I think this is what Gson does with gson.fromJson(string, class).

Please be aware that this probably wouldn't work without the right Proguard rules.

[abou7mied]

@Zhuinden gson.fromJson will return an object of the class param and If some fields are missing in the data received from the server, say (t field in Chat.java) then It will be 0 in object and will be saved 0 in Ream DB.
But with JSONObject Realm will just update the existing fields.

[Zhuinden]

@abou7mied AH i understand, I guess that makes sense.

Make sure you apply proper Proguard configuration though!

[Zhuinden]

Jokes aside, the ability to unlink the schema field name from the actual Java field name using an annotation would be nice, and should be completely independent from JSON adapters.

[felixklauke]

Any update here? I would prefer a solution without this hacky gson way. We would really appreciate a simple solution with a single annotation.

[abou7mied]

Any updates?
I made a lib for my project to map from json keys to realm java-keys and use createOrUpdateObjectFromJson for the mapped object but it would be good to make it using Realm without workarounds

[ishaileshmishra]

Hi there, Is there any way to get the list of values declared in @Fieldname("internal_name"), let's suppose I have declared 10 fields annotated with FieldName with values in it, So how would I get 10 values?

[cmelchior]

Depends on what you need. If you need only the fields annotated that way, then the only way is to use reflection on the model class. If you use realm.getSchema().get(Foo.class.getSimpleName()).getFields() it will return all the internal field names in the class.

[ishaileshmishra]

Hi @cmelchior Thanks for your quick reply, I need only fields annotated by @Fieldname

[cmelchior]

Then you need to use reflection. This should work:

        for(Field field  : Foo.class.getDeclaredFields()) {
            if (field.isAnnotationPresent(RealmField.class)) {
                RealmField ann = field.getAnnotation(RealmField.class);
                String internalName = ann.name();
            }
        }

[ishaileshmishra]

Thanks @cmelchior for your support, it worked.

[abou7mied]

Hi guys,
I made a library to help me to map SerializedNames to Realm names.

I don't know if it is a good practice to do such a workaround or not, anyway I would like to share it with you
https://github.com/abou7mied/realm-useful-helpers

[BoD]

Just wondering if this will be released at some point? In the meantime I'll be using https://github.com/cmelchior/realmfieldnameshelper which seems perfect - but it would be nicer to have this directly in Realm.

[Zhuinden]

I mostly wish for RealmFieldNamesHelper to be updated to be incremental annotation processing