Emberfire version 3

[jamesdaniels]

Emberfire v3 Proposal

I propose a rewrite of Emberfire. I've made a proof of concept application which I'll extract the Firebase bits from and put into a WIP branch this week.

Before I jump into the what, I'd like to touch on the why. A lot has changed since emberfire v2 was released:

• ember-data has stablized
  • we've lost life-cycle events we need to do realtime correctly
  • we're facing deprications every release
  • we can simplify the adapter significantly now
• Cloud Firestore, Firebase's next-generation datastore has been released in beta
• The Realtime Database now has query rules
• Ember Fastboot is stable and growing in use
• The Firebase JS SDK has matured significantly, added new API surfaces, and broken old ones
• I've become a maintainer of angularfire and have "seen the light"
• And frankly, we don't have as many Googlers interested in helping this project

We've also had a number of issues, since the inception of the project, which we've not fundamentally addressed:

• Relationships are fragile
• The realtime listeners are next to impossible to disconnect, when you don't need them
• Emberfire doesn't work well with the data-structures you need to make Firebase Realtime Database fast and secure
• Emberfire doesn't play nice with Firebase Security Rules

All of this, to me adds up to a new version; while of course trying to keep the API as close as possible.

Rewrite Goals

• Keep the API as close to v2 as possible
• Get caught up with Firebase JS SDK
• Add Cloud Firestore support to the adapter
• Add phone number authentication
• Support Ember Fastboot and Shoebox
• Decrease maintence burden
• Increase project agility

New API

Two adapters! Firebase Realtime Database and Cloud Firestore.

Also allow configuration of multiple Firebase applications and add multi-database support. This API is still being thought out.

Changes to the ember-data adapters

Querying

Fundamentally, unchanged, regardless of the adapter chosen.

this.store.query('post')

But what if you want the last 10 published posts? Rather than wrapping the query interface for RTDB and Firestore, I propose we take a lesson from angularfire and just allow modification of the Firebase Query:

this.store.query('post', ref => ref.orderBy('publishedAt', 'desc').limit(10));

Moving realtime updates to a service

One of Firebase's super-powers is it's realtime subscriptions, allowing you to get changes to your model and query in milliseconds. This however violates every assumption that ember-data makes + makes integration with Fastboot and Shoebox difficult. We've also lost life-cycle events we need to do realtime in an efficient manner.

I propose a model mixin, which would interact with a new realtime-service (rather than shoving everything in the adapter). This mixin would giving us the unload detection we need to do things efficiently, start/stop listening methods, etc.

Mixin:

TDB

Service:

const listener = this.get('firebaseRealtime').listenForChangesOn(model);
listener.unsubscribe();

Now we'd be playing nicely with Fastboot, Shoebox, reduce our adapter complexity by an order of magnitude while giving you much more control over your subscriptions.

Putting this together:

import Ember from 'ember';
import Route from '@ember/routing/route';

const { inject: { service } } = Ember;

export default Route.extend({
    fastboot: service(),
    realtime: service('firebase-realtime'),
    realtimeListener: null,
    async model() {
        const isFastBoot = this.get('fastboot.isFastBoot');
        if (isFastBoot) {
            const responseHeaders = this.get('fastboot.response.headers');
            responseHeaders.set('Cache-Control', 'public, max-age=300, s-maxage=600');
        }
        const shoebox = this.get('fastboot.shoebox');
        const shoeboxPosts = shoebox && shoebox.retrieve('posts');
        if (shoeboxPosts) {
            return shoeboxPosts;
        } else {
            const posts = await this.store.query('post', ref => ref.orderBy('title').limit(3));
            if (isFastBoot) { await shoebox.put('posts', posts); }
            return posts;
        }
    },
    afterModel(model) {
        const listener = this.get('realtime').listenForChangesOn(model);
        this.set('realtimeListener', listener);
    },
    deactivate() {
        const listener = this.get('realtimeListener');
        if (listener) { listener.unsubscribe() }
    }
});

Types

Strong typing in Firestore

In addition to serializing the built-in ember-data types to something reasonable on the Firestore side; we should add new ember-data types for those strong types, so emberfire will play nicely with native applications:

• firestore.array
• firestore.geo
• firestore.bytes
• firestore.serverTimestamp
• firestore.integer
• firestore.float
• firestore.map
• firestore.reference

I'm still working these out.

hasMany

hasMany relationships are where we need to improve the most.

Developers can now secure their queries in the Realtime Database, so we should back hasMany with a query rather than using embedded ids; which were very fragile before.

We also have sub-collections in Firestore.

Options:

Option	Argument
query	(Database) => Query	Overwrite the assumed Query, bound to model
embedded	any	If truthy, embed the records in this document
subcollection	any	If truthy, embed the records in a sub-collection (Firestore-only)

Putting this together:

import DS from 'ember-data';

const { attr, hasMany } = DS;

export default DS.Model.extend({
    name: attr('string'),
    createdAt: attr(),
    updatedAt: attr('firestore.serverTimestamp'),

    location: attr('firestore.geo'),
    tags: attr('firestore.array'),
    publicKey: attr('firestore.bytes'),
    
    // Assumed to be at /Comments/* in the database with a `user` attribute with a reference to this document
    comments: hasMany('comment'),
    
    // You can overwrite the assumed query using the query option
    ideas: hasMany('idea', { query: (database) => database.ref(`Ideas/${this.id}`) }),
    
    // Use the embedded option to store in the same document in RTDB OR as a subcollection in Cloud Firestore
    thoughts: hasMany('thought', { subcollection: true }),
    invitations: hasMany('invitation', { embedded: true })
});

Authentication

Here I propose two changes, the first of which is supporting Ember Simple Auth out of the box. Should be fairly self explanatory ;)

The next, no longer wrapping any Firebase SDK calls. Neither the Torii or Simple Auth adapters will have login options themselves, instead logging in should be done with the Firebase JS SDK. This simplifies our code and increases our agility (there's been a lot of churn on these APIs) while giving you all the power of working with the authentication calls yourself.

import Controller from '@ember/controller';
import Ember from 'ember';

import firebase from 'npm:firebase';
const { inject: { service }} = Ember;

export default Controller.extend({
    session: service('session'),
    app: service('firebase-app'),
    actions: {
        logout() {
            return this.get('session').invalidate();
        },
        login() {
            const provider = new firebase.auth.GoogleAuthProvider();
            return this.get('app').auth().signInWithPopup(provider);
        }
    }
});

{{#if session.isAuthenticated}}
    <button {{action 'logout'}}>Logout</button>
    <p>Signed in as {{ session.data.authenticated.user.displayName }}</p>
{{else}}
    <button {{action 'login'}}>Login</button>
{{/if}}

{{outlet}}

Summary

First and foremost, we should not be wrapping the Firebase JS SDK! The Firebase JS SDK is battle tested and Promise based, so it plays nicely with the Ember ecosystem out of the box. We shouldn't do anything that gets in the developer's way or decreases our ability to keep up with the Firebase SDK, like inventing our own API.

Second, support Cloud Firestore as an ember-data adapter. Firestore is Firebase's next-generation datastore and what we're recommending people to use. It combines all the awesomeness of the Realtime Database with Google's planet scale goodness and querying capabilities. Seriously check it out.

Finally, fix all the deficiencies of the project: Fastboot support, Realtime Listeners, hasMany relationships, and give the adapter options that allow you to work around Firebase's lack of a intermediate server (filtering, schemas, etc.) while decreasing the amount of code.

Let's discuss!!! I'll post my WIP branch shortly.

[jamesdaniels]

Pinging #531, #529, #522, #511, #465, #366, @olivierchatry, @rmmmp, and @tstirrat

[jamesdaniels]

These APIs are a work in progress of BTW, really looking forward to pulling in feedback.

[mikkopaderes]

@jamesdaniels, first of all thanks for putting this out. Really appreciate it.

I agree with not wrapping Firebase JS SDK in Ember Data. Though sometimes it just can't be help if we want a good developer experience in Ember. ;)

Anyways, here's my comments.

Queries

I really like the concept of using the Firebase API instead of the old one. However, this wouldn't work in the adapter as it's expecting an object rather than a callback. You'll have to extend the store instead if you really want to push it. I think it'd be better if we place it inside an object instead of passing a callback.

this.get('store').query('post', {
  filter(reference) {
    return reference.limit(10);
  }
});

Now we can add more configs in the object should there be any. I think one config we can add is to be able to specify what collection we'd like to query the data from.

const userId = 'user_a';

this.get('store').query('user', {
  // Not sure if userId would be looked up correctly here.
  // Maybe we could change it to buildReference: (db) => {} instead to make it work.
  buildReference(db) {
    return db.collection('users').doc(userId).collection('friends');
  },

  filter(reference) {
    return reference.limit(10);
  }
});

Without providing a buildReference() hook, the reference passed-in to filter() will point to the pluralized name of the query model name's collection db.collection('users').

Realtime service

Not totally convinced here. This will add a lot of codes for just to consume what brought many of us to Firebase in the first place.

I think we can provide a configuration for every request to opt-in users for the realtime feature.

this.get('store').query('user', {
  isRealtime: true,

  filter(reference) {
    return reference.limit(10);
  }
});

this.get('store').findRecord('user', 'user_a', {
  adapterOptions: {
    isRealtime: true
  }
});

this.get('store').createRecord('user', {
  name: 'Foobar',
  age: 10,
}).save({
  adapterOptions: {
    isRealtime: true
  }
});

The Adapter would definitely be more bigger and complex but those issue can be managed by clean codes. In fact, I think it'd be more complex if we have a separate service for that. I mean, how would the service figure out what limit, orderBy, collection path did you use by passing a resolved record(s) in realtime.listenForChangesOn()?

As for FastBoot, the Adapter must be forced to not fetch realtime changes regardless of the configuration when in that environment so the user wouldn't have to compromise for that. They can use Shoebox if they're not utilizing the realtime feature. Shoebox is pointless for apps that use realtime fetching as you'll end up fetching twice either way.

Custom transforms

Generally +1 on this. Although I really think the approach should be to only update the changed attributes when updating. This should fix the security rules problem of not allowing some properties to be updated and we wouldn't need to have the onlyOn option.

I can't imagine how readOnly will be useful. What are the use-cases? If there are any, I think this should be something that should be built-in in Ember Data rather than here.

hasMany

My unpopular opinion for hasMany is to not use it. It's hard to implement for schema-less databases and costly for services that rely on number of reads for pricing when every record is fetched in one go. I'd say use query() instead of hasMany.

But how would we save the relationships if we use query() instead of hasMany? The way I did it in my custom adapter is to provide a way to batch writes and save your relationships from there.

Nevertheless, there'd be some that prefer to use hasMany as it looks more ergonomic. So putting aside what I've said above, shouldn't the IDs be embedded on the belongsTo side? Also, maybe there could be a junction node/collection for many-to-many and many-to-one relationships (could be a subcollection on both sides too in Firestore). I just don't think it's practical to embed IDs in hasMany. That's just doesn't scale well. Plus I think there's a limit on the amount of props in a record in Firestore (correct me if I'm wrong).

Authentication

Haven't used ember-simple-auth. How will we integrate the session service with Firebase? Signing in with Firebase doesn't make the session.isAuthenticated() set to true, right?.

Other stuff

For Cloud Firestore, I think we should extend the Model and provide a client-side only attribute that points that to its reference in the DB. This would be useful for querying records using a firestore.reference type attribute.

this.get('store').query('post', {
  filter(ref) {
    return ref.where('author', '==', user.get('__reference__');
  }
});

Instead of

const db = firebase.firestore();

this.get('store').query('post', {
  filter(ref) {
    return ref.where('author', '==', db.collection('users').doc('user_a'));
  }
});

Lastly, the firebase service as a separate addon here is just something that would be a nice to have. I already created an addon for that but it would be just nice if it's officially owned by Firebase. :)

[olivierchatry]

I agree with everything @rmmmp wrote.

hasMany is really bad in the way it is handle by ember-data as you need to do multiple save, and can end up in a broken database state. So doing a multipath update manually seems like a good idea ( even if it is more work ).

More importantly, making realtime update a kind of second class citizen ( meaning you have to take care of it yourself ) remove most of the interest in using emberfire and firebase.

[jamesdaniels]

RE Realtime:

So my primary concern is how to stop listening, as we've lost those lifecycle events around unloading and I don't want developers to have to pull mixins into their models. In other frameworks that have embraced Rx I can just use Observables, sadly that's not the case in Ember.

@rmmmp in my current proof-of-concept, i've been putting the ref/query in the metadata. That's how the service can pick it up from the adapter. I'm also working to get an API to pass the query hash into the Firebase SDK, so on rehydration the data won't be fetched twice; just delta updates. This is a priority for all of our SSR goals in the next two quarters.

I could see, by default, having the listener attached (w/an option to disable) + then being able to detach it with the service. Let me think on this...

[jamesdaniels]

RE hasMany:

Agreed, in my proof of concept; I've hacked the adapter to just make this a query; unless the embedIds option is on there.

What I'm hearing here is that the whole embedIds option would be a nightmare and that's one of the things I want to avoid (at least by default). I could see absolutely cutting that entirely and instead pointing users that want that (for Firestore has-and-belongs-to-many type querying) to a doc describing how to do that with Cloud Functions.

Also yes, composite indexes in Firestore are limited to 20k keys.

[jamesdaniels]

RE onlyOn & readOnly:

Two things:

A) maybe it's just scar tissue from my using ember-data before it was fully baked but I don't trust ember-data's dirty state tracking, is this something I should be trusting absolutely now? Part of the problem here is that when interacting with a REST API, the backend-developer can just ignore attributes the user doesn't have permission to write (e.g, Rails). Whereas with Firebase Security Rules, this write will fail and data rolled back.

B) I'm also anxious about the case of saving an "out of date" model. I'd rather last write win, if they aren't using a transaction, over surprising the end user / developer by only updating the dirty attributes. @olivierchatry @rmmmp thoughts?

[mikkopaderes]

@jamesdaniels for stopping the listener, wouldn't extending store.unloadRecord() and store.unloadAll() be enough?

Dirty state tracking works fine although it's only for attributes. Relationships doesn't have a dirty state.

I'm a little bit confused with the security part. How I write mine is I allow updates "generally" but don't allow a specific field to change data.val() == newData.val() so I've never had problems of updating only the changed attributes.

I'm not sure what you mean in the race-condition/transaction issue. Can you give an example?

[jamesdaniels]

@rmmmp the adapter is no longer informed of unload, I don't know what version of ember-data that changed in. A mixin would be necessary to disconnect the listener and I'd rather not go down that route.

True, relationships not having dirty state is more points towards dropped embedded ids entirely.

RE the race condition, consider a user updating the same record as another but they're out of sync, maybe because one was offline or they hit save within the update window. If we're only updating the dirty attributes, say one user updating Title, another updating Body then the result will be a blended version of the two vs. last write wins. Maybe this isn't an issue but especially if backed by a form I feel this violates expectations. Maybe we give the option for a save w/transaction?

[jamesdaniels]

@rmmmp we could do automatic realtime listeners if the developer is willing to pull a firebase-realtime mixin into their model. Seem like a fair compromise?

Another thought, could we achieve the same thing with a decorator?

[olivierchatry]

I was thinking about that. I think it would be ok to have a firebase-model and hook on load / unload.

…

On Wed, Jan 31, 2018 at 8:57 PM, James Daniels ***@***.***> wrote: @rmmmp <https://github.com/rmmmp> we could do automatic realtime listeners if the developer is willing to pull a firebase-realtime mixin into your model. Seem like a fair compromise? Could we achieve this with a decorator? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/firebase/emberfire/issues/542#issuecomment-362052029>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNXG52KTkn50h_P7_PCxPLZydGEcAU5ks5tQMXBgaJpZM4RzBso> .

[mikkopaderes]

@jamesdaniels I'm not sure why you need the adapter to be informed of the unload. When someone calls the store.unloadRecord(record) we could just get the unsubscribe reference from the passed in record then call in this._super(record) to continue with what store.unloadRecord() does.

Am I missing something? Why do we need a lifecycle hook for this?

For the race-condition, I have no strong opinion on that. I don't even think that the first update of user-1 would reflect on user-2 as store.push does nothing if a record is already dirty. That means user-2 would have to refresh in order to see the first update or wait until firebase does another sync for another update. This needs to be reconfirmed though since I'm not sure if I'm remembering things correctly.

[mikkopaderes]

Here's also a snippet on how I've written my rules with Firebase before that allows updating "generally" but not allow a createdOn timestamp property to be modified.

"posts": {
  ".read": "true",

  "$post_id": {
    "createdOn": {
      ".validate": "newData.isNumber() && newData.val() == (data.val() == null ? now : data.val())"
    },
    ".write": "auth != null",
  }
}

With the security rules above, updating only dirty attributes should work.

[jamesdaniels]

Alright, removed the embedded ids and attr options from the proposal. Happy to simplify, going to play around with my proof-of-concept app tomorrow and see how I am feeling RE realtime listener patterns.

[elgordino]

Great work with this @jamesdaniels, I'm excited to see the progress :)

I am very happy to see the back of embedded id's for hasMany relationships. I have a lot of custom work outside of ember-data to support these atomically at the moment, which I would be very happy to remove!

Embedded HasMany

I currently use embedded hasMany in a few places. My use case is record A hasMany B and there are 100+ B records and they always need to be fetched when A is fetched. I changed these records to be embedded as the performance of retrieving each B record individually was too slow. In the new proposal the hasMany retrieval would be a single query, so would it be appropriate to abandon the embedded pattern and just use the new query based pattern?

HasMany <-> HasMany:

How would relationships that are hasMany<->hasMany work with the query pattern? Would we need to move to an intermediate model?

Realtime:

A lot of my records are loaded by ember-data through templates, rather than explicitly in the model hook. I don't think I would have a change to register them with a service. Either extending DS.Model or providing a mixin to might be enough to cover the situation of automatically loaded data?

The majority of times I request data I want to retrieve realtime updates. Looking at my configuration this divides on model types. Maybe consider adding a property to the Model to define the default behaviour, realtimeUpdates: true/false or something?

I do need to be able to completely unload records and removed any active ref's, this is broken in my application at the moment for the reasons being discussed. During a users session their firebase authentication can change, so I unload the records owned by the previous session. This was working fine until the changes to life cycle hooks.

With the query approach for hasMany I guess unloading the LHS of the relationship would cause the listener to be removed, and records on the RHS would remain, without realtime updates until also unloaded?

Atomic Saving of Relationship changes:

In order to save relationships atomically, I have a 'relationships' service. This updates multiple ref's in an atomic operation. In the proposal hasMany<->belongsTo relationships are no longer a problem with the query pattern. But I think belongsTo<->belongsTorelationships are still a problem as are hasMany<->hasMany.

Other:

I group all records belonging to a 'team' under /team/<team id>. so records are at /team/<team id>/things/<thing id>. Using some fiddling with ref's I did. This was mainly created for security rules purposes, but it strikes me that the layout maps quite well onto Firestore Collections. Perhaps when considering adding Firestore support, the collections pattern could also come to RTDB?

Off topic:

I would find it very useful if, in the SDK, there was a way to list all listening ref's.

[mikkopaderes]

Going back to hasMany, we could probably add the same APIs that we have on query() in it.

import { hasMany } from 'ember-data/relationships';
import Model from 'ember-data/model';
import attr from 'ember-data/attr';

export default Model.extend({
  age: attr('number'),
  username: attr('string'),
  friends: hasMany('user', {
    filter(reference, record) {
      return reference.limit(10);
    }
  }),
});

Here's how to change the relationship option dynamically for use-cases such as changing filter hook to increase the limit for infinite scrolling/pagination.

Disclaimer: I'm not sure if changing the relationship option has any drawbacks in Ember Data or if this is even the recommended way. I assume that there's none as long as you don't change built-in options such as async, inverse, etc..

const user = await this.store.findRecord('user', 'user_a');
const friends = await user.get('friends');
const newLimit = 20;

friends.relationship.relationshipMeta.options.filter = (reference, record) => {
  return reference.where('age', '>=', record.get('age')).limit(newLimit);
};
friends.reload();

This may be enough to fix the bad rap against using hasMany in Firebase... I think.

[erichiggins]

Since I haven't seen it mentioned here, will this rewrite also support Firebase Storage which also appears to be missing from Emberfire?

[olivierchatry]

Not sure how it can fit with ember data.

…

On Feb 26, 2018 23:16, "Eric Higgins" ***@***.***> wrote: Since I haven't seen it mentioned here, will this rewrite also support Firebase Storage which also appears to be missing from Emberfire? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/firebase/emberfire/issues/542#issuecomment-368750428>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNXG_6ppsA4rnWjFFMyGr8wtlEWnM-3ks5tY4_HgaJpZM4RzBso> .

[erichiggins]

@olivierchatry Honestly, I'd be happy if there was just a generic ember service included. I've had to include the following in all of my projects and I'm not even sure if it's correct.

import { getOwner } from '@ember/application';

import firebase from 'firebase';


export default {
  create(application) {
    const config = getOwner(application).resolveRegistration('config:environment');
    if (!config || typeof config.firebase !== 'object') {
      throw new Error('Please set the `firebase` property in your environment config.');
    } 
      
    let app;
  
    try {
      app = firebase.app();
    } catch (e) {
      app = firebase.initializeApp(config.firebase);
    }
  
    return app.storage().ref();
  },

  config: null,
  isServiceFactory: true
};

[zurnet]

Looking forward to this project, and checking in to see if there are any updates with EmberFire in regards to fixing any issues (babel versions) and other bugs until support for FireStore is ready :)

[brancusi]

Definitely looking forward to this!

[jamesdaniels]

Release candidate 1 has been cut. https://github.com/firebase/emberfire/releases/tag/v3.0.0-rc.1

[TorstenDittmann]

For all people having problems with the lack of documentations right now, I really suggest looking at the dummy-app in the test folder:
click here

This really helped me alot!

[jamesdaniels]

Good point @tdiddi, thanks. I'll work on getting the documentation all back in order shortly. Beyond the one or two major gaps that's my next blocker to calling it 3.0 final.

Feel free to provide any feedback you'd be up for sharing about your experience with the RC, I can be reached at jamesdaniels@google.com

[TorstenDittmann]

Good point @tdiddi, thanks. I'll work on getting the documentation all back in order shortly. Beyond the one or two major gaps that's my next blocker to calling it 3.0 final.

Feel free to provide any feedback you'd be up for sharing about your experience with the RC, I can be reached at jamesdaniels@google.com

I will for sure @jamesdaniels , after I am prototyping my project. So far the "new" emberfire is really awesome. Keep up the work!

[jamesdaniels]

FYI we sorted out NPM details and replaced the legacy build tools that were holding us back a bit.

Release candidate 2 was just published at emberfire@next, emberfire-exp has been deprecated.

[jamesdaniels]

EmberFire 3.0.0-rc.3 has been published. We now support Firebase v6 & use dynamic imports to bring in Firebase modules lazily, so your main bundle size is not impacted.