Slow performance with 100 children

[stas-sl]

Hi.

I have a model with ~100 child models. When I load it for the first time (with sideloading children), it loads quite fast. But if I'd like to reload it from server for the second time it takes much much longer.

Am I using it wrong way? How do you use it in your applications? Or you have only relatively small number of models? Or you do not reload them if they are already loaded? But what if something changed on the server?

Another idea I had is to use session.remove for all posts and comments before loading them for the second time. But I got strange errors after removing and loading again. Seems like remove doesn't clear everything correctly. Is remove supposed to be used in such scenarios?

describe "rest", ->

  adapter = null
  session = null

  beforeEach ->
    require('./_shared').setupRest.apply(this)
    adapter = @adapter
    session = @session
    Ep.__container__ = @container

    class @Post extends Ep.Model
      comments: Ep.hasMany 'comment'

    @App.Post = @Post

    class @Comment extends Ep.Model
      post: Ep.belongsTo 'post'

    @App.Comment = @Comment

    @container.register 'model:post', @Post
    @container.register 'model:comment', @Comment

  afterEach ->
    delete Ep.__container__

  context 'post with 100 comments', ->
    beforeEach ->
      adapter.r['GET:/posts'] = posts: [{id: 1, comments: [1..100]}],
      comments: ({id: num, post: 1} for num in [1..100])

    it 'fresh load works relatively fast', ->
      session.query('post').then (posts) ->
        expect(posts[0].comments.length).to.eq(100)

    it 'load if post and comments already in session is very slow', ->
      session.query('post').then (posts) ->
        session.query('post').then (posts) ->
          expect(posts[0].comments.length).to.eq(100)

  rest
    post with 100 comments
      ✓ fresh load works relatively fast (386ms)
      ✓ load if post and comments already in session is very slow (19613ms)


  2 passing (20s)

[ghempton]

My guess is that this is due to merging all of the children in. One way to make this way faster an avoid unnecessary merges is to pass a rev property down from the server (this could be as simple as serializing the updated_at column into a timestamp if you are using rails). By default, epf will check this property and know if it already has seen this version.

If you don't pass this property, epf always assumes all the data contains updates and performs a merge algorithm against it.

Let me know how it goes, cheers!

[ghempton]

Btw, some perf tests like those would be great to have in epf itself, would much appreciate a PR if you get this resolved.

[stas-sl]

Thanks, you are right, adding rev to the mock data speeded up the test 10x times from 20s to 2s.

adapter.r['GET:/posts'] = posts: [{id: 1, comments: [1..100], rev: 1}],
comments: ({id: num, post: 1, rev: 1} for num in [1..100])

rest
    post with 100 comments
      ✓ fresh load works relatively fast (345ms)
      ✓ load if post and comments already in session is very slow (19712ms)
      ✓ load if post and comments already in session with rev specified (2140ms)

Though it is already quite acceptable time, I'm wondering if you know other 'secrets' to speed up it a bit more :)

I can try to make a PR, but not sure what is correct way to implement performance tests. Is it ok to measure execution time in test? Or just check that data is correct as it is already done?

[ghempton]

Being as we don't have any perf tests yet, I'm fine with any mechanism. It could be enough to simply have the test present, but if there was some way to get the execution time inside and after hook and warn if it changes above a threshold that would be a step up.

No other big low-hanging perf tricks that I can think of off the top of my head. In our app in extremely perf sensitive areas (e.g. rendering lists of hundreds of items) we generally just load the raw json (you can do this by passing serialize: false) and then manually merge in the data using session.mergeData when we need it. In these areas we also do the rendering manually instead of using ember's normal template bindings (by overriding the render method to render a raw handlebars template).

[stas-sl]

Thanks for the tips, I'll keep in mind them.