TrackedList

[pzuraq]

Rendered

[josemarluedke]

Suggested change

	@friends = ['Yehuda', 'Melanie', 'Ricardo'];
	@tracked friends = ['Yehuda', 'Melanie', 'Ricardo'];

[josemarluedke]

Suggested change

	@friends = A(['Yehuda', 'Melanie', 'Ricardo']);
	friends = A(['Yehuda', 'Melanie', 'Ricardo']);

[chriskrycho]

I'd add to this that there's another (and IMO even more important than paradigm-preference) reason that this is valuable: performance. Immutable functional updates have many advantages, but without something like persistent data structures, they can carry a high performance penalty.

TrackedList allows users to make that tradeoff as they like, whatever their programming paradigm preference—and there's no reason that you can't provide a functional abstraction around it if that's your jam.

[chriskrycho]

One concern I have with this name is that List suggests, well, lists, which are a standard abstraction with specific characteristics. At a minimum we will need to document how this List type is implemented, so that people will have a sense of the performance characteristics – and specifically, if this is implemented on top of JS arrays rather than as a linked list, we should be explicit about that (presumably adding a note to that effect in the How We Teach This section).

[chriskrycho]

Borrowing an idea from Swift, perhaps we should name these sorted and reversed, the difference of names implying the difference in outcome?

[chriskrycho]

It'd be great if we included the full API spec in this section!

[chriskrycho]

Whoops: incomplete sentence!

[chriskrycho]

Is TrackedArray intended to be subclassed? If so, what invariants do subclasses need to uphold?

[hjdivad]

I had exactly this question.

Also, what is the intended API for a subclass to manually invalidate itself?

[pzuraq]

TrackedList is not intended to be subclassed, though there isn't a way to prevent it in JS, and there is no API for manual invalidation. If users want to implement a custom iterable, the recommendation would instead be to use the standard Symbol.iterable API to do so, as it should be fully supported by Ember at this point. If there are still some gaps, they should probably be addressed in a separate RFC.

[chriskrycho]

Suggested change

	`length` will be an immutable/readonly property on `TrackedList`. This is to
	`length` will be a readonly property on `TrackedList`. This is to

I think if I'm understanding the intent here correctly, we should describe this as readonly but not immutable, since it will change if users push or pop from the list. (With immutable data, I'd expect that the value will never change in the life of the instance; the only way to get a new value is to get a new instance.)

Also, is length auto-tracked? (I'd assume yes, but I'd love if it were explicit here.)

[chriskrycho]

A couple questions as I'm considering this:

• What would those "potentially divergent behaviors be"?
• Is it possible to ship a TrackedArray now, with future performance improvements unlocked when IE11 is removed (whenever Ember 4 ships), and if so, how do the tradeoffs of doing that compare to the tradeoffs of introducing this class now and then potentially another class later?

[pzuraq]

1. Specifically, the two divergences in this RFC are:
   • The difference in behaviors of sort and reverse
   • The readonly nature of length
2. It is not possible to ship TrackedArray without native Proxy, so we would have to wait for IE11 support. The native Proxy version is still much much slower than a native array based solution, by at least an order of magnitude, currently.

[chriskrycho]

Thanks for the clarifications!

[bendemboski]

Here's something to maybe argue about here, or maybe just to add to the drawbacks section (or as a sub-point of the first drawback). I'm generally not a fan of object APIs that look really really like standard/native object APIs, but differ slightly. All too often I find one of the significant challenges in reading and understanding code in dynamically typed languages is figuring out what a particular thing is. Good documentation or, better yet, typescript should largely ameliorate this, but it's still a significant factor.

So I'm not a huge fan of implementing something that looks really really like a native array, but is missing one or two pieces ([] getter/setter). With EmberArrays, a very high percentage of API calls are typically using the non-native(-looking) methods, so it's generally easy to figure out when something is an EmberArray and I need to use its methods. But I worry that it will be fairly common to encounter

function doSomeStuff(items) {
  let things = items.map((i) => i.getThing());
  for (let thing of things) {
    thing.doStuff();
  }
}

and maybe change it to

function doSomeStuff(items) {
  items[0].getThing().doStuff();
  let things = items.slice(1).map((i) => i.getThing());
  for (let thing of things) {
    thing.doDifferentStuff();
  }
}

except whoops, items is (or sometimes is) a TrackedList and my code blows up, or sometimes blows up because I can't [0].

I'm kinda ambivalent because I'm not sure I'd really prefer a situation where all the methods do the same thing as Array methods, but have different names (like EmberArray). But whether I want to or not, I often rely on mental duck typing when reading Javascript, and having secret agent impostor ducks makes thing harder.

[lifeart]

Can't we use [0] for values access/write?

kinda:

const impl = Symbol();
function defineKey(ctx, key) {
    if (key in ctx) {
        return;
    }
    Object.defineProperty(ctx, String(key), {
        enumerable: false,
        configurable: true,
        get() {
            return ctx[impl][key];
        },
        set(value) {
            ctx[impl][key] = value;
            defineKey(ctx, key + 1);
        }
    })
}

class MyArray  {
    constructor(orig = []) {
        Object.setPrototypeOf(this, Array.prototype);
        this[impl] = orig;
        defineKey(this, 0);
        Object.defineProperty(this, 'length', {
            enumerable: false,
            get() {
                return this[impl].length;
            },
            set(_) {
               // noop
            }
        });
        return this;
    }
}

Writing to "uninitialized" index of array - it's really "corner" case and we can skipt it at all, mention it as anti-pattern in documentation.
if we allow grow array natively, we can define setter for next index once first used

[pzuraq]

@lifeart that would be very inefficient, and a leaky abstraction since there is no way to know when a value has been initialized. At that point it would be better to wait for and use native Proxy.

@bendemboski I understand the concerns you've raised, and they're totally valid. Unfortunately, we cannot make a fully transparent array class at the moment without native Proxy, and even when it becomes possible it may not be performant enough for some use cases (my own naive micro-benchmarks show that native arrays are still 100-times faster than a naive Proxy wrapper).

Personally, I feel like EmberArray is the worst of both worlds that you describe. Some methods like pushObject and popObject do signal that the object is in fact an EmberArray, but plenty methods do not - map for instance is probably one of the most common methods I use in an array, and I could use an EmberArray in many places only using map or forEach, and thus no longer have a reliable hint. On top of that, you are forced to learn a new, confusing API if you want to use EmberArray. I think I'd prefer an API where all methods are hinted, or none are, so it is at least consistent.

We also can use native proxy in development, so for instance if you tried to do:

let first = trackedList[0];

We can throw an error to the user and let them know they need to use getAt(0). Maybe this would help with the uncanny valley effect you're describing.

[lifeart]

@pzuraq why no way? there is limited set of methods to add value to array

[pzuraq]

let arr = [];

arr[0] = 123;

Is valid, and not possible to instrument without Proxy. It would mean that you cannot transparently pass the array around, especially in non Ember code, because some other method would have to be used to setup the (extremely expensive) native getter and setter.

[lifeart]

do we really have such performance problem? (in real life apps)?
there is cases we have > 10k arrays, but I don't think it's > 20% of all app arrays (in user space)

We trying to create conceptually strong solution, instead "usable". But it don't need for 80% of app developers.

Maybe it's make sense to create "slow" array structures (convenient with classic arrays and well-known for all js developers), and "fast" one for specific usage (bottlenecks) with different APIs?

[lifeart]

let arr = [];

arr[0] = 123;

Is valid, and not possible to instrument without Proxy. It would mean that you cannot transparently pass the array around, especially in non Ember code, because some other method would have to be used to setup the (extremely expensive) native getter and setter.

we can use Babel and replace [] to new MyArray([]) and we will be able instrument
a[0] = 1 without problems

// svelte transforms

[pzuraq]

I’m personally more worried about the leaky abstraction than the performance problem. IMO, any solution that attempts to use native get/set syntax must be able to do so perfectly, so we must wait for native Proxy.

Users who don’t need to support IE11 today can of course use native proxies to create an array wrapper as is. For an official Ember solution, I think the options are to continue using EmberArray until we drop support for IE11, or to pursue something like TrackedList.

[pzuraq]

we can use Babel and replace [] to new MyArray([]) and we will be able instrument
a[0] = 1 without problems

Let me rephrase:

let arr = new MyArray([]);

arr[0] = 123;

Is also impossible to instrument. It is impossible to do this for any and all JS objects without native Proxy.

[lifeart]

@pzuraq possible #569 (comment)

In software development there is no perfect solution, it's always balance between simplicity, performance, maintainability, extensibility.

I'm afraid TrackList may become one more "Ember Array" for next years, instead of not ideal, but widely usable and well-known property patching (vue js)

[pzuraq]

@lifeart that abstraction is still leaky, which is my overall point. There may not be perfect solutions, but there are correct solutions, and native Proxy is the only one that is correct in the way you want it to be. I don't believe we can ship anything less correct.

TrackedList may indeed end up being a temporary solution. I think the performance angle is still worth considering, though I tend to agree that unless performance is absolutely critical, the benefits of a drop in wrapper are probably worth it. But the fact is, we cannot have TrackedArray today.

This has been a continual problem in our design process, that we let the perfect be the enemy of the good. The point of this proposal is to make things incrementally better. TrackedList would simplify the current design and give us a path off native prototype extensions, custom array methods, etc. It can be implemented quickly, and since its API lines up with native arrays very closely, it would make a transition path toward a hypothetical TrackedArray incrementally easier than directly from EmberArray.

Personally, I want a slightly better Ember today much more than a much better Ember a year from now. We may just have to agree to disagree on that.

[lifeart]

Is it possible to ignore mutable arrays at all?
-> Offer immutable arrays?
(don't fight with a[b]=c issues, and mention it on documentation level)

// react users living with it pretty well

all we need - offer devs use ramda or kinda library to deal with immutable arrays, instead of supporting N array apis internally.

[pzuraq]

The immutable pattern is fully supported, and we definitely should mention it in the guides and have an In Depth guide on its usage.

[hjdivad]

The RFC should probably mention, if only as an aside, what the long term preference is, if not TrackedList.

[hjdivad]

Presumably it's Proxy, but it's not totally obvious since later on when Proxy is discussed a couple of important caveats are emphasized.

[Gaurav0]

IMHO, if this is not the long term solution, it should not be added to core and should remain in an addon.

[marcoow]

I understand that this can be helpful for people who prefer the pre-Octane way of dealing with mutable/tracked (Ember) arrays but isn't the fact that @tracked removes the need for Ember-specific data structures and the cognitive overhead that comes with them actually an advantage? This would add something that's very similar to Ember.Array effectively with similar cognitive overhead again.

So while I'm not completely opposed to this, the above is just the first thought I had when seeing this. Would it be an option to release this in an addon if we want it at all and not make it part of standard Ember? The same comment applies to #577 of course.

[pzuraq]

@marcoow yeah, I definitely understand that and agree that in general we want to make the programming model really feel like it's "Just JavaScript" still, and any extra layering on top of vanilla JS should be carefully considered and thought through, since it'll add cognitive overhead.

For context, the guides and official recommendations in Octane still all use EmberArray, and I don't believe there is consensus to change that in the near future. The issue is that the "immutable model" that was originally put forward in the Tracking RFCs, where you would reassign the array via this.arr = [...this.arr, newValue], had some pitfalls:

• There are performance implications for duplicating arrays and objects every time with ...spread, and it many cases it makes more sense to mutate the array.
• Removing an item from an array using ...spread syntax is really difficult, as is adding an item in the middle of an array.
• Notification style syntax (e.g. arr.push(foo); this.arr = this.arr;) also doesn't feel like vanilla JS, and neither does manual notification (e.g. (invalidate(this, 'arr'))). Both add cognitive overload, and in can be very error prone if someone forgets to notify.
• There isn't really a good, ergonomic way to notify in all the locations an array is used, if it happens to be passed around and assigned/used in multiple locations. Both of the above styles can only provide notifications locally, and to children, but there are many cases where an array might be used in multiple locations (e.g. Ember Data collections).
• Collections like Map and Set don't really have an ergonomic solution here, spread syntax does not work as well. WeakMap and WeakSet cannot be updated with spread syntax, and many of the common use cases for them (e.g. associating values with a module-scope WeakMap) also cannot use the notification style, since they aren't associated with a specific property.

This is not to say that it would be completely off the table to change the guides and recommendations to use this style, it's just that these patterns need to be figured out. Additionally, potential future JS features like Records and Tuples would make this style much nicer and possibly solve many of the concerns listed above. Libraries like Immer exist today, but I feel like we would have to be very confident in them to recommend them by default.

I suppose from a personal perspective, I really started to feel like this was the right solution as I began to work more with autotracking and think through its principles, from a theory perspective. The fundamental assumption that autotracking makes is that all root state that is input to the system is tracked. Currently, we only expose ways to tracked specific properties by default, but mutable collections like arrays generally exist, and we want to track changes to them as well. The current solution feels like it's breaking that principle by asking you to notify on something unrelated when you mutate an array, map, or set.

FWIW, I'm really excited about Records and Tuples though and could really see them be used this way. Since they're immutable, they don't introduce any of these problems, which is perfect! I could also definitely see a guide for using Immer in Ember apps.

[mehulkar]

As exciting and cool as this is, I think the focus of 2020 should be to make the surface area of Ember smaller, not bigger. In real world scenarios, the tradeoff at PR review time is to either nudge the author of code towards understanding the reactivity model and its caveats or using TrackedList. I would much prefer the first, as it's much easier to grasp why Ember can't react to internal changes of an object, than to understand why we can't use Array as a rule (or worse, having to decide when to use an Array or TrackedList).

I would like to see this shipped as a library completely agnostic of Ember though (not an Ember addon) and let users opt in.

[marcoow]

@pzuraq thanks for the in-depth reply. I understand there are pitfalls with the immutable model and agree there needs to be a way to get around those. I think I'm just worried (similar to what @mehulkar is saying) that as we're now in a situation where @tracked allows us to get rid of many of Ember's specialties (like Ember.Array) we should not immediately go back to a similar situation as before by adding new versions of these specialties when encountering problems with the new approach (of course there are substantial differences between Ember.Array and TrackedList but essentially the latter is very similar to the former in the sense that it is Ember's special data type for arrays/lists that one has to learn in addition to learning Ember, one has to be aware when to use which etc.). I do think TrackedList makes sense in general and it would be good if it was available in some form for people that run into the above pitfalls but it seems like this could perfectly well be in an addon for now which would still not mean it could never be added to the framework itself later on or the addon could be added to the default blueprint etc.

[pzuraq]

@mehulkar @marcoow I definitely understand these concerns. I think the current state of things is though that we haven't gotten rid of those extra wrappers. Specifically, we currently cannot deprecate or remove EmberArray, since it is the default and recommended approach for dealing with lists in the framework.

We can come up with alternative solutions in the community, but until we can get consensus on an alternative default for arrays/lists, I believe EmberArray and its quirks will have to stay. Maybe it would be worth waiting for Proxy at least then, since it would mean we could potentially design a drop-in array replacement, but I personally would like to be in a position where we can deprecate EmberArray sooner rather than later.

[256dpi]

I personally like that auto-tracking is just limited to classes and the root level properties. In the past I had situations where a EmberArray was leaked to some other place and caused unintended re-computations as it was changed. Since the methods are different to native arrays the problem was easy to spot and fix. However, I don't want to think about what happens if a proxied native array gets passed into some external library and cause re-renders all over the place. Even if it would be possible, I wouldn't want arrays, POJOs etc. to magically cause re-renders if they are mutated. For that reason, I would not use and recommend tracking of such basic primitive types.

But, I agree that the auto-tracking system is not very ergonomically when it comes to arrays, POJOs and other non-trackable types. The current example in the guides shows that first hand:

class SimpleCache {
  @tracked _cache = {};

  set(key, value) {
    this._cache[key] = value;

    // trigger an update
    this._cache = this._cache;
  }

  get(key) {
    return this._cache[key];
  }
}

Yet, I belief there are other solutions that could be explored here. A quick idea of the back of my head would be to provide a magic tracked property that provides access to all other properties but automatically schedules a re-computation in the next cycle when accessed:

class SimpleCache {
  @tracked _cache = {};

  set(key, value) {
    this.tracked._cache[key] = value;
  }

  get(key) {
    return this._cache[key];
  }
}

Maybe with something like that there is no need for TrackedX. And if there is still one, it can be quickly implemented by hand and adjusted for the specific use case.

[sandstrom]

Great proposal overall!

I think designing something proxy-based would be much better and browser support is excellent. Let's come up with something modern, with longer shelf-life.

IE 11 is in rapid decline and only have a few percentage points of usage these days.

Many large software companies has already stopped supporting it, for example:

• Atlassian has dropped support for IE 11

• Zendesk has dropped IE 11 support

• Qlik View let go of IE 11 too

• We're not a huge company, but we dropped support for IE11 and none of our ~1000 customers complained. When we checked usage logs, for the few companies that still had IE11 usage 100% of them had other browsers installed on their machines already, and somewhere around 50-90% of their employees already used another browser.

The days where IE11 was the only browser installed on a computer are long gone.

Ember should drop IE 11 support and focus time on using the modern JS toolset to provide a nice framework. Any Ember user that need to extend support a while longer, e.g. end of 2020, could simply linger on an Ember LTS a while longer. It doesn't need to hold back new framework development.

Regarding the performance drawback with proxies, I don't think it would be a big problem under most real-world scenarios.

[luxzeitlos]

There isn't really a good, ergonomic way to notify in all the locations an array is used, if it happens to be passed around and assigned/used in multiple locations. Both of the above styles can only provide notifications locally, and to children, but there are many cases where an array might be used in multiple locations (e.g. Ember Data collections).

Could we not Array and add a notify method? Then I do arr[2] = foo; arr.notify().

I dont like the current way no notify a change this.foo = this.foo but I think we wont find a way around manual notification. And actually this is not bad. But whenever we get data from an external non ember-aware library we need to do manual notification. Not matter if its a nested object or an array. So learning this is important, and then better use the same pattern then to introduce another custom Array-Like structure.

[pzuraq]

@luxferresum there are two main issues I see with that potential solution:

1. Modifying the built-in prototypes is generally considered a very bad idea. Not only can it cause future compatibility issues (e.g. if Array adds its own notify method in the future), it also deoptimizes the browser's built in version of the array, which is pre-JIT'd and ready to run, forcing it to recompile and re-optimize all of the array's methods. If we didn't add the method directly to Array.prototype, then we'd be in the same boat as before with introducing a new class, and it would probably make sense to do more than adding notify.

2. Entanglement is not just about notification - we must also consume the array, which can only be done when the array is used. Consider this example, which models the game Lights Out:

   class LightsOut {
     lights = [
       [0, 0, 1, 0, 0],
       [0, 1, 0, 1, 0],
       [1, 0, 0, 0, 1],
       [0, 1, 0, 1, 0],
       [0, 0, 1, 0, 0],
     ];
   
     isLightOn(x, y) {
       return this.lights[y][x] === 1;
     }
   }

   In the autotracking model, whenever an autoracking frame wraps a calculation, all state that can affect the outcome of that calculation should entangle itself. In this case, the calculation is whenever we call isLightOn with a particular x and y coordinate. With notify() we would have no way to tell the consuming API that the arrays have been used when we call isLightOn. Normally, accessing the property that is @tracked does this, but in this case it won't work since we're working with a 2d array. The best solution here would be to notify all of the lights when any one of them has changed, by notifying the @tracked property itself.

But whenever we get data from an external non ember-aware library we need to do manual notification.

This is actually not how we expect most users to have to work with external libraries. Autotracking is a very flexible reactivity system, and is designed to allow libraries to be wrapped transparently. In general, manually entangling and instrumenting notifications from external libraries is something that should be done once in a wrapper library, and then within Ember apps it should be consumable without having to worry about the details of how the library autotracks. The assumption should be that if the library mutates state, it also correctly entangles and notifies that state.

For examples of what this looks like in action, we can look at Ember Data and tracked-built-ins. In the case of Ember Data, it provides the @attr, @belongsTo, and @hasMany decorators, which all autotrack seamlessly. These are currently computed properties, but in the future they will likely be stripped down to the bare minimum and become thin wrappers around lower level autotracking entanglement and notification APIs. In the case of tracked-built-ins, the library instruments a very basic external (at the moment) data source, and notifies on any changes to it.

This same strategy could be applied to, say, Apollo, Redux, or any number of other external libraries. We are working on stabilizing autotracking APIs so that instrumenting libraries is easier and the community can begin experimenting more easily, but I definitely think something we want to avoid is manually notifying for every single external library, every time it is used in an app. How to do it should be well documented and understood though, it's not that these wrapper layers would provide "magic", they would just would place the entanglement where it logically belongs - with the source of the data.

[wagenet]

How does this compare to https://github.com/tracked-tools/tracked-built-ins?

[chriskrycho]

I believe this is superseded by the combination of #669 and #812. #669 gave us the tools needed to implementing things like this in user-land, and #812 adds the most common built-in versions that users will need, implemented using those primitives.

[sandstrom]

@wagenet Awesome with all the RFC cleanup lately! 🎉