3.0.0.md

description
Released April 9, 2021

3.0.0

{% hint style="danger" %} Plugins compatible with Live 2.x.x are NOT compatible with Live 3.x.x {% endhint %}

New features

• Improve sensitive information security
  • All default plugins were migrated to this new API
  • [API] Unified encryption service to secure sensitive data (passwords, connection secrets, etc)
• Improved chart analysis experience, tooltip improvements
  • Chart crosshair now follows smoothly the x-axis values instead of snapping to point values
  • Removed "Find near points" option
  • Tooltip follows legend ordering
• Option to display gaps on the Cartesian widget with pipes modifiers
• [API] Point Service new observable properties
• [API] Support to partial index creation
• [API] Settings' Observers
• [API] Lookup tables can have dedicated permissions checks

Fixes and improvements

• Improve resilience against malfunctioning storage providers
• [API] Kotlin was added to the list of default dependencies provided by Live
• UI/UX improvements to the admin page
• Performance improvement in the Messenger
• Prevents invalid indexes while creating multiple indexes concurrently for the same event type in the PostgreSQL storage provider
• Improved Download of CSV, Excel, and JSON formats
• [API] Settings' API for transactions was removed

Breaking changes and migration paths

• PointService does not emit the Chart object on legacy API anymore
  • Use the new API instead
• Highcharts update from 7.0.3 to 9.0.1
  • Check if your chart still works, use the Highchart's Release notes as a reference if something breaks.
• API changes in the Settings, related to the aforementioned Settings' Observers
• Plugins that have invalid web services will be preemptively stopped by Live
• Live will block the creation of queries inside queries due to a deadlock possibility

Details

Settings

Live 3.0.0 brings a lot of changes to the Settings mechanism.

For simplicity, these changes will be presented in three categories:

• New functionalities
• APIs removed
• Performance improvements

For those that are not familiar with the Settings API, feel free to read more about it in the documentation.

New Functionalities

We're proud to announce that starting on Live 3.0.0 the Settings offers an option to register an observer to a path. With this new API, it's possible to receive SET and DELETE events for a given path. It's also possible to register an observer to a path and all of its children.

Live.Settings settings = live.settings();
SettingsObserver observer = new SettingsObserver() {
    @Override
    public void onSet(@NotNull String path, @Nullable String content) {
        //Do something :)
    }

    @Override
    public void onDelete(@NotNull String path) {
        //Do another thing :)
    }
};
Live.Action action = settings.registerObserver(live, "/myPath/", observer, true);

In this snippet, the observer is being registered for /myPath/ and all of its children (last parameter).

Therefore, if any other components in the system changed the value for the path /myPath/other/ this observer will be called.

Implementation note: onSet and onDelete runs asynchronously between observers. i.e one observer can receive and process multiple events while the other is still processing the first event.

However, it's guaranteed that all observers will receive the events in the same order.

{% hint style="info" %} This change has introduced an API change in the SettingsRoot, which will be presented below. {% endhint %}

Another change introduced in Live 3.0.0 was the new, simplified, withOptions API.

Until this point, the only API available was this:

node.withOptions(settingsOptions, ()->{
            node.set(value);
            return null;
        });

This option is still available and recommended to multiple editions. For single editions we now recommend the new API:

node.set(value, settingsOptions);

APIs removed

SettingsNode

inTransaction

SettingsProvider

transact

checkPaths

Performance improvements

• New lock mechanism
• Add information about the benchmark
• Disable log by default

The Live Settings API delivers an optional feature to keep the track on node editions. Before Live 3.0.0, the log of SettingsNode editions were enabled by default. In order to save space in the database, Live 3.0.0 disables (by default) the log for SettingsNode editions.

// The plugin which needs the log feature for SettingsNode can re-enable 
// it manually per edition as following:

node.set(someValue, new SettingsOptions().enableLog(true));

// Multiple editions are also supported using `withOptions` method:

node.withOptions(new SettingsOptions().enableLog(true), () -> {
    node.set(someValue);
    node.cd("child-54").set(anotherValue);
    node.cd("child-73").delete();
    return null;
});

[API] Support to partial index creation

The Live API was modified to include a withPartial builder method on the entity which describes the index details (i.e.IndexDef) by providing a compiled version of a Pipes filter:

IndexDef partial = new IndexDef("mycol", "field").withPartial(filter);

Then plugins can register the partial index using the traditional registerIndexmethod.

live.index().registerIndex(partial);

Besides the feature is available in Live API, each storage provider may support partial indexes to be effective. At version 3.0.0, only the MongoDB storage provider supports partial indexes. PostgreSQL storage providers will support them soon in future versions.

See more details about the partial index feature.

Improve resilience against malfunctioning storage providers

By default, Live supports storing high-frequency data in multiple storage providers. Then, any pipes query will read all data stored in all active storage providers for the same event type.

However, in case of a storage provider fails to read data, Live aborts the query and no other data will be delivered. This causes a lack of data due to some providers were never queried yet.

Starting in 3.0.0 Live will return as much data as possible, instead of failing the whole query. In case of any storage failure, Live will generate warning control events with the error messages.

Prevents invalid indexes while creating multiple indexes concurrently for the same event type in the PostgreSQL storage provider

Under some situations, mainly at Live startup, the PostgreSQL storage provider tries to ensure the indexes and it's allowed to perform the index creation concurrently.

However, when multiple indexes are created for the same event type concurrently, it can result in PostgreSQL exceptions like:

org.postgresql.util.PSQLException: ERROR: deadlock detected Detail:
Process 15109 waits for ShareLock on virtual transaction 175873; blocked by process 15116.
Process 15116 waits for ShareUpdateExclusiveLock on relation 62186 of database 17468; blocked by process 15109.

Starting in 3.0.0 Live will rely on a lock (managed per event type in Java) to prevent multiple indexes be created at the same time for the same event type. Thus, index creation will occur concurrently in fact only for different event types (each event type represents a table in the PostgreSQL provider).

Improve sensitive information security

Live introduces a password encryption API that provides privacy and protection of extension-related sensitive data.

To achieve data privacy and protection, we have introduced a password‐based encryption approach that is based on symmetric cryptography and provides the security of extension-related sensitive data.

To start using the new password encryption feature, include the security.passwordEncryptionPassword property in the live.properties file, and if any value exists for this property, delete it and then enter a new value. This value will be used as the master key of all encrypted data so if it ever changes, all sensitive data related to extensions will be lost and need to be entered again.

{% hint style="info" %} If the property security.passwordEncryptionPassword is empty the interface will not change and no data will be encrypted. {% endhint %}

{% hint style="success" %} Existing extensions from compatible plugins will have their sensitive data migrated to encrypted format once this feature is enabled. {% endhint %}

Whenever Live wants to authenticate to the proposed external system, it does as follows:

1. Retrieve the data from the database in an encrypted format
2. Decrypt the data using the Live API.
3. Uses the decrypted data to authenticate with the third-party service.

{% hint style="info" %} The same password used in different plugins results in different encrypted data stored in the database. {% endhint %}

For integrations instances that need only one password, such as MongoDB, the encryption field shows an accept option:

After accepting the password will be hidden. To unlock and define a new password use the lock option

In case of an empty password, additional confirmation is required.

For the tables that have multiple users will be shown a new interface to handle encrypted data. It is required to accept each password in each row:

Improved Download of CSV, Excel, and JSON formats

Several bug fixes and improvements when exporting a widget, dashboard or query result as csv, excel or json:\

• CSV and Excel formats are now able to combine multiple events in one row based on the event index field value.
• CSV and Excel formats are now able to display all fields as columns, no matter how often or how late they appear on the query result.
• Several improvements in the filename of the generated archive. Some of which include the addition of the current span to the filename.
• When downloading multiple queries, the result is now aggregated in just one file. If some of the queries share some fields in common, they will be prefixed to properly distinguish which field belongs to which query.