Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial version of interactive dataset explorer. Closes #1692 #1694

Merged
merged 68 commits into from
Jul 27, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
8ad9142
WIP Add initial commit of DatasetExplorer
brollb May 5, 2020
ae337e6
WIP use artifact name in DataExplorer
brollb May 5, 2020
a164ce2
WIP Add plot editor to dataset explorer
brollb May 7, 2020
277608b
WIP add PlotEditor and css
brollb May 7, 2020
7fe251d
Add UI elements for adding/removing data
brollb May 8, 2020
00e7694
Fix UI issue with editing plotted data
brollb May 8, 2020
9792e13
WIP Add utility for parsing python slices
brollb May 11, 2020
7ef7b6d
Add slice string validation
brollb May 11, 2020
6b39f9b
UI support for validating data slice
brollb May 11, 2020
5a2bb74
Update figure updates given the figure data
brollb May 11, 2020
6be4a6f
Add explorer_helpers.py for DatasetExplorer
brollb May 11, 2020
1c5eb58
Add explorer helpers to session for getting metadata
brollb May 11, 2020
1d1e42a
dynamically create variable names for dropdown
brollb May 11, 2020
6426b71
Add tests for variable name creation from metadata
brollb May 11, 2020
a220061
Plot actual data selected from "Add Data" button
brollb May 11, 2020
831c158
Fix plot height
brollb May 12, 2020
cedad0f
Add basic 3D plot support to DataExplorer
brollb May 12, 2020
e12232f
Fix async issue with multiple lines
brollb May 12, 2020
5bdf1da
Fix x-axis, y-axis labels
brollb May 12, 2020
954212f
Add validation for plotting data
brollb May 12, 2020
96bb01f
Remove .only from PythonSlice test suite
brollb May 12, 2020
61a7e27
Add basic artifact loader to DatasetExplorer
brollb May 12, 2020
5a0ae54
Only show artifacts with data in artifact loader
brollb May 12, 2020
af549f5
Update metadata on artifact load into session
brollb May 12, 2020
15546b9
Removed hardcoded examples from html
brollb May 12, 2020
d434d01
Add some support for colors (uniform only)
brollb May 12, 2020
5bc0cf3
Add color support for individual points
brollb May 12, 2020
6a31f58
Fix selection of keys from artifacts
brollb May 13, 2020
77cfd2f
WIP Use session with queue in dataset explorer
brollb May 13, 2020
c968fc9
WIP code cleanup dataset explorer
brollb May 13, 2020
73a18b9
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb May 14, 2020
d1ad052
Add compute creation (and shield) for DatasetExplorer
brollb May 14, 2020
5f45c75
Don't show slice syntax errors until change event
brollb May 14, 2020
c870763
Fix artifacts with extensions. minor code cleanup
brollb Jun 16, 2020
d1cc2e0
Increase territory for access to initialization code
brollb Jun 16, 2020
6f33b3e
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jun 20, 2020
7065d4f
Rename DatasetVisualizer -> TensorPlotter
brollb Jun 22, 2020
4b7c1c1
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 1, 2020
365677c
Rename scss,css files
brollb Jul 2, 2020
d749c42
Add "save" action to floating action button
brollb Jul 2, 2020
38cfe75
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 2, 2020
84bc27e
Only load jscolor in the browser
brollb Jul 2, 2020
44cf5da
Skip jscolor library when linting
brollb Jul 2, 2020
226d627
Add InteractiveEditor base class
brollb Jul 3, 2020
cb741a3
Update to use InteractiveEditor base classes
brollb Jul 3, 2020
8abafef
Add getSnapshot to tensor plotter
brollb Jul 6, 2020
1fd7044
Fix setting the data dialog on open
brollb Jul 7, 2020
67ea9ee
WIP working on operation code...
brollb Jul 8, 2020
d554a91
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 9, 2020
827d49b
Include all artifacts in TensorPlotter
brollb Jul 9, 2020
650697d
Fixed python slice parsing
brollb Jul 9, 2020
196eb67
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 9, 2020
cf66703
Add InteractiveExplorer base class
brollb Jul 9, 2020
b67e85d
Use inform dialog w/ auth errors
brollb Jul 9, 2020
1fc5bb7
Update TensorPlotter to inherit from InteractiveExplorer
brollb Jul 15, 2020
806578e
fix css linting issue
brollb Jul 15, 2020
53f49a2
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 15, 2020
32c7458
unregister action on destroy
brollb Jul 15, 2020
5fbf256
Remove OperationTemplate
brollb Jul 16, 2020
7105553
Clean up python commands and add dialog on error
brollb Jul 16, 2020
9ed18f4
Ensure that name,title are set on save
brollb Jul 16, 2020
bc6b068
refactor executing python code
brollb Jul 17, 2020
1a730a2
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 17, 2020
a9debb2
Merge branch 'master' into 1692-interactive-dataset-exploration
brollb Jul 27, 2020
67d628d
Improve error dialog
brollb Jul 27, 2020
d58de92
Misc code cleanup
brollb Jul 27, 2020
f25e49f
Skip simple grid lib in linting
brollb Jul 27, 2020
53e0efb
Fix quotes in tests
brollb Jul 27, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .codeclimate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,5 @@ exclude_paths:
- src/visualizers/widgets/PipelineEditor/klay.js
- src/visualizers/widgets/PlotlyGraph/lib/plotly.min.js
- src/plugins/GenerateJob/templates/utils.build.js
- src/visualizers/widgets/TensorPlotter/lib/
- src/visualizers/widgets/TensorPlotter/styles/simple-grid.min.css
14 changes: 13 additions & 1 deletion src/visualizers/Visualizers.json
Original file line number Diff line number Diff line change
Expand Up @@ -148,5 +148,17 @@
"title": "OperationDepEditor",
"panel": "panels/OperationDepEditor/OperationDepEditorPanel",
"DEBUG_ONLY": false
},
{
"id": "TensorPlotter",
"title": "TensorPlotter",
"panel": "panels/TensorPlotter/TensorPlotterPanel",
"DEBUG_ONLY": false
},
{
"id": "InteractiveExplorer",
"title": "InteractiveExplorer",
"panel": "panels/InteractiveExplorer/InteractiveExplorerPanel",
"DEBUG_ONLY": false
}
]
]
42 changes: 42 additions & 0 deletions src/visualizers/panels/TensorPlotter/TensorPlotterControl.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
/*globals define */
/**
* Generated by VisualizerGenerator 1.7.0 from webgme on Mon May 04 2020 17:09:31 GMT-0500 (Central Daylight Time).
*/

define([
'panels/InteractiveExplorer/InteractiveExplorerControl',
], function (
InteractiveExplorerControl,
) {

'use strict';

class TensorPlotterControl extends InteractiveExplorerControl {

getObjectDescriptor(nodeId) {
const desc = super.getObjectDescriptor(nodeId);

if (desc) {
const node = this.client.getNode(nodeId);
desc.data = node.getAttribute('data');
desc.type = node.getAttribute('type');
}

return desc;
}

getTerritory(nodeId) {
const territory = {};
const node = this.client.getNode(nodeId);
const parentId = node.getParentId();
territory[parentId] = {children: 1};

const omitParentNode = event => event.eid !== parentId;
this.territoryEventFilters = [omitParentNode];

return territory;
}
}

return TensorPlotterControl;
});
101 changes: 101 additions & 0 deletions src/visualizers/panels/TensorPlotter/TensorPlotterPanel.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
/*globals define, _, WebGMEGlobal*/
/**
* Generated by VisualizerGenerator 1.7.0 from webgme on Mon May 04 2020 17:09:31 GMT-0500 (Central Daylight Time).
*/

define([
'js/PanelBase/PanelBaseWithHeader',
'js/PanelManager/IActivePanel',
'widgets/TensorPlotter/TensorPlotterWidget',
'./TensorPlotterControl'
], function (
PanelBaseWithHeader,
IActivePanel,
TensorPlotterWidget,
TensorPlotterControl
) {
'use strict';

function TensorPlotterPanel(layoutManager, params) {
var options = {};
//set properties from options
options[PanelBaseWithHeader.OPTIONS.LOGGER_INSTANCE_NAME] = 'TensorPlotterPanel';
options[PanelBaseWithHeader.OPTIONS.FLOATING_TITLE] = true;

//call parent's constructor
PanelBaseWithHeader.apply(this, [options, layoutManager]);

this._client = params.client;
this._embedded = params.embedded;

//initialize UI
this._initialize();

this.logger.debug('ctor finished');
}

//inherit from PanelBaseWithHeader
_.extend(TensorPlotterPanel.prototype, PanelBaseWithHeader.prototype);
_.extend(TensorPlotterPanel.prototype, IActivePanel.prototype);

TensorPlotterPanel.prototype._initialize = function () {
var self = this;

//set Widget title
this.setTitle('');

this.widget = new TensorPlotterWidget(this.logger, this.$el);

this.widget.setTitle = function (title) {
self.setTitle(title);
};

this.control = new TensorPlotterControl({
logger: this.logger,
client: this._client,
embedded: this._embedded,
widget: this.widget
});

this.onActivate();
};

/* OVERRIDE FROM WIDGET-WITH-HEADER */
/* METHOD CALLED WHEN THE WIDGET'S READ-ONLY PROPERTY CHANGES */
TensorPlotterPanel.prototype.onReadOnlyChanged = function (isReadOnly) {
//apply parent's onReadOnlyChanged
PanelBaseWithHeader.prototype.onReadOnlyChanged.call(this, isReadOnly);

};

TensorPlotterPanel.prototype.onResize = function (width, height) {
this.logger.debug('onResize --> width: ' + width + ', height: ' + height);
this.widget.onWidgetContainerResize(width, height);
};

/* * * * * * * * Visualizer life cycle callbacks * * * * * * * */
TensorPlotterPanel.prototype.destroy = function () {
this.control.destroy();
this.widget.destroy();

PanelBaseWithHeader.prototype.destroy.call(this);
WebGMEGlobal.KeyboardManager.setListener(undefined);
WebGMEGlobal.Toolbar.refresh();
};

TensorPlotterPanel.prototype.onActivate = function () {
this.widget.onActivate();
this.control.onActivate();
WebGMEGlobal.KeyboardManager.setListener(this.widget);
WebGMEGlobal.Toolbar.refresh();
};

TensorPlotterPanel.prototype.onDeactivate = function () {
this.widget.onDeactivate();
this.control.onDeactivate();
WebGMEGlobal.KeyboardManager.setListener(undefined);
WebGMEGlobal.Toolbar.refresh();
};

return TensorPlotterPanel;
});
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,9 @@ define([
DeepForge.unregisterAction('Save');
}
}

updateNode(/*desc*/) {
}
}

return InteractiveEditorWidget;
Expand Down
6 changes: 6 additions & 0 deletions src/visualizers/widgets/TensorPlotter/ArtifactLoader.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
<h4>Available Artifacts</h4>
<ul class="list-group artifacts">
<li class="list-group-item">Artifact
<span class="glyphicon glyphicon-remove pull-right" aria-hidden="true"></span>
</li>
</ul>
137 changes: 137 additions & 0 deletions src/visualizers/widgets/TensorPlotter/ArtifactLoader.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
/*globals define, $*/
define([
'deepforge/EventEmitter',
'deepforge/storage/index',
'underscore',
'text!./ArtifactLoader.html',
'css!./styles/ArtifactLoader.css',
], function(
EventEmitter,
Storage,
_,
Html,
) {
class ArtifactLoader extends EventEmitter {
constructor(container) {
super();
this.session = null;
this.$el = container;
this.$el.addClass('artifact-loader');
this.$el.append($(Html));
this.$artifacts = this.$el.find('.artifacts');
this.artifacts = [];
this.render = _.debounce(this.render.bind(this), 250);
}

register(desc) {
this.artifacts.push(new Artifact(desc));
this.render();
}

unregister(artifactId) {
const index = this.artifacts.findIndex(artifact => artifact.id === artifactId);
if (index > -1) {
this.artifacts.splice(index, 1);
this.render();
}
}

async load(artifact) {
const desc = artifact.data;
const dataInfo = JSON.parse(desc.data);
const config = await this.getAuthenticationConfig(dataInfo);

const pyName = desc.name.replace(/\..*$/, '');
const loading = this.session.addArtifact(pyName, dataInfo, desc.type, config);
artifact.state = ArtifactState.LOADING;
this.render();
await loading;
artifact.state = ArtifactState.LOADED;
this.render();
this.emit('load', desc);
}

async getAuthenticationConfig (dataInfo) {
const {backend} = dataInfo;
const metadata = Storage.getStorageMetadata(backend);
metadata.configStructure = metadata.configStructure
.filter(option => option.isAuth);

if (metadata.configStructure.length) {
const configDialog = this.getConfigDialog();
const title = `Authenticate with ${metadata.name}`;
const iconClass = `glyphicon glyphicon-download-alt`;
const config = await configDialog.show(metadata, {title, iconClass});

return config[backend];
}
}

render() {
this.$artifacts.empty();
this.artifacts.forEach(artifact => {
const $element = artifact.element();
if (artifact.state === ArtifactState.NOT_LOADED) {
$element.on('click', event => {
event.stopPropagation();
event.preventDefault();
this.load(artifact);
});
}
this.$artifacts.append($element);
});
}
}

class ArtifactState {
constructor(clazz, text, icon) {
this.class = clazz || '';
this.text = text || '';
this.icon = icon || '';
}

configure($element) {
if (this.text) {
const $state = $('<span>', {class: 'pull-right artifact-state'});
$state.text(this.text);
$element.append($state);
}
if (this.class) {
$element.addClass(this.class);
}
if (this.icon) {
const $icon = glyph(this.icon);
$element.append($icon);
}
}
}

ArtifactState.LOADING = new ArtifactState('list-group-item-warning', 'Loading...');
ArtifactState.NOT_LOADED = new ArtifactState(null, null, 'upload');
ArtifactState.LOADED = new ArtifactState('list-group-item-success', 'Available');

class Artifact {
constructor(data) {
this.id = data.id;
this.data = data;
this.state = ArtifactState.NOT_LOADED;
}

element() {
const $element = $('<li>', {class: 'list-group-item'});
$element.text(this.data.name);

this.state.configure($element);

return $element;
}
}

function glyph(name) {
const $el = $('<span>', {class: `glyphicon glyphicon-${name} pull-right`});
$el.attr('aria-hidden', true);
return $el;
}

return ArtifactLoader;
});
51 changes: 51 additions & 0 deletions src/visualizers/widgets/TensorPlotter/DataEditorBase.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/* globals define, $ */
define([
'underscore',
'deepforge/EventEmitter',
], function(
_,
EventEmitter,
) {

class DataEditorBase extends EventEmitter {
constructor(html, dataFields, updateOnChange) {
super();
this.$el = $(html);

this.$elements = {};
dataFields.forEach(name => {
this.$elements[name] = this.$el.find(`#${name}`);
if (updateOnChange) {
this.$elements[name].change(() => this.onUpdate());
}
});
}

set(values) {
Object.entries(this.$elements).map(entry => {
const [name, element] = entry;
if (values.hasOwnProperty(name)) {
element.val(values[name]);
}
});
}

data() {
const entries = Object.entries(this.$elements)
.map(entry => {
const [name, element] = entry;
const value = element.val();
return [name, value];
})
.filter(entry => !!entry[1]);
return _.object(entries);
}

onUpdate() {
const values = this.data();
this.emit('update', values);
}
}

return DataEditorBase;
});
18 changes: 18 additions & 0 deletions src/visualizers/widgets/TensorPlotter/PlotEditor.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
<form>
<div class="form-group">
<label for="title">Title</label>
<input type="text", class="form-control" id="title"/>
</div>
<div class="form-group">
<label for="xaxis">X Axis Label</label>
<input type="text", class="form-control" id="xaxis"/>
</div>
<div class="form-group">
<label for="yaxis">Y Axis Label</label>
<input type="text", class="form-control" id="yaxis"/>
</div>
<h4>Plotted Data</h4>
<ul class="list-group plotted-data">
</ul>
<button class="btn btn-primary">Add Data</button>
</form>
Loading