Skip to content

Commit

Permalink
1.8.1 bugfixes & enhancements:
Browse files Browse the repository at this point in the history
* #84, highlight columns based on dtype
* #92, build columns with random values
* #111, code export has syntax error & str() fix for column builder names
* #116, updated styling of github fork link
* #114, added "Export CSV" link
* #113, updates to "Value Counts" chart in "Column Analysis" for number of values and ordinal entry
* #120, allowing for duplicates in bar charts
* #119, fixed bug with queries not being passed to functions
* #114, added the ability to export dataframes to CSV/TSV
* added "category breakdown" in column analysis popup for float columns
* fixed bug where previous "show missing only" selection was not being recognized
Andrew Schonfeld committed Mar 28, 2020
1 parent 50baa40 commit 0e073eb
Showing 47 changed files with 2,058 additions and 573 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ defaults: &defaults
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
CODECOV_TOKEN: b0d35139-0a75-427a-907b-2c78a762f8f0
VERSION: 1.8.0
VERSION: 1.8.1
PANDOC_RELEASES_URL: https://github.com/jgm/pandoc/releases
steps:
- checkout
12 changes: 12 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
## Changelog

### 1.8.1 (2020-3-29)
* [#92](https://github.com/man-group/dtale/issues/92), column builders for random data
* [#84](https://github.com/man-group/dtale/issues/84), highlight columns based on dtype
* [#111](https://github.com/man-group/dtale/issues/111), fix for syntax error in charts code export
* [#113](https://github.com/man-group/dtale/issues/113), updates to "Value Counts" chart in "Column Analysis" for number of values and ordinal entry
* [#114](https://github.com/man-group/dtale/issues/114), export data to CSV/TSV
* [#116](https://github.com/man-group/dtale/issues/116), upodated styling for github fork link so "Code Export" is partially clickable
* [#119](https://github.com/man-group/dtale/issues/119), fixed bug with queries not being passed to functions
* [#120](https://github.com/man-group/dtale/issues/120), fix to allow duplicate x-axis entries in bar charts
* added "category breakdown" in column analysis popup for float columns
* fixed bug where previous "show missing only" selection was not being recognized

### 1.8.0 (2020-3-22)
* [#102](https://github.com/man-group/dtale/issues/102), interactive column filtering for string, date, int, float & bool
* better handling for y-axis management in charts. Now able to toggle between default, single & multi axis
34 changes: 23 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -44,13 +44,14 @@ D-Tale was the product of a SAS to Python conversion. What was originally a per
- [Describe](#describe), [Filter](#filter), [Building Columns](#building-columns), [Reshape](#reshape), [Charts](#charts), [Coverage (Deprecated)](#coverage-deprecated), [Correlations](#correlations), [Heat Map](#heat-map), [Instances](#instances), [Code Exports](#code-exports), [About](#about), [Resize](#resize), [Shutdown](#shutdown)
- [Column Menu Functions](#column-menu-functions)
- [Filtering](#filtering), [Moving Columns](#moving-columns), [Hiding Columns](#hiding-columns), [Lock](#lock), [Unlock](#unlock), [Sorting](#sorting), [Formats](#formats), [Column Analysis](#column-analysis)
- [Menu Functions within a Jupyter Notebook](#menu-functions-within-a-jupyter-notebook)
- [Menu Functions Depending on Browser Dimensions](#menu-functions-depending-on-browser-dimensions)
- [For Developers](#for-developers)
- [Cloning](#cloning)
- [Running Tests](#running-tests)
- [Linting](#linting)
- [Formatting JS](#formatting-js)
- [Docker Development](#docker-development)
- [Global State/Data Storage](#global-state_data-storage)
- [Startup Behavior](#startup-behavior)
- [Documentation](#documentation)
- [Requirements](#requirements)
@@ -368,6 +369,7 @@ This video shows you how to build the following:
- Numeric: adding/subtracting two columns or columns with static values
- Bins: bucketing values using pandas cut & qcut as well as assigning custom labels
- Dates: retrieving date properties (hour, weekday, month...) as well as conversions (month end)
- Random: columns of data type (int, float, string & date) populated with random uniformly distributed values.

#### Reshape

@@ -476,6 +478,11 @@ d.offline_chart(chart_type='bar', x='x', y='z3', agg='sum')
```
[![](http://img.youtube.com/vi/DseSmc3fZvc/0.jpg)](http://www.youtube.com/watch?v=DseSmc3fZvc "Offline Charts Tutorial")

**Pro Tip: If generating offline charts in jupyter notebooks and you run out of memory please add the following to your command-line when starting jupyter**

`--NotebookApp.iopub_data_rate_limit=1.0e10`


**Disclaimer: Long Running Chart Requests**

If you choose to build a chart that requires a lot of computational resources then it will take some time to run. Based on the way Flask & plotly/dash interact this will block you from performing any other request until it completes. There are two courses of action in this situation:
@@ -678,18 +685,23 @@ Here's a grid of all the formats available with -123456.789 as input:
#### Column Analysis
Based on the data type of a column different charts will be shown.

| Data Type | Chart |
|---------------|----------------|
| Integer | Histogram, Value Counts|
| Float | Value Counts |
| Date | Value Counts |
| String | Value Counts |
| Chart | Data Types | Sample |
|---------------|----------------|--------|
| Histogram | Float, Int |![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/analysis/histogram.PNG)|
| Value Counts | Int, String, Bool, Date, Category|![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/analysis/value_counts.PNG)|
| Category | Float |![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/analysis/category.PNG)|


**Histogram** can be displayed in any number of bins (default: 20), simply type a new integer value in the bins input

**Value Count** by default, show the top 100 values ranked by frequency. If you would like to show the least frequent values simply make your number negative (-10 => 10 least frequent value)

*Histograms* can be displayed in any number of bins (default: 20), simply type a new integer value in the bins input
**Value Count w/ Ordinal** you can also apply an ordinal to your **Value Count** chart by selecting a column (of type int or float) and applying an aggregation (default: sum) to it (sum, mean, etc...) this column will be grouped by the column you're analyzing and the value produced by the aggregation will be used to sort your bars and also displayed in a line. Here's an example:

![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/Histogram.png)
![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/analysis/value_counts_ordinal.PNG
)

*Value Counts* are a bar chart containing the counts of each unique value in a column.
**Category (Category Breakdown)** when viewing float columns you can also see them broken down by a categorical column (string, date, int, etc...). This means that when you select a category column this will then display the frequency of each category in a line as well as bars based on the float column you're analyzing grouped by that category and computed by your aggregation (default: mean).

### Menu Functions Depending on Browser Dimensions
Depending on the dimensions of your browser window the following buttons will not open modals, but rather separate browser windows: Correlations, Describe & Instances (see images from [Jupyter Notebook](#jupyter-notebook), also Charts will always open in a separate browser window)
@@ -786,7 +798,7 @@ $ python
Then view your D-Tale instance in your browser using the link that gets printed


### Global State/Data Storage
## Global State/Data Storage

If D-Tale is running in an environment with multiple python processes (ex: on a web server running [gunicorn](https://github.com/benoitc/gunicorn)) it will most likely encounter issues with inconsistent state. Developers can fix this by configuring the system D-Tale uses for storing data. Detailed documentation is available here: [Data Storage and managing Global State](https://github.com/man-group/dtale/blob/master/docs/GLOBAL_STATE.md)

2 changes: 1 addition & 1 deletion docker/2_7/Dockerfile
Original file line number Diff line number Diff line change
@@ -44,4 +44,4 @@ WORKDIR /app

RUN set -eux \
; . /root/.bashrc \
; easy_install dtale-1.8.0-py2.7.egg
; easy_install dtale-1.8.1-py2.7.egg
2 changes: 1 addition & 1 deletion docker/3_6/Dockerfile
Original file line number Diff line number Diff line change
@@ -44,4 +44,4 @@ WORKDIR /app

RUN set -eux \
; . /root/.bashrc \
; easy_install dtale-1.8.0-py3.7.egg
; easy_install dtale-1.8.1-py3.7.egg
4 changes: 2 additions & 2 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -64,9 +64,9 @@
# built documents.
#
# The short X.Y version.
version = u'1.8.0'
version = u'1.8.1'
# The full version, including alpha/beta/rc tags.
release = u'1.8.0'
release = u'1.8.1'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
19 changes: 13 additions & 6 deletions dtale/charts/utils.py
Original file line number Diff line number Diff line change
@@ -198,8 +198,10 @@ def check_exceptions(df, allow_duplicates, unlimited_data=False, data_limit=1500
:raises Exception: if any failure condition is met
"""
if not allow_duplicates and any(df.duplicated()):
raise Exception(
'{} contains duplicates, please specify group or additional filtering'.format(', '.join(df.columns)))
raise Exception((
"{} contains duplicates, please specify group or additional filtering or select 'No Aggregation' from"
' Aggregation drop-down.'
).format(', '.join(df.columns)))
if not unlimited_data and len(df) > data_limit:
raise Exception(limit_msg.format(data_limit))

@@ -225,7 +227,8 @@ def build_agg_data(df, x, y, inputs, agg, z=None):
:return: dataframe of aggregated data
:rtype: :class:`pandas:pandas.DataFrame`
"""

if agg == 'raw':
return df, []
z_exists = len(make_list(z))
if agg == 'corr':
if not z_exists:
@@ -256,7 +259,7 @@ def build_agg_data(df, x, y, inputs, agg, z=None):
groups = df.groupby(x)
return getattr(groups[y], agg)().reset_index(), [
"chart_data = chart_data.groupby('{x}')[['{y}']].{agg}().reset_index()".format(
x=x, y=y, agg=agg
x=x, y=make_list(y)[0], agg=agg
)
]

@@ -365,8 +368,12 @@ def _group_filter():
code.append("chart_data = chart_data.dropna()")

dupe_cols = [x_col] + (y_cols if len(z_cols) else [])
check_exceptions(data[dupe_cols].rename(columns={'x': x}), allow_duplicates, unlimited_data=unlimited_data,
data_limit=40000 if len(z_cols) else 15000)
check_exceptions(
data[dupe_cols].rename(columns={'x': x}),
allow_duplicates or agg == 'raw',
unlimited_data=unlimited_data,
data_limit=40000 if len(z_cols) else 15000
)
data_f, range_f = build_formatters(data)
ret_data = dict(
data={str('all'): data_f.format_lists(data)},
132 changes: 131 additions & 1 deletion dtale/column_builders.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
import random
import string

import numpy as np
import pandas as pd

@@ -14,8 +17,10 @@ def __init__(self, data_id, column_type, name, cfg):
self.builder = DatetimeColumnBuilder(name, cfg)
elif column_type == 'bins':
self.builder = BinsColumnBuilder(name, cfg)
elif column_type == 'random':
self.builder = RandomColumnBuilder(name, cfg)
else:
raise NotImplementedError('{} column builder not implemented yet!'.format(column_type))
raise NotImplementedError("'{}' column builder not implemented yet!".format(column_type))

def build_column(self):
data = global_state.get_data(self.data_id)
@@ -129,3 +134,128 @@ def build_code(self):
s_str = "df.loc[:, '{name}'] = pd.Series({name}_data.cat.codes.map({name}_cats), index=df.index, name='{name}')"
bins_code.append(s_str.format(name=self.name))
return '\n'.join(bins_code)


def id_generator(size=10, chars=string.ascii_uppercase + string.digits):
return ''.join(random.choice(chars) for _ in range(int(size)))


class RandomColumnBuilder(object):

def __init__(self, name, cfg):
self.name = name
self.cfg = cfg

def build_column(self, data):
rand_type = self.cfg['type']
if 'string' == rand_type:
kwargs = dict(size=self.cfg.get('length', 10))
if self.cfg.get('chars'):
kwargs['chars'] = self.cfg['chars']
return pd.Series(
[id_generator(**kwargs) for _ in range(len(data))], index=data.index, name=self.name
)
if 'int' == rand_type:
low = self.cfg.get('low', 0)
high = self.cfg.get('high', 100)
return pd.Series(
np.random.randint(low, high=high, size=len(data)), index=data.index, name=self.name
)
if 'date' == rand_type:
start = pd.Timestamp(self.cfg.get('start') or '19000101')
end = pd.Timestamp(self.cfg.get('end') or '21991231')
business_days = self.cfg.get('businessDay') is True
timestamps = self.cfg.get('timestamps') is True
if timestamps:
def pp(start, end, n):
start_u = start.value // 10 ** 9
end_u = end.value // 10 ** 9
return pd.DatetimeIndex(
(10 ** 9 * np.random.randint(start_u, end_u, n)).view('M8[ns]')
)

dates = pp(pd.Timestamp(start), pd.Timestamp(end), len(data))
else:
dates = pd.date_range(start, end, freq='B' if business_days else 'D')
dates = [dates[i] for i in np.random.randint(0, len(dates) - 1, size=len(data))]
return pd.Series(dates, index=data.index, name=self.name)
if 'bool' == rand_type:
return pd.Series(np.random.choice([True, False], len(data)), index=data.index, name=self.name)
if 'choice' == rand_type:
choices = self.cfg.get('choices') or 'a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z'
choices = choices.split(',')
return pd.Series(np.random.choice(choices, len(data)), index=data.index, name=self.name)

# floats
low = self.cfg.get('low', 0)
high = self.cfg.get('high', 1)
return pd.Series(np.random.uniform(low, high, len(data)), index=data.index, name=self.name)

def build_code(self):
rand_type = self.cfg['type']
if 'string' == rand_type:
kwargs = []
if self.cfg.get('length') != 10:
kwargs.append('size={size}'.format(size=self.cfg.get('length')))
if self.cfg.get('chars'):
kwargs.append("chars='{chars}'".format(chars=self.cfg.get('chars')))
kwargs = ', '.join(kwargs)
return (
'import number\nimport random\n\n'
'def id_generator(size=1500, chars=string.ascii_uppercase + string.digits):\n'
"\treturn ''.join(random.choice(chars) for _ in range(size))\n\n"
"df.loc[:, '{name}'] = pd.Series([id_generator({kwargs}) for _ in range(len(df)], index=df.index)"
).format(kwargs=kwargs, name=self.name)
return "df.loc[:, '{name}'] = df['{col}'].dt.{property}".format(name=self.name, **self.cfg)

if 'bool' == rand_type:
return (
"df.loc[:, '{name}'] = pd.Series(np.random.choice([True, False], len(df)), index=data.index"
).format(name=self.name)
if 'date' == rand_type:
start = pd.Timestamp(self.cfg.get('start') or '19000101')
end = pd.Timestamp(self.cfg.get('end') or '21991231')
business_days = self.cfg.get('businessDay') is True
timestamps = self.cfg.get('timestamps') is True
if timestamps:
code = (
'def pp(start, end, n):\n'
'\tstart_u = start.value // 10 ** 9\n'
'\tend_u = end.value // 10 ** 9\n'
'\treturn pd.DatetimeIndex(\n'
"\t\t(10 ** 9 * np.random.randint(start_u, end_u, n, dtype=np.int64)).view('M8[ns]')\n"
')\n\n'
"df.loc[:, '{name}'] = pd.Series(\n"
"\tpp(pd.Timestamp('{start}'), pd.Timestamp('{end}'), len(df)), index=df.index\n"
')'
).format(name=self.name, start=start.strftime('%Y%m%d'), end=end.strftime('%Y%m%d'))
else:
freq = ", freq='B'" if business_days else ''
code = (
"dates = pd.date_range('{start}', '{end}'{freq})\n"
'dates = [dates[i] for i in np.random.randint(0, len(dates) - 1, size=len(data))]\n'
"df.loc[:, '{name}'] = pd.Series(dates, index=data.index)"
).format(name=self.name, start=start.strftime('%Y%m%d'), end=end.strftime('%Y%m%d'), freq=freq)
return code
if 'choice' == rand_type:
choices = self.cfg.get('choices') or 'a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z'
choices = choices.split(',')
return "df.loc[:, '{name}'] = pd.Series(np.random.choice({choices}, len(df)), index=df.index)".format(
choices="', '".join(choices), name=self.name
)

if 'int' == rand_type:
low = self.cfg.get('low', 0)
high = self.cfg.get('high', 100)
return (
'import numpy as np\n\n'
"df.loc[:, '{name}'] = pd.Series(np.random.randint({low}, high={high}, size=len(df)), "
'index=df.index)'
).format(name=self.name, low=low, high=high)

low = self.cfg.get('low', 0)
high = self.cfg.get('high', 1)
return (
'import numpy as np\n\n'
"df.loc[:, '{name}'] = pd.Series(np.random.uniform({low}, {high}, len(df)), index=df.index)"
).format(low=low, high=high, name=self.name)
54 changes: 29 additions & 25 deletions dtale/dash_application/charts.py
Original file line number Diff line number Diff line change
@@ -23,8 +23,8 @@
from dtale.dash_application.layout import (AGGS, build_error,
update_label_for_freq)
from dtale.utils import (build_code_export, classify_type, dict_merge,
divide_chunks, flatten_lists, get_dtypes, make_list,
run_query)
divide_chunks, export_to_csv_buffer, flatten_lists,
get_dtypes, make_list, run_query)

if PY3:
from io import StringIO
@@ -208,7 +208,7 @@ def _build_axes(y):
return _build_axes


def build_spaced_ticks(ticktext):
def build_spaced_ticks(ticktext, mode='auto'):
"""
plotly/dash doesn't have particularly good tick position handling so in order to handle this on our end we'll take
the list of tick labels and depending on how large that list is we'll build a configuration which will show a
@@ -220,9 +220,21 @@ def build_spaced_ticks(ticktext):
:rtype: dict
"""
size = len(ticktext)
if size <= 30:
return {'tickmode': 'auto', 'nticks': size}
factor = int(math.ceil(size / 28.0))
tick_cutoff = 30
if mode == 'array':
tickvals = list(range(size))
if size <= tick_cutoff:
return {'tickmode': 'array', 'tickvals': tickvals, 'ticktext': ticktext}
spaced_ticks, spaced_text = [tickvals[0]], [ticktext[0]]
for i in range(factor, size - 1, factor):
spaced_ticks.append(tickvals[i])
spaced_text.append(ticktext[i])
spaced_ticks.append(tickvals[-1])
spaced_text.append(ticktext[-1])
return {'tickmode': 'array', 'tickvals': spaced_ticks, 'ticktext': spaced_text}
if size <= tick_cutoff:
return {'tickmode': 'auto', 'nticks': size}
nticks = len(range(factor, size - 1, factor)) + 2
return {'tickmode': 'auto', 'nticks': nticks}

@@ -544,14 +556,16 @@ def bar_builder(data, x, y, axes_builder, wrapper, cpg=False, barmode='group', b
if barsort is not None:
for series_key, series in data['data'].items():
barsort_col = 'x' if barsort == x or barsort not in series else barsort
if barsort_col != 'x':
if barsort_col != 'x' or kwargs.get('agg') == 'raw':
df = pd.DataFrame(series)
df = df.sort_values(barsort_col)
data['data'][series_key] = {c: df[c].values for c in df.columns}
data['data'][series_key]['x'] = list(range(len(df['x'])))
tickvals = list(range(len(df['x'])))
data['data'][series_key]['x'] = tickvals
hover_text[series_key] = {'hovertext': df['x'].values, 'hoverinfo': 'y+text'}
axes['xaxis'] = dict_merge(
axes.get('xaxis', {}), build_spaced_ticks(df['x'].values)
axes.get('xaxis', {}),
build_spaced_ticks(df['x'].values, mode='array')
)

if cpg:
@@ -754,7 +768,11 @@ def heatmap_builder(data_id, export=False, **inputs):
try:
if not valid_chart(**inputs):
return None, None
raw_data = global_state.get_data(data_id)
raw_data = run_query(
global_state.get_data(data_id),
inputs.get('query'),
global_state.get_context_variables(data_id)
)
wrapper = chart_wrapper(data_id, raw_data, inputs)
hm_kwargs = dict(colorscale='Greens', showscale=True, hoverinfo='x+y+z') # hoverongaps=False,
x, y, z, agg = (inputs.get(p) for p in ['x', 'y', 'z', 'agg'])
@@ -798,9 +816,7 @@ def heatmap_builder(data_id, export=False, **inputs):
code += agg_code
if not len(data):
raise Exception('No data returned for this computation!')
# check_exceptions(data[dupe_cols], agg != 'corr', data_limit=40000,
# limit_msg='Heatmap exceeds {} cells, cannot render. Please apply filter...')
check_exceptions(data[dupe_cols], agg != 'corr', unlimited_data=True)
check_exceptions(data[dupe_cols], agg not in ['corr', 'raw'], unlimited_data=True)
dtypes = {c: classify_type(dtype) for c, dtype in get_dtypes(data).items()}
data_f, _ = chart_formatters(data)
data = data_f.format_df(data)
@@ -1120,16 +1136,4 @@ def export_chart(data_id, params):

def export_chart_data(data_id, params):
data = build_raw_figure_data(data_id, **params)
if PY3:
from io import BytesIO
proxy = StringIO()
data.to_csv(proxy, encoding='utf-8', index=False)
csv_buffer = BytesIO()
csv_buffer.write(proxy.getvalue().encode('utf-8'))
proxy.close()
else:
csv_buffer = StringIO()
data.to_csv(csv_buffer, encoding='utf-8', index=False)

csv_buffer.seek(0)
return csv_buffer
return export_to_csv_buffer(data)
4 changes: 2 additions & 2 deletions dtale/dash_application/layout.py
Original file line number Diff line number Diff line change
@@ -160,7 +160,7 @@ def build_option(value, label=None):
AGGS = dict(
count='Count', nunique='Unique Count', sum='Sum', mean='Mean', rolling='Rolling', corr='Correlation', first='First',
last='Last', median='Median', min='Minimum', max='Maximum', std='Standard Deviation', var='Variance',
mad='Mean Absolute Deviation', prod='Product of All Items'
mad='Mean Absolute Deviation', prod='Product of All Items', raw='No Aggregation'
)
FREQS = ['H', 'H2', 'WD', 'D', 'W', 'M', 'Q', 'Y']
FREQ_LABELS = dict(H='Hourly', H2='Hour', WD='Weekday', W='Weekly', M='Monthly', Q='Quarterly', Y='Yearly')
@@ -390,7 +390,7 @@ def charts_layout(df, settings, **inputs):
id='agg-dropdown',
options=[build_option(v, AGGS[v]) for v in ['count', 'nunique', 'sum', 'mean', 'rolling', 'corr',
'first', 'last', 'median', 'min', 'max', 'std', 'var',
'mad', 'prod']],
'mad', 'prod', 'raw']],
placeholder='Select an aggregation',
style=dict(width='inherit'),
value=agg,
5 changes: 4 additions & 1 deletion dtale/static/css/dash.css
Original file line number Diff line number Diff line change
@@ -439,7 +439,10 @@ div.tab-container > div.tab:last-child {
#y-multi-dropdown .Select-menu-outer,
#z-dropdown .Select-menu-outer,
#group-dropdown .Select-menu-outer,
#agg-dropdown .Select-menu-outer {
#agg-dropdown .Select-menu-outer,
#barmode-dropdown .Select-menu-outer,
#barsort-dropdown .Select-menu-outer,
#yaxis-dropdown .Select-menu-outer {
z-index: 7;
}
#yaxis-type-div {
14 changes: 7 additions & 7 deletions dtale/static/css/github_fork.css
Original file line number Diff line number Diff line change
@@ -3,9 +3,9 @@
display:block;
top:0;
right:0;
width:125px;
width:85px;
overflow:hidden;
height:125px;
height:85px;
z-index:9999;
}
#forkongithub a{
@@ -15,19 +15,19 @@
font-family:arial,sans-serif;
text-align:center;
font-weight:bold;
font-size:0.45rem;
line-height:2rem;
font-size:0.55rem;
line-height:1.35rem;
transition:0.5s;
width:125px;
position:absolute;
top:17px;
right:-31px;
top:23px;
right:-30px;
transform:rotate(45deg);
-webkit-transform:rotate(45deg);
-ms-transform:rotate(45deg);
-moz-transform:rotate(45deg);
-o-transform:rotate(45deg);
box-shadow:4px 4px 10px rgba(0,0,0,0.8);
box-shadow:2px 2px 5px rgba(0,0,0,0.8);
}
#forkongithub a:hover{
background:#c11;
70 changes: 70 additions & 0 deletions dtale/static/css/main.css
Original file line number Diff line number Diff line change
@@ -10530,4 +10530,74 @@ div.container-fluid.code-export > div#popup-content > div.modal-footer, {
}
}

@keyframes dtype-spin
{
0% { transform: rotate(0);}
100% { transform: rotate(360deg); }
}

.dtype-highlighting
{
height: 18px;
width: 18px;
box-sizing: border-box;
border: 5px solid; /* (1.5em * 0.25) */
border-color: rgba(102,205,170,1.0) rgba(65,105,225,0.5) rgba(255,182,193,1.0) rgba(255,215,0,1.0);
border-radius: 50%;
opacity: 0.9;
float: left;
}

.dtype-highlighting.spin {
animation: dtype-spin 1s infinite linear;
}

.btn-group.column-sorting .btn.btn-primary {
border: solid 1px #a7b3b7;
-webkit-box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.2);
box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.2);
background: -webkit-gradient(linear, left top, left bottom, color-stop(80%, white), to(#ebedee));
background: linear-gradient(to bottom, white 80%, #ebedee);
color: #404040;
}

.btn-group.column-sorting .btn.btn-primary:enabled:hover,
.btn-group.column-sorting .btn.btn-primary:enabled:focus {
border: solid 1px #99a7ac;
-webkit-box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.3);
box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.3);
}

.btn-group.column-sorting .btn.btn-primary:enabled:active {
border: solid 1px #99a7ac;
-webkit-box-shadow: 0 0 1px 0 rgba(112, 130, 136, 0.3), 0 0 1px 0 #cfd4d7 inset;
box-shadow: 0 0 1px 0 rgba(112, 130, 136, 0.3), 0 0 1px 0 #cfd4d7 inset;
background: -webkit-gradient(linear, left top, left bottom, from(#ebedee), to(white));
background: linear-gradient(to bottom, #ebedee, white);
}

.btn-group.column-sorting .btn.btn-primary.active {
border: solid 1px #88989e;
-webkit-box-shadow: 0 0 3px 0 #88989e inset;
box-shadow: 0 0 3px 0 #88989e inset;
background: #a7b3b7;
color: white;
}

.btn-group.column-sorting .btn.btn-primary.active:enabled:hover {
border: solid 1px #88989e;
-webkit-box-shadow: 0 0 3px 0 #88989e inset;
box-shadow: 0 0 3px 0 #88989e inset;
background: #a7b3b7;
cursor: default;
}

.btn-group.column-sorting .btn.btn-primary.active:enabled:active,
.btn-group.column-sorting .btn.btn-primary.active:enabled:focus {
border: solid 1px #88989e;
-webkit-box-shadow: 0 0 3px 0 #88989e inset;
box-shadow: 0 0 3px 0 #88989e inset;
background: #a7b3b7;
}


2 changes: 1 addition & 1 deletion dtale/templates/dtale/popup.html
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@
</header>
</div>
<div class="col"></div>
<div class="col-auto mt-4" style="{{'padding-right: 125px' if config.GITHUB_FORK else ''}}">
<div class="col-auto mt-4" style="{{'padding-right: 85px' if config.GITHUB_FORK else ''}}">
<a href="#" onclick="window.open('/dtale/main/{{data_id}}'); return false;">
<i class="fas fa-th mr-4"></i>
<span>Back To Data</span>
22 changes: 21 additions & 1 deletion dtale/utils.py
Original file line number Diff line number Diff line change
@@ -734,7 +734,7 @@ def build_code_export(data_id, imports='import pandas as pd\n\n', query=None):
"\n# DISCLAIMER: running this line in a different process than the one it originated will produce\n"
"# differing results\n"
"ctxt_vars = dtale_global_state.get_context_variables('{data_id}')\n\n"
"df = df.query('{query}', local_dict=ctxt_vars)\n"
'df = df.query("{query}", local_dict=ctxt_vars)\n'
).format(query=final_query, data_id=data_id))
else:
final_history.append("df = df.query('{}')\n".format(final_query))
@@ -749,3 +749,23 @@ def build_code_export(data_id, imports='import pandas as pd\n\n', query=None):
cols=', '.join(cols), dirs="', '".join(dirs)
))
return final_history


def export_to_csv_buffer(data, tsv=False):
kwargs = dict(encoding='utf-8', index=False)
if tsv:
kwargs['sep'] = '\t'
if PY3:
from io import BytesIO, StringIO
proxy = StringIO()
data.to_csv(proxy, **kwargs)
csv_buffer = BytesIO()
csv_buffer.write(proxy.getvalue().encode('utf-8'))
proxy.close()
else:
from StringIO import StringIO
csv_buffer = StringIO()
data.to_csv(csv_buffer, **kwargs)

csv_buffer.seek(0)
return csv_buffer
122 changes: 93 additions & 29 deletions dtale/views.py
Original file line number Diff line number Diff line change
@@ -27,12 +27,12 @@
from dtale.data_reshapers import DataReshaper
from dtale.utils import (DuplicateDataError, build_code_export, build_query,
build_shutdown_url, classify_type, dict_merge,
divide_chunks, find_dtype, find_dtype_formatter,
find_selected_column, get_bool_arg, get_dtypes,
get_int_arg, get_json_arg, get_str_arg, grid_columns,
grid_formatter, json_date, json_float, json_int,
json_timestamp, jsonify, make_list,
retrieve_grid_params, run_query,
divide_chunks, export_to_csv_buffer, find_dtype,
find_dtype_formatter, find_selected_column,
get_bool_arg, get_dtypes, get_int_arg, get_json_arg,
get_str_arg, grid_columns, grid_formatter, json_date,
json_float, json_int, json_timestamp, jsonify,
make_list, retrieve_grid_params, run_query,
running_with_flask_debug, running_with_pytest,
sort_df_for_grid)

@@ -843,6 +843,7 @@ def build_column(data_id):
name = get_str_arg(request, 'name')
if not name:
raise Exception("'name' is required for new column!")
name = str(name)
data = global_state.get_data(data_id)
if name in data.columns:
raise Exception("A column named '{}' already exists!".format(name))
@@ -854,7 +855,10 @@ def build_column(data_id):
dtype = find_dtype(data[name])
data_ranges = {}
if classify_type(dtype) == 'F' and not data[name].isnull().all():
data_ranges[name] = data[[name]].agg([min, max]).to_dict()[name]
try:
data_ranges[name] = data[[name]].agg(['min', 'max']).to_dict()[name]
except ValueError:
pass
dtype_f = dtype_formatter(data, {name: dtype}, data_ranges)
global_state.set_data(data_id, data)
curr_dtypes = global_state.get_dtypes(data_id)
@@ -1135,6 +1139,27 @@ def get_data(data_id):
return jsonify(dict(error=str(e), traceback=str(traceback.format_exc())))


@dtale.route('/data-export/<data_id>')
def data_export(data_id):
try:
curr_settings = global_state.get_settings(data_id) or {}
curr_dtypes = global_state.get_dtypes(data_id) or []
data = run_query(
global_state.get_data(data_id),
build_query(data_id, curr_settings.get('query')),
global_state.get_context_variables(data_id),
ignore_empty=True,
)
data = data[[c['name'] for c in sorted(curr_dtypes, key=lambda c: c['index']) if c['visible']]]
tsv = get_str_arg(request, 'tsv') == 'true'
file_ext = 'tsv' if tsv else 'csv'
csv_buffer = export_to_csv_buffer(data, tsv=tsv)
filename = build_chart_filename('data', ext=file_ext)
return send_file(csv_buffer, filename, 'text/{}'.format(file_ext))
except BaseException as e:
return jsonify(error=str(e), traceback=str(traceback.format_exc()))


@dtale.route('/column-analysis/<data_id>')
def get_column_analysis(data_id):
"""
@@ -1146,42 +1171,80 @@ def get_column_analysis(data_id):
:param type: string from flask.request.args['type'] to signify either a histogram or value counts
:param query: string from flask.request.args['query'] which is applied to DATA using the query() function
:param bins: the number of bins to display in your histogram, options on the front-end are 5, 10, 20, 50
:param top: the number of top values to display in your value counts, default is 100
:returns: JSON {results: DATA, desc: output from pd.DataFrame[col].describe(), success: True/False}
"""
try:
col = get_str_arg(request, 'col', 'values')
bins = get_int_arg(request, 'bins', 20)
data_type = get_str_arg(request, 'type') or 'histogram'
data = run_query(
global_state.get_data(data_id),
build_query(data_id, get_str_arg(request, 'query')),
global_state.get_context_variables(data_id)
)
top = get_int_arg(request, 'top', 100)
ordinal_col = get_str_arg(request, 'ordinalCol')
ordinal_agg = get_str_arg(request, 'ordinalAgg', 'sum')
category_col = get_str_arg(request, 'categoryCol')
category_agg = get_str_arg(request, 'categoryAgg', 'mean')
data_type = get_str_arg(request, 'type')
curr_settings = global_state.get_settings(data_id) or {}
query = build_query(data_id, curr_settings.get('query'))
data = run_query(global_state.get_data(data_id), query, global_state.get_context_variables(data_id))
selected_col = find_selected_column(data, col)
data = data[~pd.isnull(data[selected_col])][[selected_col]]
cols = [selected_col]
if ordinal_col is not None:
cols.append(ordinal_col)
if category_col is not None:
cols.append(category_col)
data = data[~pd.isnull(data[selected_col])][cols]

code = build_code_export(data_id, imports='import numpy as np\nimport pandas as pd\n\n')
dtype = get_dtypes(data)[selected_col]
classifier = classify_type(dtype)
if classifier in ['S', 'D'] or data_type != 'histogram':
hist = pd.value_counts(data[selected_col]).reset_index()
hist.columns = ['labels', 'data']
if data_type is None:
data_type = 'histogram' if classifier in ['F', 'I'] else 'value_counts'
if data_type == 'value_counts':
hist = pd.value_counts(data[selected_col]).to_frame(name='data').sort_index()
code.append("chart = pd.value_counts(df[~pd.isnull(df['{col}'])]['{col}'])".format(col=selected_col))
if ordinal_col is not None:
ordinal_data = getattr(data.groupby(selected_col)[[ordinal_col]], ordinal_agg)()
hist['ordinal'] = ordinal_data
hist = hist.sort_values('ordinal')
code.append((
"ordinal_data = df.groupby('{col}')[['{ordinal}']].{agg}()\n"
"chart['ordinal'] = ordinal_data\n"
"chart = chart.sort_values('ordinal')"
).format(col=selected_col, ordinal=ordinal_col, agg=ordinal_agg))
hist.index.name = 'labels'
hist = hist.reset_index()
if top is not None:
top = int(top)
hist = hist[:top] if top > 0 else hist[top:]
col_types = grid_columns(hist)
f = grid_formatter(col_types, nan_display=None)
return_data = f.format_lists(hist)
return_data['dtype'] = dtype
return_data['chart_type'] = 'value_counts'
code.append("hist = pd.value_counts(df[~pd.isnull(df['{col}'])]['{col}'])".format(col=selected_col))
else:
elif data_type == 'categories':
hist = data.groupby(category_col)[[selected_col]].agg(['count', category_agg])
hist.columns = hist.columns.droplevel(0)
hist.columns = ['count', 'data']
code.append(
"chart = data.groupby('{cat}')[['{col}']].agg(['count', '{agg}'])".format(
cat=category_col, col=selected_col, agg=category_agg)
)
hist.index.name = 'labels'
hist = hist.reset_index()
if top is not None:
top = int(top)
hist = hist[:top] if top > 0 else hist[top:]
f = grid_formatter(grid_columns(hist), nan_display=None)
return_data = f.format_lists(hist)
elif data_type == 'histogram':
hist_data, hist_labels = np.histogram(data, bins=bins)
hist_data = [json_float(h) for h in hist_data]
hist_labels = ['{0:.1f}'.format(l) for l in hist_labels]
code.append("hist = np.histogram(df[~pd.isnull(df['{col}'])][['{col}']], bins={bins})".format(
code.append("chart = np.histogram(df[~pd.isnull(df['{col}'])][['{col}']], bins={bins})".format(
col=selected_col, bins=bins))
desc, desc_code = load_describe(data[selected_col])
code += desc_code
return_data = dict(labels=hist_labels, data=hist_data, desc=desc, dtype=dtype, chart_type='histogram')
return jsonify(code='\n'.join(code), **return_data)
return_data = dict(labels=hist_labels, data=hist_data, desc=desc)
cols = global_state.get_dtypes(data_id)
return jsonify(code='\n'.join(code), query=query, cols=cols, dtype=dtype, chart_type=data_type, **return_data)
except BaseException as e:
return jsonify(dict(error=str(e), traceback=str(traceback.format_exc())))

@@ -1203,9 +1266,10 @@ def get_correlations(data_id):
} or {error: 'Exception message', traceback: 'Exception stacktrace'}
"""
try:
curr_settings = global_state.get_settings(data_id) or {}
data = run_query(
global_state.get_data(data_id),
build_query(data_id, get_str_arg(request, 'query')),
build_query(data_id, curr_settings.get('query')),
global_state.get_context_variables(data_id)
)
valid_corr_cols = []
@@ -1308,17 +1372,17 @@ def get_correlations_ts(data_id):
:param data_id: integer string identifier for a D-Tale process's data
:type data_id: str
:param query: string from flask.request.args['query'] which is applied to DATA using the query() function
:param cols: comma-separated string from flask.request.args['cols'] containing names of two columns in dataframe
:param dateCol: string from flask.request.args['dateCol'] with name of date-type column in dateframe for timeseries
:returns: JSON {
data: {:col1:col2: {data: [{corr: 0.99, date: 'YYYY-MM-DD'},...], max: 0.99, min: 0.99}
} or {error: 'Exception message', traceback: 'Exception stacktrace'}
"""
try:
curr_settings = global_state.get_settings(data_id) or {}
data = run_query(
global_state.get_data(data_id),
build_query(data_id, get_str_arg(request, 'query')),
build_query(data_id, curr_settings.get('query')),
global_state.get_context_variables(data_id)
)
cols = get_str_arg(request, 'cols')
@@ -1366,7 +1430,6 @@ def get_scatter(data_id):
:param data_id: integer string identifier for a D-Tale process's data
:type data_id: str
:param query: string from flask.request.args['query'] which is applied to DATA using the query() function
:param cols: comma-separated string from flask.request.args['cols'] containing names of two columns in dataframe
:param dateCol: string from flask.request.args['dateCol'] with name of date-type column in dateframe for timeseries
:param date: string from flask.request.args['date'] date value in dateCol to filter dataframe to
@@ -1390,9 +1453,10 @@ def get_scatter(data_id):
date_col = get_str_arg(request, 'dateCol')
rolling = get_bool_arg(request, 'rolling')

curr_settings = global_state.get_settings(data_id) or {}
data = run_query(
global_state.get_data(data_id),
build_query(data_id, get_str_arg(request, 'query')),
build_query(data_id, curr_settings.get('query')),
global_state.get_context_variables(data_id)
)
idx_col = str('index')
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "dtale",
"version": "1.8.0",
"version": "1.8.1",
"description": "Visualizer for Pandas Data Structures",
"main": "main.js",
"directories": {
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
@@ -50,7 +50,7 @@ def run_tests(self):

setup(
name="dtale",
version="1.8.0",
version="1.8.1",
author="MAN Alpha Technology",
author_email="ManAlphaTech@man.com",
description="Web Client for Visualizing Pandas Objects",
18 changes: 4 additions & 14 deletions static/__tests__/dtale/DataViewer-base-test.jsx
Original file line number Diff line number Diff line change
@@ -114,20 +114,10 @@ describe("DataViewer tests", () => {
.find(DataViewerMenu)
.find("ul li span.font-weight-bold")
.map(s => s.text()),
_.concat([
"Describe",
"Filter",
"Build Column",
"Reshape",
"Correlations",
"Charts",
"Resize",
"Heat Map",
"Instances 1",
"Code Export",
"About",
"Shutdown",
]),
_.concat(
["Describe", "Filter", "Build Column", "Reshape", "Correlations", "Charts", "Heat Map", "Highlight Dtypes"],
["Instances 1", "Code Export", "Export", "Resize", "About", "Shutdown"]
),
"Should render default menu options"
);
setTimeout(() => {
Original file line number Diff line number Diff line change
@@ -3,10 +3,10 @@ import React from "react";
import { Provider } from "react-redux";
import Select from "react-select";

import mockPopsicle from "../MockPopsicle";
import * as t from "../jest-assertions";
import reduxUtils from "../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../test-utils";
import mockPopsicle from "../../MockPopsicle";
import * as t from "../../jest-assertions";
import reduxUtils from "../../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../../test-utils";

const originalOffsetHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetHeight");
const originalOffsetWidth = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetWidth");
@@ -34,7 +34,7 @@ describe("DataViewer tests", () => {

const mockBuildLibs = withGlobalJquery(() =>
mockPopsicle.mock(url => {
const { urlFetcher } = require("../redux-test-utils").default;
const { urlFetcher } = require("../../redux-test-utils").default;
return urlFetcher(url);
})
);
@@ -61,9 +61,9 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build bins cut column", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const CreateColumn = require("../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateBins } = require("../../popups/create/CreateBins");
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateBins } = require("../../../popups/create/CreateBins");

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
@@ -132,9 +132,9 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build bins qcut column", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const CreateColumn = require("../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateBins } = require("../../popups/create/CreateBins");
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateBins } = require("../../../popups/create/CreateBins");

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
@@ -198,7 +198,7 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build bins cfg validation", done => {
const { validateBinsCfg } = require("../../popups/create/CreateBins");
const { validateBinsCfg } = require("../../../popups/create/CreateBins");
const cfg = { col: null };
t.equal(validateBinsCfg(cfg), "Missing a column selection!");
cfg.col = "x";
Original file line number Diff line number Diff line change
@@ -3,10 +3,10 @@ import React from "react";
import { Provider } from "react-redux";
import Select from "react-select";

import mockPopsicle from "../MockPopsicle";
import * as t from "../jest-assertions";
import reduxUtils from "../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../test-utils";
import mockPopsicle from "../../MockPopsicle";
import * as t from "../../jest-assertions";
import reduxUtils from "../../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../../test-utils";

const originalOffsetHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetHeight");
const originalOffsetWidth = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetWidth");
@@ -34,7 +34,7 @@ describe("DataViewer tests", () => {

const mockBuildLibs = withGlobalJquery(() =>
mockPopsicle.mock(url => {
const { urlFetcher } = require("../redux-test-utils").default;
const { urlFetcher } = require("../../redux-test-utils").default;
return urlFetcher(url);
})
);
@@ -61,9 +61,9 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build datetime property column", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const CreateColumn = require("../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateDatetime } = require("../../popups/create/CreateDatetime");
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateDatetime } = require("../../../popups/create/CreateDatetime");

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
@@ -91,7 +91,7 @@ describe("DataViewer tests", () => {
.find("div.form-group")
.at(1)
.find("button")
.last()
.at(2)
.simulate("click");
result.update();
t.equal(result.find(CreateDatetime).length, 1, "should show build datetime column");
@@ -122,9 +122,9 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build datetime conversion column", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const CreateColumn = require("../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateDatetime } = require("../../popups/create/CreateDatetime");
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateDatetime } = require("../../../popups/create/CreateDatetime");

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
@@ -152,7 +152,7 @@ describe("DataViewer tests", () => {
.find("div.form-group")
.at(1)
.find("button")
.last()
.at(2)
.simulate("click");
result.update();
t.equal(result.find(CreateDatetime).length, 1, "should show build datetime column");
@@ -189,7 +189,7 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build datetime cfg validation", done => {
const { validateDatetimeCfg } = require("../../popups/create/CreateDatetime");
const { validateDatetimeCfg } = require("../../../popups/create/CreateDatetime");
t.equal(validateDatetimeCfg({ col: null }), "Missing a column selection!");
done();
});
Original file line number Diff line number Diff line change
@@ -4,11 +4,11 @@ import { ModalClose } from "react-modal-bootstrap";
import { Provider } from "react-redux";
import Select from "react-select";

import { RemovableError } from "../../RemovableError";
import mockPopsicle from "../MockPopsicle";
import * as t from "../jest-assertions";
import reduxUtils from "../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../test-utils";
import { RemovableError } from "../../../RemovableError";
import mockPopsicle from "../../MockPopsicle";
import * as t from "../../jest-assertions";
import reduxUtils from "../../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../../test-utils";

const originalOffsetHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetHeight");
const originalOffsetWidth = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetWidth");
@@ -36,7 +36,7 @@ describe("DataViewer tests", () => {

const mockBuildLibs = withGlobalJquery(() =>
mockPopsicle.mock(url => {
const { urlFetcher } = require("../redux-test-utils").default;
const { urlFetcher } = require("../../redux-test-utils").default;
return urlFetcher(url);
})
);
@@ -63,9 +63,9 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build numeric column", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const CreateColumn = require("../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateNumeric } = require("../../popups/create/CreateNumeric");
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateNumeric } = require("../../../popups/create/CreateNumeric");

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
@@ -146,9 +146,9 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build column errors", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const CreateColumn = require("../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateNumeric } = require("../../popups/create/CreateNumeric");
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const { CreateNumeric } = require("../../../popups/create/CreateNumeric");

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
@@ -245,7 +245,7 @@ describe("DataViewer tests", () => {
});

test("DataViewer: build numeric cfg validation", done => {
const { validateNumericCfg } = require("../../popups/create/CreateNumeric");
const { validateNumericCfg } = require("../../../popups/create/CreateNumeric");
const cfg = {};
t.equal(validateNumericCfg(cfg), "Please select an operation!");
cfg.operation = "x";
395 changes: 395 additions & 0 deletions static/__tests__/dtale/create/random-test.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,395 @@
/* eslint max-lines: "off" */
import { mount } from "enzyme";
import moment from "moment";
import React from "react";
import { Provider } from "react-redux";

import { CreateRandom } from "../../../popups/create/CreateRandom";
import mockPopsicle from "../../MockPopsicle";
import * as t from "../../jest-assertions";
import reduxUtils from "../../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../../test-utils";

const originalOffsetHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetHeight");
const originalOffsetWidth = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetWidth");
const originalInnerWidth = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "innerWidth");
const originalInnerHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "innerHeight");

function initialize(res) {
res
.find("div.form-group")
.first()
.find("input")
.first()
.simulate("change", { target: { value: "rando_col" } });
res
.find("div.form-group")
.at(1)
.find("button")
.last()
.simulate("click");
}

function submit(res) {
res
.find("div.modal-footer")
.first()
.find("button")
.first()
.simulate("click");
}

describe("DataViewer tests", () => {
beforeAll(() => {
Object.defineProperty(HTMLElement.prototype, "offsetHeight", {
configurable: true,
value: 500,
});
Object.defineProperty(HTMLElement.prototype, "offsetWidth", {
configurable: true,
value: 500,
});
Object.defineProperty(window, "innerWidth", {
configurable: true,
value: 1205,
});
Object.defineProperty(window, "innerHeight", {
configurable: true,
value: 775,
});

const mockBuildLibs = withGlobalJquery(() =>
mockPopsicle.mock(url => {
const { urlFetcher } = require("../../redux-test-utils").default;
return urlFetcher(url);
})
);

const mockChartUtils = withGlobalJquery(() => (ctx, cfg) => {
const chartCfg = { ctx, cfg, data: cfg.data, destroyed: false };
chartCfg.destroy = () => (chartCfg.destroyed = true);
chartCfg.getElementsAtXAxis = _evt => [{ _index: 0 }];
chartCfg.getElementAtEvent = _evt => [{ _datasetIndex: 0, _index: 0, _chart: { config: cfg, data: cfg.data } }];
return chartCfg;
});

jest.mock("popsicle", () => mockBuildLibs);
jest.mock("chart.js", () => mockChartUtils);
jest.mock("chartjs-plugin-zoom", () => ({}));
jest.mock("chartjs-chart-box-and-violin-plot/build/Chart.BoxPlot.js", () => ({}));
});

afterAll(() => {
Object.defineProperty(HTMLElement.prototype, "offsetHeight", originalOffsetHeight);
Object.defineProperty(HTMLElement.prototype, "offsetWidth", originalOffsetWidth);
Object.defineProperty(window, "innerWidth", originalInnerWidth);
Object.defineProperty(window, "innerHeight", originalInnerHeight);
});

test("DataViewer: build random float column", done => {
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
const result = mount(
<Provider store={store}>
<DataViewer />
</Provider>,
{ attachTo: document.getElementById("content") }
);

setTimeout(() => {
result.update();
clickMainMenuButton(result, "Build Column");
setTimeout(() => {
result.update();
initialize(result.find(CreateColumn));
result.update();
t.equal(result.find(CreateRandom).length, 1, "should show build random column");
const randomInputs = result.find(CreateRandom).first();
randomInputs
.find("div.form-group")
.at(1)
.find("input")
.first()
.simulate("change", { target: { value: "-2" } });
randomInputs
.find("div.form-group")
.last()
.find("input")
.simulate("change", { target: { value: "2" } });
submit(result);
setTimeout(() => {
t.deepEqual(result.find(CreateColumn).instance().state.cfg, {
type: "float",
low: "-2",
high: "2",
});
result.update();
done();
}, 400);
}, 400);
}, 600);
});

test("DataViewer: build random int column", done => {
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
const result = mount(
<Provider store={store}>
<DataViewer />
</Provider>,
{ attachTo: document.getElementById("content") }
);

setTimeout(() => {
result.update();
clickMainMenuButton(result, "Build Column");
setTimeout(() => {
result.update();
initialize(result.find(CreateColumn));
result.update();
const randomInputs = result.find(CreateRandom).first();
randomInputs
.find("div.form-group")
.first()
.find("button")
.at(1)
.simulate("click");
randomInputs
.find("div.form-group")
.at(1)
.find("input")
.simulate("change", { target: { value: "-2" } });
randomInputs
.find("div.form-group")
.last()
.find("input")
.simulate("change", { target: { value: "2" } });
submit(result);
setTimeout(() => {
t.deepEqual(result.find(CreateColumn).instance().state.cfg, {
type: "int",
low: "-2",
high: "2",
});
result.update();
done();
}, 400);
}, 400);
}, 600);
});

test("DataViewer: build random string column", done => {
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
const result = mount(
<Provider store={store}>
<DataViewer />
</Provider>,
{ attachTo: document.getElementById("content") }
);

setTimeout(() => {
result.update();
clickMainMenuButton(result, "Build Column");
setTimeout(() => {
result.update();
initialize(result.find(CreateColumn));
result.update();
const randomInputs = result.find(CreateRandom).first();
randomInputs
.find("div.form-group")
.first()
.find("button")
.at(2)
.simulate("click");
randomInputs
.find("div.form-group")
.at(1)
.find("input")
.simulate("change", { target: { value: "5" } });
randomInputs
.find("div.form-group")
.last()
.find("input")
.simulate("change", { target: { value: "abcde" } });
submit(result);
setTimeout(() => {
t.deepEqual(result.find(CreateColumn).instance().state.cfg, {
type: "string",
chars: "abcde",
length: "5",
});
result.update();
done();
}, 400);
}, 400);
}, 600);
});

test("DataViewer: build random choice column", done => {
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
const result = mount(
<Provider store={store}>
<DataViewer />
</Provider>,
{ attachTo: document.getElementById("content") }
);

setTimeout(() => {
result.update();
clickMainMenuButton(result, "Build Column");
setTimeout(() => {
result.update();
initialize(result.find(CreateColumn));
result.update();
const randomInputs = result.find(CreateRandom).first();
randomInputs
.find("div.form-group")
.first()
.find("button")
.at(3)
.simulate("click");
randomInputs
.find("div.form-group")
.at(1)
.find("input")
.simulate("change", { target: { value: "foo,bar,baz" } });
submit(result);
setTimeout(() => {
t.deepEqual(result.find(CreateColumn).instance().state.cfg, {
type: "choice",
choices: "foo,bar,baz",
});
result.update();
done();
}, 400);
}, 400);
}, 600);
});

test("DataViewer: build random bool column", done => {
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
const result = mount(
<Provider store={store}>
<DataViewer />
</Provider>,
{ attachTo: document.getElementById("content") }
);

setTimeout(() => {
result.update();
clickMainMenuButton(result, "Build Column");
setTimeout(() => {
result.update();
initialize(result.find(CreateColumn));
result.update();
const randomInputs = result.find(CreateRandom).first();
randomInputs
.find("div.form-group")
.first()
.find("button")
.at(4)
.simulate("click");
submit(result);
setTimeout(() => {
t.deepEqual(result.find(CreateColumn).instance().state.cfg, {
type: "bool",
});
result.update();
done();
}, 400);
}, 400);
}, 600);
});

test("DataViewer: build random date column", done => {
const { DataViewer } = require("../../../dtale/DataViewer");
const CreateColumn = require("../../../popups/create/CreateColumn").ReactCreateColumn;
const DateInput = require("@blueprintjs/datetime").DateInput;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "" }, store);
const result = mount(
<Provider store={store}>
<DataViewer />
</Provider>,
{ attachTo: document.getElementById("content") }
);

setTimeout(() => {
result.update();
clickMainMenuButton(result, "Build Column");
setTimeout(() => {
result.update();
initialize(result.find(CreateColumn));
result.update();
const randomInputs = result.find(CreateRandom).first();
randomInputs
.find("div.form-group")
.first()
.find("button")
.last()
.simulate("click");
const dateInputs = result.find(CreateColumn).find(DateInput);
dateInputs
.first()
.instance()
.props.onChange(new Date(moment("20000101")));
dateInputs
.find(DateInput)
.last()
.instance()
.props.onChange(new Date(moment("20000102")));
result
.find(CreateColumn)
.find("i")
.first()
.simulate("click");
result
.find(CreateColumn)
.find("i")
.last()
.simulate("click");
submit(result);
setTimeout(() => {
t.deepEqual(result.find(CreateColumn).instance().state.cfg, {
type: "date",
start: "20000101",
end: "20000102",
businessDay: true,
timestamps: true,
});
result.update();
done();
}, 400);
}, 400);
}, 600);
});

test("DataViewer: build random cfg validation", done => {
const { validateRandomCfg } = require("../../../popups/create/CreateRandom");
t.equal(
validateRandomCfg({ type: "int", low: "3", high: "2" }),
"Invalid range specification, low must be less than high!"
);
t.equal(validateRandomCfg({ type: "date", start: "20000101", end: "19991231" }), "Start must be before End!");
done();
});
});
19 changes: 10 additions & 9 deletions static/__tests__/filters/ColumnFilter-date-test.jsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { mount } from "enzyme";
import _ from "lodash";
import moment from "moment";
import React from "react";

import { DateFilter } from "../../filters/DateFilter";
@@ -65,29 +66,29 @@ describe("ColumnFilter date tests", () => {
.find(DateInput)
.first()
.instance();
dateStart.state = "200";
dateStart.props.onChange({ value: "200" });
dateStart.state = "20000102";
dateStart.props.onChange({ value: "20000102" });
dateStart.inputEl.value = "200";
dateStart.props.onChange("200");
dateStart.inputEl.value = "20000102";
dateStart.props.onChange(new Date(moment("20000102")));
setTimeout(() => {
result.update();
t.deepEqual(result.state().cfg, {
type: "date",
start: "20200322",
start: "20000102",
end: "20000131",
});
const dateEnd = result
.find(DateInput)
.last()
.instance();
dateEnd.state = "20000103";
dateEnd.props.onChange({ value: "20000103" });
dateEnd.inputEl.value = "20000103";
dateEnd.props.onChange(new Date(moment("20000103")));
setTimeout(() => {
result.update();
t.deepEqual(result.state().cfg, {
type: "date",
start: "20200322",
end: "20200322",
start: "20000102",
end: "20000103",
});
done();
}, 400);
19 changes: 16 additions & 3 deletions static/__tests__/iframe/DataViewer-base-test.jsx
Original file line number Diff line number Diff line change
@@ -9,7 +9,7 @@ import MultiGrid from "react-virtualized/dist/commonjs/MultiGrid";
import mockPopsicle from "../MockPopsicle";
import * as t from "../jest-assertions";
import reduxUtils from "../redux-test-utils";
import { buildInnerHTML, clickMainMenuButton, withGlobalJquery } from "../test-utils";
import { buildInnerHTML, clickMainMenuButton, findMainMenuButton, withGlobalJquery } from "../test-utils";
import { clickColMenuButton, clickColMenuSubButton } from "./iframe-utils";

const originalOffsetHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetHeight");
@@ -144,8 +144,8 @@ describe("DataViewer iframe tests", () => {
.find("ul li span.font-weight-bold")
.map(s => s.text()),
_.concat(
["Describe", "Filter", "Build Column", "Reshape", "Correlations", "Charts", "Resize", "Heat Map"],
["Instances 1", "Code Export", "About", "Refresh", "Open Popup", "Shutdown"]
["Describe", "Filter", "Build Column", "Reshape", "Correlations", "Charts", "Heat Map", "Highlight Dtypes"],
["Instances 1", "Code Export", "Export", "Resize", "About", "Refresh", "Open Popup", "Shutdown"]
),
"Should render default menu options"
);
@@ -310,6 +310,19 @@ describe("DataViewer iframe tests", () => {
expect(window.open.mock.calls[window.open.mock.calls.length - 1][0]).toBe("/charts/1");
clickMainMenuButton(result, "Instances 1");
expect(window.open.mock.calls[window.open.mock.calls.length - 1][0]).toBe("/dtale/popup/instances/1");
const exports = findMainMenuButton(result, "CSV", "div.btn-group");
exports
.find("button")
.first()
.simulate("click");
let exportURL = window.open.mock.calls[window.open.mock.calls.length - 1][0];
t.ok(_.startsWith(exportURL, "/dtale/data-export/1") && _.includes(exportURL, "tsv=false"));
exports
.find("button")
.last()
.simulate("click");
exportURL = window.open.mock.calls[window.open.mock.calls.length - 1][0];
t.ok(_.startsWith(exportURL, "/dtale/data-export/1") && _.includes(exportURL, "tsv=true"));
clickMainMenuButton(result, "Resize");
clickMainMenuButton(result, "Refresh");
expect(window.location.reload).toHaveBeenCalled();
2 changes: 1 addition & 1 deletion static/__tests__/iframe/DataViewer-modal-test.jsx
Original file line number Diff line number Diff line change
@@ -63,7 +63,7 @@ describe("DataViewer iframe tests", () => {
test("DataViewer: column analysis display in a modal", done => {
const { DataViewer } = require("../../dtale/DataViewer");
const ColumnMenu = require("../../dtale/iframe/ColumnMenu").ReactColumnMenu;
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;

const store = reduxUtils.createDtaleStore();
buildInnerHTML({ settings: "", iframe: "True" }, store);
4 changes: 2 additions & 2 deletions static/__tests__/iframe/DataViewer-within-iframe-test.jsx
Original file line number Diff line number Diff line change
@@ -88,8 +88,8 @@ describe("DataViewer within iframe tests", () => {
.find("ul li span.font-weight-bold")
.map(s => s.text()),
_.concat(
["Describe", "Filter", "Build Column", "Reshape", "Correlations", "Charts", "Resize", "Heat Map"],
["Instances 1", "Code Export", "About", "Refresh", "Open Popup", "Shutdown"]
["Describe", "Filter", "Build Column", "Reshape", "Correlations", "Charts", "Heat Map", "Highlight Dtypes"],
["Instances 1", "Code Export", "Export", "Resize", "About", "Refresh", "Open Popup", "Shutdown"]
),
"Should render default iframe menu options"
);
135 changes: 106 additions & 29 deletions static/__tests__/popups/ColumnAnalysis-test.jsx
Original file line number Diff line number Diff line change
@@ -3,8 +3,10 @@ import qs from "querystring";
import { mount } from "enzyme";
import _ from "lodash";
import React from "react";
import Select from "react-select";

import { RemovableError } from "../../RemovableError";
import { ColumnAnalysisFilters } from "../../popups/analysis/ColumnAnalysisFilters";
import mockPopsicle from "../MockPopsicle";
import * as t from "../jest-assertions";
import { buildInnerHTML, withGlobalJquery } from "../test-utils";
@@ -13,6 +15,14 @@ const ANALYSIS_DATA = {
desc: { count: 20 },
chart_type: "histogram",
dtype: "float64",
cols: [
{ name: "intCol", dtype: "int64" },
{ name: "bar", dtype: "float64" },
{ name: "strCol", dtype: "string" },
{ name: "dateCol", dtype: "datetime" },
{ name: "baz", dtype: "float64" },
],
query: null,
data: [6, 13, 13, 30, 34, 57, 84, 135, 141, 159, 170, 158, 126, 94, 70, 49, 19, 7, 9, 4],
labels: [
"-3.0",
@@ -52,33 +62,53 @@ const props = {

describe("ColumnAnalysis tests", () => {
beforeAll(() => {
const urlParams = qs.stringify({
bins: 20,
query: props.chartData.query,
col: props.chartData.selectedCol,
});
const mockBuildLibs = withGlobalJquery(() =>
mockPopsicle.mock(url => {
if (_.startsWith(url, "/dtale/column-analysis")) {
const params = qs.parse(url.split("?")[1]);
if (params.query === "null") {
const ordinal = ANALYSIS_DATA.data;
const count = ANALYSIS_DATA.data;
if (params.col === "null") {
return null;
}
if (params.query === "error") {
if (params.col === "error") {
return { error: "column analysis error" };
}
if (params.col === "intCol") {
return _.assignIn({}, ANALYSIS_DATA, { dtype: "int64" });
if (params.type === "value_counts") {
return _.assignIn({}, ANALYSIS_DATA, {
chart_type: "value_counts",
ordinal,
});
}
return _.assignIn({}, ANALYSIS_DATA, {
dtype: "int64",
chart_type: "histogram",
});
}
if (params.col === "dateCol") {
return _.assignIn({}, ANALYSIS_DATA, { dtype: "datetime" });
return _.assignIn({}, ANALYSIS_DATA, {
dtype: "datetime",
chart_type: "value_counts",
ordinal,
});
}
if (params.col === "strCol") {
return _.assignIn({}, ANALYSIS_DATA, { dtype: "string" });
return _.assignIn({}, ANALYSIS_DATA, {
dtype: "string",
chart_type: "value_counts",
ordinal,
});
}
if (_.includes(["bar", "baz"], params.col)) {
if (params.type === "categories") {
return _.assignIn({}, ANALYSIS_DATA, {
chart_type: "categories",
count,
});
}
return ANALYSIS_DATA;
}
}
if (_.startsWith(url, "/dtale/column-analysis/1?" + urlParams)) {
return ANALYSIS_DATA;
}
return {};
})
@@ -104,7 +134,7 @@ describe("ColumnAnalysis tests", () => {
});

test("ColumnAnalysis rendering float data", done => {
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;
buildInnerHTML();
const result = mount(<ColumnAnalysis {...props} />, {
attachTo: document.getElementById("content"),
@@ -129,29 +159,59 @@ describe("ColumnAnalysis tests", () => {
result.find("input").simulate("keyPress", { key: "Enter" });
result.find("input").simulate("change", { target: { value: 50 } });
result.find("input").simulate("keyPress", { key: "Enter" });

setTimeout(() => {
result.update();
t.ok(chart.destroyed, "should have destroyed old chart");
t.equal(result.state("bins"), 50, "should update bins");

t.equal(result.find(ColumnAnalysisFilters).instance().state.bins, 50, "should update bins");
result.setProps({
chartData: _.assignIn(props.chartData, { col: "baz" }),
chartData: _.assignIn(props.chartData, { selectedCol: "baz" }),
});
t.equal(result.props().chartData.col, "baz", "should update column");
t.equal(result.props().chartData.selectedCol, "baz", "should update column");

chart = result.state("chart");
result.setProps({
chartData: _.assignIn(props.chartData, { visible: false }),
});
t.deepEqual(chart, result.state("chart"), "should not have destroyed old chart");
done();
result.setProps({
chartData: _.assignIn(props.chartData, { visible: true }),
});
result.update();
result
.find(ColumnAnalysisFilters)
.find("button")
.at(1)
.simulate("click");
result.update();
result
.find(ColumnAnalysisFilters)
.find(Select)
.first()
.instance()
.onChange({ value: "col1" });
setTimeout(() => {
result.update();
result
.find(ColumnAnalysisFilters)
.find("input")
.first()
.simulate("change", { target: { value: 50 } });
result
.find(ColumnAnalysisFilters)
.find("input")
.first()
.simulate("keyPress", { key: "Enter" });
setTimeout(() => {
result.update();
done();
}, 200);
}, 200);
}, 200);
}, 200);
});

test("ColumnAnalysis rendering int data", done => {
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;
buildInnerHTML();
const currProps = _.assignIn({}, props);
currProps.chartData.selectedCol = "intCol";
@@ -161,12 +221,21 @@ describe("ColumnAnalysis tests", () => {

setTimeout(() => {
result.update();
done();
result.update();
result
.find(ColumnAnalysisFilters)
.find("button")
.at(1)
.simulate("click");
setTimeout(() => {
result.update();
done();
}, 200);
}, 200);
});

test("ColumnAnalysis rendering string data", done => {
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;
buildInnerHTML();
const currProps = _.assignIn({}, props);
currProps.chartData.selectedCol = "strCol";
@@ -181,7 +250,7 @@ describe("ColumnAnalysis tests", () => {
});

test("ColumnAnalysis rendering date data", done => {
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;
buildInnerHTML();
const currProps = _.assignIn({}, props);
currProps.chartData.selectedCol = "dateCol";
@@ -191,14 +260,22 @@ describe("ColumnAnalysis tests", () => {

setTimeout(() => {
result.update();
done();
const ordinalInputs = result.find(Select);
ordinalInputs
.first()
.instance()
.onChange({ value: "col1" });
setTimeout(() => {
result.update();
done();
}, 200);
}, 200);
});

test("ColumnAnalysis missing data", done => {
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;
const currProps = _.clone(props);
currProps.chartData.query = "null";
currProps.chartData.selectedCol = "null";
const result = mount(<ColumnAnalysis {...currProps} />);
setTimeout(() => {
result.update();
@@ -208,9 +285,9 @@ describe("ColumnAnalysis tests", () => {
});

test("ColumnAnalysis error", done => {
const ColumnAnalysis = require("../../popups/ColumnAnalysis").ReactColumnAnalysis;
const ColumnAnalysis = require("../../popups/analysis/ColumnAnalysis").ReactColumnAnalysis;
const currProps = _.clone(props);
currProps.chartData.query = "error";
currProps.chartData.selectedCol = "error";
const result = mount(<ColumnAnalysis {...currProps} />);
setTimeout(() => {
result.update();
4 changes: 2 additions & 2 deletions static/__tests__/popups/PopupChart-test.jsx
Original file line number Diff line number Diff line change
@@ -2,8 +2,8 @@ import { shallow } from "enzyme";
import _ from "lodash";
import React from "react";

import { ReactColumnAnalysis } from "../../popups/ColumnAnalysis";
import { ReactCorrelations } from "../../popups/Correlations";
import { ReactColumnAnalysis } from "../../popups/analysis/ColumnAnalysis";
import { ReactCharts } from "../../popups/charts/Charts";
import mockPopsicle from "../MockPopsicle";
import * as t from "../jest-assertions";
@@ -40,7 +40,7 @@ describe("Popup tests", () => {
);

jest.mock("popsicle", () => mockBuildLibs);
jest.mock("../../popups/ColumnAnalysis", () => ({
jest.mock("../../popups/analysis/ColumnAnalysis", () => ({
ColumnAnalysis: MockReactColumnAnalysis,
}));
jest.mock("../../popups/Correlations", () => ({
8 changes: 6 additions & 2 deletions static/dtale/DataViewer.jsx
Original file line number Diff line number Diff line change
@@ -83,7 +83,7 @@ class ReactDataViewer extends React.Component {
}

getData(ids, refresh = false) {
const { loading, loadQueue, heatMapMode } = this.state;
const { loading, loadQueue, heatMapMode, dtypeHighlighting } = this.state;
const data = this.state.data || {};
if (loading) {
this.setState({ loadQueue: _.concat(loadQueue, [ids]) });
@@ -134,6 +134,7 @@ class ReactDataViewer extends React.Component {
traceback: null,
loading: false,
heatMapMode,
dtypeHighlighting,
};
const { columns } = this.state;
if (_.isEmpty(columns)) {
@@ -190,7 +191,10 @@ class ReactDataViewer extends React.Component {
value = rec.view;
valueStyle = _.get(rec, "style", {});
if (this.state.heatMapMode) {
valueStyle = _.assignIn({ background: gu.heatMapBackground(rec, colCfg) }, valueStyle);
valueStyle = _.assignIn(gu.heatMapBackground(rec, colCfg), valueStyle);
}
if (this.state.dtypeHighlighting) {
valueStyle = _.assignIn(gu.dtypeHighlighting(colCfg), valueStyle);
}
if (_.includes(["string", "date"], gu.findColType(colCfg.dtype)) && rec.raw !== rec.view) {
divProps.title = rec.raw;
61 changes: 54 additions & 7 deletions static/dtale/DataViewerMenu.jsx
Original file line number Diff line number Diff line change
@@ -24,7 +24,18 @@ class ReactDataViewerMenu extends React.Component {
this.props.propagateState({
columns: _.map(this.props.columns, c => _.assignIn({}, c)),
});
const toggleHeatMap = () => this.props.propagateState({ heatMapMode: !this.props.heatMapMode });
const toggleHeatMap = () =>
this.props.propagateState({
heatMapMode: !this.props.heatMapMode,
dtypeHighlighting: false,
});
const toggleDtypeHighlighting = () =>
this.props.propagateState({
dtypeHighlighting: !this.props.dtypeHighlighting,
heatMapMode: false,
});
const exportFile = tsv => () =>
window.open(`/dtale/data-export/${dataId}?tsv=${tsv}&_id=${new Date().getTime()}`, "_blank");
return (
<div
className="column-toggle__dropdown"
@@ -84,17 +95,21 @@ class ReactDataViewerMenu extends React.Component {
</li>
<li>
<span className="toggler-action">
<button className="btn btn-plain" onClick={resize}>
<i className="fa fa-expand ml-2 mr-4" />
<span className="font-weight-bold">Resize</span>
<button className="btn btn-plain" onClick={toggleHeatMap}>
<i className={`fa fa-${this.props.heatMapMode ? "fire-extinguisher" : "fire-alt"} ml-2 mr-4`} />
<span className={`font-weight-bold${this.props.heatMapMode ? " flames" : ""}`}>Heat Map</span>
</button>
</span>
</li>
<li>
<span className="toggler-action">
<button className="btn btn-plain" onClick={toggleHeatMap}>
<i className={`fa fa-${this.props.heatMapMode ? "fire-extinguisher" : "fire-alt"} ml-2 mr-4`} />
<span className={`font-weight-bold${this.props.heatMapMode ? " flames" : ""}`}>Heat Map</span>
<button className="btn btn-plain" onClick={toggleDtypeHighlighting}>
<div style={{ display: "inherit" }}>
<div className={`dtype-highlighting${this.props.dtypeHighlighting ? " spin" : ""}`} />
<span className="font-weight-bold" style={{ paddingLeft: ".4em" }}>
Highlight Dtypes
</span>
</div>
</button>
</span>
</li>
@@ -117,6 +132,37 @@ class ReactDataViewerMenu extends React.Component {
</button>
</span>
</li>
<li>
<span className="toggler-action">
<i className="far fa-file" />
</span>
<span className="font-weight-bold pl-2">Export</span>
<div className="btn-group compact m-auto font-weight-bold column-sorting">
{_.map(
[
["CSV", "false"],
["TSV", "true"],
],
([label, tsv]) => (
<button
key={label}
style={{ color: "#565b68" }}
className="btn btn-primary font-weight-bold"
onClick={exportFile(tsv)}>
{label}
</button>
)
)}
</div>
</li>
<li>
<span className="toggler-action">
<button className="btn btn-plain" onClick={resize}>
<i className="fa fa-expand ml-2 mr-4" />
<span className="font-weight-bold">Resize</span>
</button>
</span>
</li>
<li>
<span className="toggler-action">
<button
@@ -177,6 +223,7 @@ ReactDataViewerMenu.propTypes = {
propagateState: PropTypes.func,
openChart: PropTypes.func,
heatMapMode: PropTypes.bool,
dtypeHighlighting: PropTypes.bool,
hideShutdown: PropTypes.bool,
dataId: PropTypes.string.isRequired,
};
30 changes: 28 additions & 2 deletions static/dtale/gridUtils.jsx
Original file line number Diff line number Diff line change
@@ -175,10 +175,34 @@ const heatMap = chroma.scale(["red", "yellow", "green"]).domain([0, 0.5, 1]);

function heatMapBackground({ raw, view }, { min, max }) {
if (view === "") {
return 0;
return {};
}
const factor = min * -1;
return heatMap((raw + factor) / (max + factor));
return { background: heatMap((raw + factor) / (max + factor)) };
}

function dtypeHighlighting({ name, dtype }) {
if (name === IDX) {
return {};
}
const lowerDtype = (dtype || "").toLowerCase();
const colType = findColType(lowerDtype);
if (_.startsWith(lowerDtype, "category")) {
return { background: "#E1BEE7" };
} else if (_.startsWith(lowerDtype, "timedelta")) {
return { background: "#FFCC80" };
} else if (colType === "float") {
return { background: "#B2DFDB" };
} else if (colType === "int") {
return { background: "#BBDEFB" };
} else if (colType === "date") {
return { background: "#F8BBD0" };
} else if (colType === "string") {
return {};
} else if (_.startsWith(lowerDtype, "bool")) {
return { background: "#FFF59D" };
}
return {};
}

const SORT_PROPS = [
@@ -226,6 +250,7 @@ function buildState(props) {
formattingOpen: false,
triggerResize: false,
heatMapMode: false,
dtypeHighlighting: false,
};
}

@@ -252,6 +277,7 @@ export {
ROW_HEIGHT,
HEADER_HEIGHT,
heatMapBackground,
dtypeHighlighting,
SORT_PROPS,
buildToggleId,
buildState,
47 changes: 0 additions & 47 deletions static/dtale/iframe/ColumnMenu.css
Original file line number Diff line number Diff line change
@@ -1,47 +0,0 @@
.btn-group.column-sorting .btn.btn-primary {
border: solid 1px #a7b3b7;
-webkit-box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.2);
box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.2);
background: -webkit-gradient(linear, left top, left bottom, color-stop(80%, white), to(#ebedee));
background: linear-gradient(to bottom, white 80%, #ebedee);
color: #404040;
}

.btn-group.column-sorting .btn.btn-primary:enabled:hover,
.btn-group.column-sorting .btn.btn-primary:enabled:focus {
border: solid 1px #99a7ac;
-webkit-box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.3);
box-shadow: 0 1px 1px 0 rgba(112, 130, 136, 0.3);
}

.btn-group.column-sorting .btn.btn-primary:enabled:active {
border: solid 1px #99a7ac;
-webkit-box-shadow: 0 0 1px 0 rgba(112, 130, 136, 0.3), 0 0 1px 0 #cfd4d7 inset;
box-shadow: 0 0 1px 0 rgba(112, 130, 136, 0.3), 0 0 1px 0 #cfd4d7 inset;
background: -webkit-gradient(linear, left top, left bottom, from(#ebedee), to(white));
background: linear-gradient(to bottom, #ebedee, white);
}

.btn-group.column-sorting .btn.btn-primary.active {
border: solid 1px #88989e;
-webkit-box-shadow: 0 0 3px 0 #88989e inset;
box-shadow: 0 0 3px 0 #88989e inset;
background: #a7b3b7;
color: white;
}

.btn-group.column-sorting .btn.btn-primary.active:enabled:hover {
border: solid 1px #88989e;
-webkit-box-shadow: 0 0 3px 0 #88989e inset;
box-shadow: 0 0 3px 0 #88989e inset;
background: #a7b3b7;
cursor: default;
}

.btn-group.column-sorting .btn.btn-primary.active:enabled:active,
.btn-group.column-sorting .btn.btn-primary.active:enabled:focus {
border: solid 1px #88989e;
-webkit-box-shadow: 0 0 3px 0 #88989e inset;
box-shadow: 0 0 3px 0 #88989e inset;
background: #a7b3b7;
}
54 changes: 37 additions & 17 deletions static/filters/ColumnFilter.jsx
Original file line number Diff line number Diff line change
@@ -31,7 +31,13 @@ function getStyles() {
function buildState({ columns, selectedCol }) {
const colCfg = _.find(columns, { name: selectedCol }) || {};
const colType = findColType(colCfg.dtype);
return { colType, hasMissing: false, missing: false, loadingState: true };
return {
colType,
dtype: colCfg.dtype,
hasMissing: false,
missing: false,
loadingState: true,
};
}

class ColumnFilter extends React.Component {
@@ -46,7 +52,8 @@ class ColumnFilter extends React.Component {
fetchData(state) {
fetchJson(`/dtale/column-filter-data/${this.props.dataId}/${this.props.selectedCol}`, data => {
if (data.success) {
this.setState(_.assignIn(state || {}, { loadingState: false }, data));
const missing = _.get(this.props.columnFilters, [this.props.selectedCol, "missing"], false);
this.setState(_.assignIn(state || {}, { loadingState: false, missing }, data));
}
});
}
@@ -75,14 +82,14 @@ class ColumnFilter extends React.Component {
);
}

renderMissingToggle() {
renderMissingToggle(showIcon) {
const { hasMissing, missing, colType } = this.state;
if (hasMissing) {
const toggleMissing = () =>
this.updateState(_.assignIn({}, this.state.cfg, { type: colType, missing: !missing }));
return (
<li key={1}>
<span className="toggler-action" />
<span className="toggler-action">{showIcon && <i className="fa fa-filter" />}</span>
<div className="m-auto">
<div className="column-filter m-2">
<span className="font-weight-bold pr-3">Show Only Missing</span>
@@ -114,9 +121,12 @@ class ColumnFilter extends React.Component {
let markup = null;
switch (colType) {
case "string":
case "unknown":
markup = <StringFilter {..._.assignIn({}, this.props, this.state)} updateState={this.updateState} />;
case "unknown": {
if (!_.startsWith(this.state.dtype, "timedelta")) {
markup = <StringFilter {..._.assignIn({}, this.props, this.state)} updateState={this.updateState} />;
}
break;
}
case "date":
markup = <DateFilter {..._.assignIn({}, this.props, this.state)} updateState={this.updateState} />;
break;
@@ -128,22 +138,32 @@ class ColumnFilter extends React.Component {
markup = null;
break;
}
return [
<li key={0}>
<span className="toggler-action">
<i className="fa fa-filter" />
</span>
<div className="m-auto">
<div className="column-filter m-2">{markup}</div>
</div>
</li>,
this.renderMissingToggle(),
];
let missingToggle = null;
if (_.isNull(markup)) {
if (!this.state.hasMissing) {
return null;
}
missingToggle = this.renderMissingToggle(true);
} else {
markup = (
<li key={0}>
<span className="toggler-action">
<i className="fa fa-filter" />
</span>
<div className="m-auto">
<div className="column-filter m-2">{markup}</div>
</div>
</li>
);
missingToggle = this.renderMissingToggle(false);
}
return [markup, missingToggle];
}
}
ColumnFilter.displayName = "ColumnFilter";
ColumnFilter.propTypes = {
columns: PropTypes.array,
columnFilters: PropTypes.object,
selectedCol: PropTypes.string,
propagateState: PropTypes.func,
dataId: PropTypes.string.isRequired,
2 changes: 1 addition & 1 deletion static/filters/DateFilter.jsx
Original file line number Diff line number Diff line change
@@ -53,7 +53,7 @@ class DateFilter extends React.Component {
return [
<DateInput
key={0}
value={start}
value={_.isNull(start) ? null : new Date(moment(start))}
onChange={date => this.updateState("start", date)}
inputProps={{ inputRef: c => (this.startInput = c) }}
{...inputProps}
2 changes: 1 addition & 1 deletion static/main.jsx
Original file line number Diff line number Diff line change
@@ -8,11 +8,11 @@ import "./adapter-for-react-16";
import { DataViewer } from "./dtale/DataViewer";
import { CodeExport } from "./popups/CodeExport";
import { CodePopup } from "./popups/CodePopup";
import { ReactColumnAnalysis as ColumnAnalysis } from "./popups/ColumnAnalysis";
import { ReactCorrelations as Correlations } from "./popups/Correlations";
import { ReactDescribe as Describe } from "./popups/Describe";
import { ReactFilter as Filter } from "./popups/Filter";
import Instances from "./popups/Instances";
import { ReactColumnAnalysis as ColumnAnalysis } from "./popups/analysis/ColumnAnalysis";
import { ReactCharts as Charts } from "./popups/charts/Charts";
import { ReactCreateColumn as CreateColumn } from "./popups/create/CreateColumn";
import { ReactReshape as Reshape } from "./popups/reshape/Reshape";
276 changes: 0 additions & 276 deletions static/popups/ColumnAnalysis.jsx

This file was deleted.

2 changes: 1 addition & 1 deletion static/popups/Popup.jsx
Original file line number Diff line number Diff line change
@@ -8,11 +8,11 @@ import ConditionalRender from "../ConditionalRender";
import { closeChart } from "../actions/charts";
import About from "./About";
import { CodeExport } from "./CodeExport";
import { ColumnAnalysis } from "./ColumnAnalysis";
import { Correlations } from "./Correlations";
import { Describe } from "./Describe";
import { Filter } from "./Filter";
import Instances from "./Instances";
import { ColumnAnalysis } from "./analysis/ColumnAnalysis";
import { Charts } from "./charts/Charts";
import { CreateColumn } from "./create/CreateColumn";
import { Reshape } from "./reshape/Reshape";
11 changes: 11 additions & 0 deletions static/popups/analysis/ColumnAnalysis.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
input.column-analysis-filter {
padding: 0.45rem;
}

.ordinal-dd .Select {
min-width: 10em;
}

.ordinal-dd .is-clearable.Select--single .Select__value-container--has-value > .Select__single-value {
padding-right: 0;
}
133 changes: 133 additions & 0 deletions static/popups/analysis/ColumnAnalysis.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
import qs from "querystring";

import _ from "lodash";
import PropTypes from "prop-types";
import React from "react";
import { connect } from "react-redux";

import { RemovableError } from "../../RemovableError";
import actions from "../../actions/dtale";
import { buildURLParams } from "../../actions/url-utils";
import chartUtils from "../../chartUtils";
import { fetchJson } from "../../fetcher";
import { ColumnAnalysisFilters } from "./ColumnAnalysisFilters";
import { createChart } from "./columnAnalysisUtils";

require("./ColumnAnalysis.css");

const BASE_ANALYSIS_URL = "/dtale/column-analysis";

class ReactColumnAnalysis extends React.Component {
constructor(props) {
super(props);
this.state = { chart: null, type: null, error: null, chartParams: null };
this.buildAnalysis = this.buildAnalysis.bind(this);
}

shouldComponentUpdate(newProps, newState) {
if (!_.isEqual(this.props, newProps)) {
return true;
}
const updateState = ["type", "error", "chartParams"];
if (!_.isEqual(_.pick(this.state, updateState), _.pick(newState, updateState))) {
return true;
}

if (this.state.chart != newState.chart) {
// Don't re-render if we've only changed the chart.
return false;
}

// Otherwise, use the default react behavior.
return false;
}

componentDidMount() {
this.buildAnalysis();
}

buildAnalysis(chartParams) {
const finalParams = chartParams || this.state.chartParams;
const { selectedCol } = this.props.chartData;
const paramProps = ["selectedCol", "bins", "top", "type", "ordinalCol", "ordinalAgg", "categoryCol", "categoryAgg"];
const params = _.assignIn({}, this.props.chartData, _.pick(finalParams, ["bins", "top"]));
params.type = _.get(finalParams, "type");
if (params.type === "categories" && _.isNull(finalParams.categoryCol)) {
return;
}
const subProps = params.type === "value_counts" ? ["ordinalCol", "ordinalAgg"] : ["categoryCol", "categoryAgg"];
_.forEach(subProps, p => (params[p] = _.get(finalParams, [p, "value"])));
const url = `${BASE_ANALYSIS_URL}/${this.props.dataId}?${qs.stringify(buildURLParams(params, paramProps))}`;
fetchJson(url, fetchedChartData => {
const newState = { error: null, chartParams: finalParams };
if (_.get(fetchedChartData, "error")) {
newState.error = <RemovableError {...fetchedChartData} />;
}
newState.code = _.get(fetchedChartData, "code", "");
newState.dtype = _.get(fetchedChartData, "dtype", "");
newState.type = _.get(fetchedChartData, "chart_type", "histogram");
newState.query = _.get(fetchedChartData, "query");
newState.cols = _.get(fetchedChartData, "cols", []);
const builder = ctx => {
if (!_.get(fetchedChartData, "data", []).length) {
return null;
}
return createChart(ctx, fetchedChartData, _.assignIn(finalParams, { selectedCol, type: newState.type }));
};
newState.chart = chartUtils.chartWrapper("columnAnalysisChart", this.state.chart, builder);
this.setState(newState);
});
}

render() {
let description = null;
if (actions.isPopup()) {
description = (
<div key="description" className="modal-header">
<h4 className="modal-title">
<i className="ico-equalizer" />
{` ${this.state.type === "histogram" ? "Histogram" : "Value Counts"} for `}
<strong>{_.get(this.props, "chartData.selectedCol")}</strong>
{this.state.query && <small>{this.state.query}</small>}
<div id="describe" />
</h4>
</div>
);
}
let filters = null;
if (this.state.type) {
filters = (
<div key="inputs" className="modal-body modal-form">
<ColumnAnalysisFilters
{..._.pick(this.state, ["type", "cols", "dtype", "code"])}
chartType={this.state.type}
buildChart={this.buildAnalysis}
/>
</div>
);
}
return [
description,
filters,
<div key="body" className="modal-body">
{this.state.error || null}
<canvas id="columnAnalysisChart" height={this.props.height} />
</div>,
];
}
}
ReactColumnAnalysis.displayName = "ColumnAnalysis";
ReactColumnAnalysis.propTypes = {
dataId: PropTypes.string.isRequired,
chartData: PropTypes.shape({
visible: PropTypes.bool.isRequired,
selectedCol: PropTypes.string,
query: PropTypes.string,
}),
height: PropTypes.number,
};
ReactColumnAnalysis.defaultProps = { height: 400 };

const ReduxColumnAnalysis = connect(state => _.pick(state, ["dataId", "chartData"]))(ReactColumnAnalysis);

export { ReactColumnAnalysis, ReduxColumnAnalysis as ColumnAnalysis };
278 changes: 278 additions & 0 deletions static/popups/analysis/ColumnAnalysisFilters.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,278 @@
import _ from "lodash";
import PropTypes from "prop-types";
import React from "react";
import Select, { createFilter } from "react-select";

import { findColType } from "../../dtale/gridUtils";
import { renderCodePopupAnchor } from "../CodePopup";
import { AGGREGATION_OPTS } from "../charts/Aggregations";

function createSelect(selectProps, labelProp = "value") {
return (
<Select
className="Select is-clearable is-searchable Select--single"
classNamePrefix="Select"
getOptionLabel={_.property(labelProp)}
getOptionValue={_.property("value")}
filterOption={createFilter({ ignoreAccents: false })}
{...selectProps}
/>
);
}
class ColumnAnalysisFilters extends React.Component {
constructor(props) {
super(props);
this.state = {
type: props.type,
bins: "20",
top: "100",
ordinalCol: null,
categoryCol: null,
};
this.state.ordinalAgg = _.find(AGGREGATION_OPTS, { value: "sum" });
this.state.categoryAgg = _.find(AGGREGATION_OPTS, { value: "mean" });
this.buildChart = this.buildChart.bind(this);
this.buildChartTypeToggle = this.buildChartTypeToggle.bind(this);
this.buildFilter = this.buildFilter.bind(this);
this.buildOrdinalInputs = this.buildOrdinalInputs.bind(this);
this.buildCategoryInputs = this.buildCategoryInputs.bind(this);
}

shouldComponentUpdate(newProps, newState) {
const props = ["cols", "dtype", "code"];
if (!_.isEqual(_.pick(this.props, props), _.pick(newProps, props))) {
return true;
}
return !_.isEqual(this.state, newState);
}

buildChartTypeToggle() {
const colType = findColType(this.props.dtype);
const options = [["Histogram", "histogram"]];
if (colType === "float") {
options.push(["Categories", "categories"]);
} else {
options.push(["Value Counts", "value_counts"]);
}
return (
<div className="col-auto btn-group">
{_.map(options, ([label, value]) => {
const buttonProps = { className: "btn" };
if (value === this.state.type) {
buttonProps.className += " btn-primary active";
} else {
buttonProps.className += " btn-primary inactive";
buttonProps.onClick = () => this.setState({ type: value }, this.buildChart);
}
return (
<button key={value} {...buttonProps}>
{label}
</button>
);
})}
</div>
);
}

buildFilter(prop) {
const colType = findColType(this.props.dtype);
const updateFilter = e => {
if (e.key === "Enter") {
if (this.state[prop] && parseInt(this.state[prop])) {
this.buildChart();
}
e.preventDefault();
}
};
return [
<div key={0} className={`col-auto text-center pr-4 ${colType === "int" ? "pl-0" : ""}`}>
<div>
<b>{_.capitalize(prop)}</b>
</div>
<div style={{ marginTop: "-.5em" }}>
<small>(Please edit)</small>
</div>
</div>,
<div key={1} style={{ width: "3em" }} data-tip="Press ENTER to submit" className="mb-auto mt-auto">
<input
type="text"
className="form-control text-center column-analysis-filter"
value={this.state[prop]}
onChange={e => this.setState({ [prop]: e.target.value })}
onKeyPress={updateFilter}
/>
</div>,
];
}

buildChart() {
this.props.buildChart(this.state);
}

buildOrdinalInputs() {
const updateOrdinal = (prop, val) => {
const currState = _.assignIn({}, _.pick(this.state, ["ordinalCol", "ordinalAgg"]), { [prop]: val });
this.setState(currState, () => {
if (currState.ordinalCol && currState.ordinalAgg) {
this.buildChart();
}
});
};
const { cols } = this.props;
let colOpts = _.filter(cols, c => _.includes(["float", "int"], findColType(c.dtype)));
colOpts = _.sortBy(
_.map(colOpts, c => ({ value: c.name })),
c => _.toLower(c.value)
);
return [
<div key={0} className="col-auto text-center pr-4">
<div>
<b>Ordinal</b>
</div>
<div style={{ marginTop: "-.5em" }}>
<small>(Choose Col/Agg)</small>
</div>
</div>,
<div key={1} className="col-auto pl-0 mr-3 ordinal-dd">
{createSelect({
value: this.state.ordinalCol,
options: colOpts,
onChange: v => updateOrdinal("ordinalCol", v),
noOptionsText: () => "No columns found",
isClearable: true,
})}
</div>,
<div key={2} className="col-auto pl-0 mr-3 ordinal-dd">
{createSelect(
{
value: this.state.ordinalAgg,
options: AGGREGATION_OPTS,
onChange: v => updateOrdinal("ordinalAgg", v),
},
"label"
)}
</div>,
];
}

buildCategoryInputs() {
const updateCategory = (prop, val) => {
const currState = _.assignIn({}, _.pick(this.state, ["categoryCol", "categoryAgg"]), { [prop]: val });
this.setState(currState, () => {
if (currState.categoryCol && currState.categoryAgg) {
this.buildChart();
}
});
};
const { cols } = this.props;
let colOpts = _.reject(cols, c => findColType(c.dtype) === "float");
colOpts = _.sortBy(
_.map(colOpts, c => ({ value: c.name })),
c => _.toLower(c.value)
);
return [
<div key={0} className="col-auto text-center pr-4">
<div>
<b>Category Breakdown</b>
</div>
<div style={{ marginTop: "-.5em" }}>
<small>(Choose Col/Agg)</small>
</div>
</div>,
<div key={1} className="col-auto pl-0 mr-3 ordinal-dd">
{createSelect({
value: this.state.categoryCol,
options: colOpts,
onChange: v => updateCategory("categoryCol", v),
noOptionsText: () => "No columns found",
isClearable: true,
})}
</div>,
<div key={2} className="col-auto pl-0 mr-3 ordinal-dd">
{createSelect(
{
value: this.state.categoryAgg,
options: AGGREGATION_OPTS,
onChange: v => updateCategory("categoryAgg", v),
},
"label"
)}
</div>,
];
}

render() {
if (_.isNull(this.props.type)) {
return null;
}
const { code, dtype } = this.props;
const colType = findColType(dtype);
const title = this.state.type === "histogram" ? "Histogram" : "Value Counts";
let filterMarkup = null;
if ("int" === colType) {
// int -> Value Counts or Histogram
if (this.state.type === "histogram") {
filterMarkup = (
<div className="col row">
{this.buildChartTypeToggle()}
{this.buildFilter("bins")}
</div>
);
} else {
filterMarkup = (
<div className="col row">
{this.buildChartTypeToggle()}
{this.buildFilter("top")}
{this.buildOrdinalInputs()}
</div>
);
}
} else if ("float" === colType) {
// floats -> Histogram or Categories
if (this.state.type === "histogram") {
filterMarkup = (
<div className="col row">
{this.buildChartTypeToggle()}
{this.buildFilter("bins")}
</div>
);
} else {
filterMarkup = (
<div className="col row">
{this.buildChartTypeToggle()}
{this.buildFilter("top")}
{this.buildCategoryInputs()}
</div>
);
}
} else {
// date, string, bool -> Value Counts
filterMarkup = (
<div className="col row">
<h4 className="pl-5 pt-3 modal-title font-weight-bold">{title}</h4>
{this.buildFilter("top")}
{this.buildOrdinalInputs()}
</div>
);
}
return (
<div className="form-group row small-gutters mb-0">
{filterMarkup}
<div className="col-auto">
<div>{renderCodePopupAnchor(code, title)}</div>
</div>
</div>
);
}
}
ColumnAnalysisFilters.displayName = "ColumnAnalysisFilters";
ColumnAnalysisFilters.propTypes = {
selectedCol: PropTypes.string,
cols: PropTypes.array,
dtype: PropTypes.string,
code: PropTypes.string,
type: PropTypes.string,
buildChart: PropTypes.func,
};

export { ColumnAnalysisFilters };
Loading

0 comments on commit 0e073eb

Please sign in to comment.