Limit visualization editor based on rollup capabilities #23637

jen-huang · 2018-10-01T08:53:27Z

PR also:

Migrates new type and typeMeta field mappings for old indices (using new Index Migration feature)
Disables timefield for rollup index patterns in discover so that the raw rollup documents can be shown using the default search strategy
Restructure index pattern list registry
Disable scripted fields tab for rollup index patterns
Allow count in visualization editor
Misc clean up

Rollup documents in Discover

…aw rollup docs can be viewed

…r selecting initial agg

elasticmachine · 2018-10-01T20:02:54Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

cjcenizal

I got this weird error when creating a visualization:

The request does include max and min aggs. We ran into this problem before didn't we?

[  
  {  
    "index":"test_rollup_index",
    "query":{  
      "size":0,
      "aggregations":{  
        "maxAgg":{  
          "max":{  
            "field":"memory"
          }
        },
        "minAgg":{  
          "min":{  
            "field":"memory"
          }
        }
      },
      "query":{  
        "bool":{  
          "must":[  
            {  
              "match_all":{}
            },
            {  
              "range":{  
                "utc_time":{  
                  "gte":1538418388395,
                  "lte":1538432788395,
                  "format":"epoch_millis"
                }
              }
            }
          ],
          "filter":[],
          "should":[],
          "must_not":[]
        }
      }
    }
  }
]

Here's the job config:

{
  "config": {
    "id": "test rollup job",
    "index_pattern": "log*",
    "rollup_index": "test_rollup_index",
    "cron": "0 * * * * ?",
    "groups": {
      "date_histogram": {
        "interval": "1m",
        "field": "utc_time",
        "time_zone": "UTC"
      },
      "histogram": {
        "interval": 100,
        "fields": [
          "machine.ram",
          "meta.user.lastname",
          "id",
          "bytes",
          "phpmemory",
          "memory"
        ]
      },
      "terms": {
        "fields": [
          "geo.srcdest",
          "id",
          "longValuesWithSpaces.keyword"
        ]
      }
    },
    "metrics": [
      {
        "field": "id",
        "metrics": [
          "min"
        ]
      },
      {
        "field": "meta.user.lastname",
        "metrics": [
          "min"
        ]
      },
      {
        "field": "memory",
        "metrics": [
          "min"
        ]
      },
      {
        "field": "phpmemory",
        "metrics": [
          "min"
        ]
      },
      {
        "field": "machine.ram",
        "metrics": [
          "min"
        ]
      }
    ],
    "timeout": "20s",
    "page_size": 100
  },
  "status": {
    "job_state": "started",
    "current_position": {
      "bytes.histogram": 700,
      "geo.srcdest.terms": "BF:PT",
      "id.histogram": null,
      "id.terms": null,
      "longValuesWithSpaces.keyword.terms": null,
      "machine.ram.histogram": 11811160000,
      "memory.histogram": null,
      "meta.user.lastname.histogram": null,
      "phpmemory.histogram": null,
      "utc_time.date_histogram": 1538433000000
    },
    "upgraded_doc_id": true
  },
  "stats": {
    "pages_processed": 142,
    "documents_processed": 8600,
    "rollups_indexed": 8600,
    "trigger_count": 32
  }
}

cjcenizal · 2018-10-01T22:17:28Z

...plugins/kibana/public/management/sections/indices/index_pattern_list/components/list/list.js

+    ) : null;
+  }
+
+  renderNoDefaultMessage() {


What do I need to do to get this to display? I tried deleting the default index pattern but then it just reassigned another one as default.

have no index patterns defined

cjcenizal · 2018-10-01T22:18:29Z

...e_plugins/kibana/public/management/sections/indices/index_pattern_list/index_pattern_list.js


 import { Header } from './components/header';
 import { List } from './components/list';

-export class IndexPatternList extends Component {
-  static propTypes = {


Does it make sense to keep the propTypes around?

IndexPatternList.propTypes = { indexPatternCreationOptions: PropTypes.array.isRequired, defaultIndex: PropTypes.string, indexPatterns: PropTypes.array.isRequired, };

we could, but the propTypes are enforced by components that this component renders so I don't think it's necessary

cjcenizal · 2018-10-01T22:22:49Z

x-pack/plugins/rollup/public/visualize/editor_config.js

+      },
+      interval: {
+        base: interval,
+        help: `Must be a multiple of rollup configuration interval: ${interval}`


I seem to get an error even when using a valid multiple, though the error doesn't have an explanation:

1m is actually a calendar interval, whereas 10m is fixed 🙂

Bah, of course! I think we'll need some improved error reporting here in the next release.

cjcenizal · 2018-10-01T22:29:53Z

x-pack/plugins/rollup/public/visualize/agg_type_filter.js

@@ -17,6 +17,6 @@ aggTypeFilters.addFilter(
    }
    const aggName = aggType.name;
    const aggs = indexPattern.typeMeta.aggs;
-    return Object.keys(aggs).includes(aggName);
+    return aggName === 'count' || Object.keys(aggs).includes(aggName);


Does this merit a comment since it looks like a special case?

good catch, will add one

jen-huang · 2018-10-01T23:19:14Z

@cjcenizal I was able to replicate your error. In this case, it's the result of the queried histogram field (memory) not having max and min metrics. Visualization queries for max and min to set the auto bounds for visualization. I'll keep investigating to figure out options and best fix.

jen-huang · 2018-10-02T05:54:05Z

@cjcenizal I added a catch for the min/max aggs: https://github.com/elastic/kibana/pull/23637/files#diff-1be9dd2b7946d5364296485200c35927R100

This will allow the visualization to render without having the min/max information. I think a more complete solution would be when the user selects a histogram field in the job wizard, we automatically select the same field in metrics as well with min/max aggs preselected, maybe with a flag saying that this will allow for a better visualization experience. This could get a little tricky if the user deletes the histogram field (do we keep the metrics one around?) and other back and forth cases. I think it would be a good enhancement for phase 2.

elasticmachine · 2018-10-02T06:18:24Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

cjcenizal

I tested the new catch and it works great! Thanks for digging into that. Code LGTM!

cjcenizal · 2018-10-02T16:33:00Z

src/ui/public/agg_types/buckets/histogram.js

+          .catch(() => {
+            toastNotifications.addWarning(`
+              Unable to retrieve max and min values to auto-scale histogram buckets. 
+              This may lead to poor visualization performance.


Is it a performance issue? Unless I'm misunderstanding, it seems like it's more of a problem with the veracity of the visualization's results. Would "This may lead to incorrect visualization results" be more accurate?

it's actually to increase the size of the buckets in case the given minimum interval causes too many buckets, see this comment. when I tested using a rollup field that doesn't have min/max after adding the catch, I get a visualization that looks like this:

if it had min/max, the interval would have been increased to decrease the number of bars. this example loaded fine for me, but I imagine a greater number of bars could be problematic on certain machines. hence performance issue 🙂

Makes sense, thanks!

* Limit vis editor based on rollup capabilities * Migrate new `type` and `typeMeta` field mappings for old indices * Disable timefield for rollup index patterns in discover so that the raw rollup docs can be viewed * Restructure index pattern list registry * Disable scripted fields tab for rollup index patterns * Allow count in list of available vis aggs * Add error when min/max aggs are not available for histograms

jen-huang added 5 commits October 1, 2018 01:46

Limit vis editor based on rollup capabilities

518e010

Migrate new type and typeMeta field mappings for old indices

99ac8ca

Disable timefield for rollup index patterns in discover so that the r…

b791cc7

…aw rollup docs can be viewed

Rename file to .ts to fix typings

42c7500

Restructure index pattern list registry and some cleanup

382d7e0

jen-huang requested a review from cjcenizal October 1, 2018 08:53

jen-huang added 4 commits October 1, 2018 09:21

Disable scripted fields tab for rollup index patterns

7c77b00

Allow count in list of available vis aggs, revert previous changes fo…

0c64cba

…r selecting initial agg

Fix vis agg fields filtering

87612f7

Fix field capabilities info on index pattern fields

311a14d

elastic deleted a comment from elasticmachine Oct 1, 2018

cjcenizal added the :Management label Oct 1, 2018

cjcenizal reviewed Oct 1, 2018

View reviewed changes

jen-huang added 2 commits October 1, 2018 22:45

Add error when min/max aggs are not available for histograms

8a70801

Add comment for returning count as an available agg

dbfb953

cjcenizal approved these changes Oct 2, 2018

View reviewed changes

jen-huang merged commit c765ecb into elastic:feature/rollups Oct 2, 2018

jen-huang deleted the feature/rollups-jen branch October 5, 2018 21:04

cjcenizal added the Feature:Rollups label May 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit visualization editor based on rollup capabilities #23637

Limit visualization editor based on rollup capabilities #23637

jen-huang commented Oct 1, 2018 •

edited by cjcenizal

Loading

elasticmachine commented Oct 1, 2018

cjcenizal left a comment

cjcenizal Oct 1, 2018

jen-huang Oct 1, 2018

cjcenizal Oct 1, 2018

jen-huang Oct 1, 2018

cjcenizal Oct 1, 2018

jen-huang Oct 1, 2018

cjcenizal Oct 2, 2018

cjcenizal Oct 1, 2018

jen-huang Oct 1, 2018

jen-huang commented Oct 1, 2018

jen-huang commented Oct 2, 2018 •

edited

Loading

elasticmachine commented Oct 2, 2018

cjcenizal left a comment

cjcenizal Oct 2, 2018

jen-huang Oct 2, 2018

cjcenizal Oct 2, 2018

Limit visualization editor based on rollup capabilities #23637

Limit visualization editor based on rollup capabilities #23637

Conversation

jen-huang commented Oct 1, 2018 • edited by cjcenizal Loading

Rollup documents in Discover

elasticmachine commented Oct 1, 2018

💔 Build Failed

cjcenizal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jen-huang commented Oct 1, 2018

jen-huang commented Oct 2, 2018 • edited Loading

elasticmachine commented Oct 2, 2018

💔 Build Failed

cjcenizal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jen-huang commented Oct 1, 2018 •

edited by cjcenizal

Loading

jen-huang commented Oct 2, 2018 •

edited

Loading