Web console: use new sampler features #14017
Conversation
| // const ingestionSpec: IngestionSpec = { | ||
| // type: 'index_parallel', | ||
| // spec: { |
There was a problem hiding this comment.
these commented blocks supposed to be here?
| { type: 'json', name: 'spend', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { type: 'string', name: 'id', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { type: 'json', name: 'tags', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { type: 'json', name: 'nums', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| ], | ||
| physicalDimensions: [ | ||
| { type: 'json', name: 'user', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { | ||
| type: 'json', | ||
| name: 'followers', | ||
| multiValueHandling: 'SORTED_ARRAY', | ||
| createBitmapIndex: true, | ||
| }, | ||
| { type: 'json', name: 'spend', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { type: 'json', name: 'id', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { type: 'json', name: 'tags', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, | ||
| { type: 'json', name: 'nums', multiValueHandling: 'SORTED_ARRAY', createBitmapIndex: true }, |
There was a problem hiding this comment.
i suppose I changed this underneath you in #14014, but it might be nice to update this at some point (also I have a bug to fix here since right now the sampler will show the new 'auto' in the physical schema but still have 'json' in the logical schema, so i think its ok to hold off on updating this)
| export function guessDimensionsFromSampleResponse(sampleResponse: SampleResponse): DimensionSpec[] { | ||
| const { logicalDimensions, physicalDimensions, data } = sampleResponse; | ||
| return logicalDimensions.map(d => { | ||
| // Boolean column are currently reported as "long" so let's turn them into "string" |
There was a problem hiding this comment.
im still looking into this, since technically looking at stuff it looks like it should be dependent on the value of druid.expressions.useStrictBooleans. Additionally, 'long' really probably is better when using 'auto', since longs do have indexes in this mode, so im a bit conflicted about this staying like this long term, but i think its fine at least until we make 'auto' the default schemaless (or add indexes to classic 'long' schema)
There was a problem hiding this comment.
the reason it is picking string is that if you pick long then it forces the true / false in the input to null (instead of 1 / 0)
This is the UI part that follows, #13653 and #13900
This simplifies the data loader code by using schema discovery and also adds support for the Kafka input format
This PR also:
headerLabelPrefixwithheaderColumnPrefix(that one tripped me up)