Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Telemetry] [Schema] remove number type and support all es number types #81774

Merged
merged 6 commits into from
Oct 30, 2020
Merged

[Telemetry] [Schema] remove number type and support all es number types #81774

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
23ae823
remove number type and support all es number types
Bamieh Oct 27, 2020
9807b16
update docs
Bamieh Oct 27, 2020
1d89ce1
fix tests
Bamieh Oct 27, 2020
16ecd53
Merge branch 'master' of github.com:elastic/kibana into telemetry/sch…
Bamieh Oct 27, 2020
0065080
long -> float for averages
Bamieh Oct 29, 2020
467ea47
Merge branch 'master' of github.com:elastic/kibana into telemetry/sch…
Bamieh Oct 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions packages/kbn-telemetry-tools/GUIDELINE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,14 +148,7 @@ usageCollection.makeUsageCollector<Usage>({
Any field property in the schema accepts a `type` field. By default the type is `object` which accepts nested properties under it. Currently we accept the following property types:

```
AllowedSchemaTypes =
| 'keyword'
| 'text'
| 'number'
| 'boolean'
| 'long'
| 'date'
| 'float';
'long', 'integer', 'short', 'byte', 'double', 'float', 'keyword', 'text', 'boolean', 'date'
```


Expand Down
12 changes: 6 additions & 6 deletions packages/kbn-telemetry-tools/src/tools/__fixture__/mock_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@
"my_index_signature_prop": {
"properties": {
"avg": {
"type": "number"
"type": "float"
},
"count": {
"type": "number"
"type": "long"
},
"max": {
"type": "number"
"type": "long"
},
"min": {
"type": "number"
"type": "long"
}
}
},
Expand All @@ -27,7 +27,7 @@
"my_objects": {
"properties": {
"total": {
"type": "number"
"type": "long"
},
"type": {
"type": "boolean"
Expand All @@ -39,7 +39,7 @@
"items": {
"properties": {
"total": {
"type": "number"
"type": "long"
},
"type": {
"type": "boolean"
Expand Down
2 changes: 1 addition & 1 deletion ...elemetry-tools/src/tools/__fixture__/parsed_indexed_interface_with_not_matching_schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ export const parsedIndexedInterfaceWithNoMatchingSchema: ParsedUsageCollection =
value: {
something: {
count_1: {
type: 'number',
type: 'long',
},
},
},
Expand Down
2 changes: 1 addition & 1 deletion ...kbn-telemetry-tools/src/tools/__fixture__/parsed_schema_defined_with_spreads_collector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ export const parsedSchemaDefinedWithSpreadsCollector: ParsedUsageCollection = [
},
my_objects: {
total: {
type: 'number',
type: 'long',
},
type: {
type: 'boolean',
Expand Down
12 changes: 6 additions & 6 deletions packages/kbn-telemetry-tools/src/tools/__fixture__/parsed_working_collector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,21 +34,21 @@ export const parsedWorkingCollector: ParsedUsageCollection = [
},
my_index_signature_prop: {
avg: {
type: 'number',
type: 'float',
},
count: {
type: 'number',
type: 'long',
},
max: {
type: 'number',
type: 'long',
},
min: {
type: 'number',
type: 'long',
},
},
my_objects: {
total: {
type: 'number',
type: 'long',
},
type: {
type: 'boolean',
Expand All @@ -58,7 +58,7 @@ export const parsedWorkingCollector: ParsedUsageCollection = [
type: 'array',
items: {
total: {
type: 'number',
type: 'long',
},
type: { type: 'boolean' },
},
Expand Down
14 changes: 7 additions & 7 deletions packages/kbn-telemetry-tools/src/tools/__snapshots__/extract_collectors.test.ts.snap
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions packages/kbn-telemetry-tools/src/tools/check_collector__integrity.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ describe('checkMatchingMapping', () => {
it('returns diff on mismatching parsedCollections and stored mapping', async () => {
const mockSchema = await parseJsonFile('mock_schema.json');
const malformedParsedCollector = cloneDeep(parsedWorkingCollector);
const fieldMapping = { type: 'number' };
const fieldMapping = { type: 'long' };
malformedParsedCollector[1].schema.value.flat = fieldMapping;

const diffs = checkMatchingMapping([malformedParsedCollector], mockSchema);
Expand All @@ -61,9 +61,9 @@ describe('checkMatchingMapping', () => {
const mockSchema = await parseJsonFile('mock_schema.json');
const malformedParsedCollector = cloneDeep(parsedWorkingCollector);
const collectorName = 'New Collector in town!';
const collectorMapping = { some_usage: { type: 'number' } };
const collectorMapping = { some_usage: { type: 'long' } };
malformedParsedCollector[1].collectorName = collectorName;
malformedParsedCollector[1].schema.value = { some_usage: { type: 'number' } };
malformedParsedCollector[1].schema.value = { some_usage: { type: 'long' } };

const diffs = checkMatchingMapping([malformedParsedCollector], mockSchema);
expect(diffs).toEqual({
Expand Down
18 changes: 8 additions & 10 deletions packages/kbn-telemetry-tools/src/tools/manage_schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,9 @@

import { ParsedUsageCollection } from './ts_parser';

export type AllowedSchemaTypes =
| 'keyword'
| 'text'
| 'number'
| 'boolean'
| 'long'
| 'date'
| 'float';
export type AllowedSchemaNumberTypes = 'long' | 'integer' | 'short' | 'byte' | 'double' | 'float';

export type AllowedSchemaTypes = AllowedSchemaNumberTypes | 'keyword' | 'text' | 'boolean' | 'date';

export function compatibleSchemaTypes(type: AllowedSchemaTypes | 'array') {
switch (type) {
Expand All @@ -36,9 +31,12 @@ export function compatibleSchemaTypes(type: AllowedSchemaTypes | 'array') {
return 'string';
case 'boolean':
return 'boolean';
case 'number':
case 'float':
case 'long':
case 'integer':
case 'short':
case 'byte':
case 'double':
case 'float':
return 'number';
case 'array':
return 'array';
Expand Down
2 changes: 1 addition & 1 deletion src/fixtures/telemetry_collectors/schema_defined_with_spreads_collector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ const someSchema: MakeSchemaFrom<Pick<Usage, 'flat' | 'my_str'>> = {
const someOtherSchema: MakeSchemaFrom<Pick<Usage, 'my_objects'>> = {
my_objects: {
total: {
type: 'number',
type: 'long',
},
type: { type: 'boolean' },
},
Expand Down
12 changes: 6 additions & 6 deletions src/fixtures/telemetry_collectors/working_collector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,25 +85,25 @@ export const myCollector = makeUsageCollector<Usage>({
},
my_objects: {
total: {
type: 'number',
type: 'long',
},
type: { type: 'boolean' },
},
my_array: {
type: 'array',
items: {
total: {
type: 'number',
type: 'long',
},
type: { type: 'boolean' },
},
},
my_str_array: { type: 'array', items: { type: 'keyword' } },
my_index_signature_prop: {
count: { type: 'number' },
avg: { type: 'number' },
max: { type: 'number' },
min: { type: 'number' },
count: { type: 'long' },
avg: { type: 'float' },
max: { type: 'long' },
min: { type: 'long' },
},
},
});
6 changes: 3 additions & 3 deletions src/plugins/data/server/search/collectors/register.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ export async function registerUsageCollector(
isReady: () => true,
fetch: fetchProvider(context.config.legacy.globalConfig$),
schema: {
successCount: { type: 'number' },
errorCount: { type: 'number' },
averageDuration: { type: 'long' },
successCount: { type: 'long' },
errorCount: { type: 'long' },
averageDuration: { type: 'float' },
},
});
usageCollection.registerCollector(collector);
Expand Down
6 changes: 3 additions & 3 deletions src/plugins/telemetry/schema/oss_plugins.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@
"search": {
"properties": {
"successCount": {
"type": "number"
"type": "long"
},
"errorCount": {
"type": "number"
"type": "long"
},
"averageDuration": {
"type": "long"
"type": "float"
}
}
},
Expand Down
6 changes: 3 additions & 3 deletions src/plugins/usage_collection/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ The `schema` field is a proscribed data model assists with detecting changes in
The `AllowedSchemaTypes` is the list of allowed schema types for the usage fields getting reported:

```
'keyword', 'text', 'number', 'boolean', 'long', 'date', 'float'
'long', 'integer', 'short', 'byte', 'double', 'float', 'keyword', 'text', 'boolean', 'date'
```

### Arrays
Expand Down Expand Up @@ -171,7 +171,7 @@ export const myCollector = makeUsageCollector<Usage>({
},
some_obj: {
total: {
type: 'number',
type: 'long',
},
},
some_array: {
Expand All @@ -182,7 +182,7 @@ export const myCollector = makeUsageCollector<Usage>({
type: 'array',
items: {
total: {
type: 'number',
type: 'long',
},
},
},
Expand Down
11 changes: 3 additions & 8 deletions src/plugins/usage_collection/server/collector/collector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,9 @@ import {

export type CollectorFormatForBulkUpload<T, U> = (result: T) => { type: string; payload: U };

export type AllowedSchemaTypes =
| 'keyword'
| 'text'
| 'number'
| 'boolean'
| 'long'
| 'date'
| 'float';
export type AllowedSchemaNumberTypes = 'long' | 'integer' | 'short' | 'byte' | 'double' | 'float';

export type AllowedSchemaTypes = AllowedSchemaNumberTypes | 'keyword' | 'text' | 'boolean' | 'date';

export interface SchemaField {
type: string;
Expand Down
1 change: 1 addition & 0 deletions src/plugins/usage_collection/server/collector/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ export { CollectorSet } from './collector_set';
export {
Collector,
AllowedSchemaTypes,
AllowedSchemaNumberTypes,
SchemaField,
MakeSchemaFrom,
CollectorOptions,
Expand Down
2 changes: 1 addition & 1 deletion x-pack/plugins/security/server/usage_collector/security_usage_collector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ export function registerSecurityUsageCollector({ usageCollection, config, licens
type: 'boolean',
},
authProviderCount: {
type: 'number',
type: 'long',
Comment on lines 61 to +62
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Realistically, we wouldn't ever have enough auth providers in Kibana to warrant a "long". But we don't enforce an actual limit, so it's probably the most appropriate type to use.

},
enabledAuthProviders: {
type: 'array',
Expand Down
2 changes: 1 addition & 1 deletion x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3250,7 +3250,7 @@
"type": "boolean"
},
"authProviderCount": {
"type": "number"
"type": "long"
},
"enabledAuthProviders": {
"type": "array",
Expand Down