Skip to content

Commit

Permalink
Merge pull request #1897 from himanshug/new_sketch_aggregation
Browse files Browse the repository at this point in the history 
complex aggregator based on http://datasketches.github.io
  • Loading branch information
@fjy
fjy committed Nov 12, 2015
2 parents 465cbcf + 7788f7c commit 148153b
Show file tree
Hide file tree
Showing 40 changed files with 9,656 additions and 0 deletions.
140 changes: 140 additions & 0 deletions docs/content/development/datasketches-aggregators.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
---
layout: doc_page
---

## DataSketches aggregator
Druid aggregators based on [datasketches]()http://datasketches.github.io/) library. Note that sketch algorithms are approxiate, see details in the "Accuracy" section of the datasketches doc.
At ingestion time, this aggregator creates the theta sketch objects which get stored in Druid segments. Logically speaking, a theta sketch object can be thought of as a Set data structure. At query time, sketches are read and aggregated(set unioned) together. In the end, by default, you receive the estimate of number of unique entries in the sketch object. Also, You can use post aggregators to do union, intersection or difference on sketch columns in the same row.

### Aggregators

```json
{
"type" : "thetaSketch",
"name" : <output_name>,
"fieldName" : <metric_name>,

//following boolean field is optional. This should only be used at
//indexing time if your input data contains theta sketch objects.
//that would be the case if you use datasketches library outside of Druid,
//say with Pig/Hive, to produce the data that you are ingesting into Druid
"isInputThetaSketch": false

//following field is optional, default = 16384. must be a power of 2.
//Internally, size refers to the maximum number
//of entries sketch object will retain, higher size would mean higher
//accuracy but higher space needed to store those sketches.
//note that after you index with a particular size, druid will persist sketch in segments
//and you will use size greater or equal to that at query time.
//See [theta-size](http://datasketches.github.io/docs/ThetaSize.html) for details.
//In general, We recommend just sticking to default size, which has worked well.
"size": 16384
}
```

### Post Aggregators

#### Sketch Estimator
```json
{ "type" : "thetaSketchEstimate", "name": <output name>, "fieldName" : <the name field value of the thetaSketch aggregator>}
```

#### Sketch Operations
```json
{ "type" : "thetaSketchSetOp", "name": <output name>, "func": <UNION|INTERSECT|NOT>, "fields" : <the name field value of the thetaSketch aggregators>}
```

### Examples

Assuming, you have a dataset containing (timestamp, product, user_id). You want to answer questions like

How many unique users visited product A?
How many unique users visited both product A and product B?

to answer above questions, you would index your data using following aggregator.

```json
{ "type": "thetaSketch", "name": "user_id_sketch", "fieldName": "user_id" }
```

then, sample query for, How many unique users visited product A?
```json
{
"queryType": "groupBy",
"dataSource": "test_datasource",
"granularity": "ALL",
"dimensions": [],
"aggregations": [
{ "type": "thetaSketch", "name": "unique_users", "fieldName": "user_id_sketch" }
],
"filter": { "type": "selector", "dimension": "product", "value": "A" },
"intervals": [ "2014-10-19T00:00:00.000Z/2014-10-22T00:00:00.000Z" ]
}
```

sample query for, How many unique users visited both product A and B?

```json
{
"queryType": "groupBy",
"dataSource": "test_datasource",
"granularity": "ALL",
"dimensions": [],
"filter": {
"type": "or",
"fields": [
{"type": "selector", "dimension": "product", "value": "A"},
{"type": "selector", "dimension": "product", "value": "B"}
]
},
"aggregations": [
{
"type" : "filtered",
"filter" : {
"type" : "selector",
"dimension" : "product",
"value" : "A"
},
"aggregator" : {
"type": "thetaSketch", "name": "A_unique_users", "fieldName": "user_id_sketch"
}
},
{
"type" : "filtered",
"filter" : {
"type" : "selector",
"dimension" : "product",
"value" : "B"
},
"aggregator" : {
"type": "thetaSketch", "name": "B_unique_users", "fieldName": "user_id_sketch"
}
}
],
"postAggregations": [
{
"type": "thetaSketchEstimate",
"name": "final_unique_users",
"field":
{
"type": "thetaSketchSetOp",
"name": "final_unique_users_sketch",
"func": "INTERSECT",
"fields": [
{
"type": "fieldAccess",
"fieldName": "A_unique_users"
},
{
"type": "fieldAccess",
"fieldName": "B_unique_users"
}
]
}
}
],
"intervals": [
"2014-10-19T00:00:00.000Z/2014-10-22T00:00:00.000Z"
]
}
```
1 change: 1 addition & 0 deletions docs/content/toc.textile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ h2. Development
** "Geographic Queries":../development/geo.html
** "Select Query":../development/select-query.html
** "Approximate Histograms and Quantiles":../development/approximate-histograms.html
** "Datasketches based Aggregators":../development/datasketches-aggregators.html
** "Router node":../development/router.html
** "New Kafka Firehose":../development/kafka-simple-consumer-firehose.html

Expand Down
9 changes: 9 additions & 0 deletions extensions/datasketches/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
This module provides druid aggregators based on http://datasketches.github.io/ .

Credits: This module is a result of feedback and work done by following people.

https://github.com/cheddar
https://github.com/himanshug
https://github.com/leerho
https://github.com/will-lauer

119 changes: 119 additions & 0 deletions extensions/datasketches/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to Metamarkets Group Inc. (Metamarkets) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. Metamarkets licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>io.druid.extensions</groupId>
<artifactId>druid-datasketches</artifactId>
<name>druid-datasketches</name>
<description>Druid Aggregators based on datasketches lib http://datasketches.github.io/</description>

<parent>
<groupId>io.druid</groupId>
<artifactId>druid</artifactId>
<version>0.9.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<dependencies>
<dependency>
<groupId>com.yahoo.datasketches</groupId>
<artifactId>sketches-core</artifactId>
<version>0.1.1</version>
</dependency>
<dependency>
<groupId>io.druid</groupId>
<artifactId>druid-api</artifactId>
<version>${druid.api.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.druid</groupId>
<artifactId>druid-processing</artifactId>
<version>${project.parent.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-guava</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-joda</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-smile</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-smile-provider</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>

<!-- Test Dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.druid</groupId>
<artifactId>druid-processing</artifactId>
<version>${project.parent.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
</dependencies>

</project>
71 changes: 71 additions & 0 deletions ...es/src/main/java/io/druid/query/aggregation/datasketches/theta/EmptySketchAggregator.java
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
/*
* Licensed to Metamarkets Group Inc. (Metamarkets) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. Metamarkets licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package io.druid.query.aggregation.datasketches.theta;

import io.druid.query.aggregation.Aggregator;

public class EmptySketchAggregator implements Aggregator
{
private final String name;

public EmptySketchAggregator(String name)
{
this.name = name;
}

@Override
public void aggregate()
{
}

@Override
public void reset()
{
}

@Override
public Object get()
{
return SketchOperations.EMPTY_SKETCH;
}

@Override
public float getFloat()
{
throw new UnsupportedOperationException("Not implemented");
}

@Override
public long getLong()
{
throw new UnsupportedOperationException("Not implemented");
}

@Override
public String getName()
{
return name;
}

@Override
public void close()
{
}
}
Loading

0 comments on commit 148153b

Please sign in to comment.