Adding new backend for MapD #1419

xmnlab · 2018-04-13T20:45:24Z

Also resolves #1418 and resolves #893

Added mapd backend initial files.

Improved mapd client and compiler; Added initial documentation.

README updated; Initial changes to use execute method.

Improving ibis.mapd client and compiler

Added Math, trigonometric and geometric operations

Resolves ibis ibis-project#1418 and resolves ibis-project#893

cpcloud

Thanks @xmnlab! This is solid progress. Let's do a few more review cycles before we merge this in and try to clean up a bit of the duplication. Overall, though, this is pretty close.

cpcloud · 2018-04-14T15:14:13Z

ibis/expr/api.py

@@ -388,6 +389,8 @@ def row_number():

 e = ops.E().to_expr()

+pi = ops.Pi().to_expr().name('pi')


I would leave this unnamed for now. No reason to make this choice for users.

cpcloud · 2018-04-14T15:15:35Z

ibis/expr/api.py

+acos = _unary_op('acos', ops.Acos)
+asin = _unary_op('asin', ops.Asin)
+atan = _unary_op('atan', ops.Atan)
+atan2 = _generic_op('atan2', ops.Atan2)


This is a binary operation right? There should be something like _binary_op function lying around here.

you're right. I think this is the right function: _binop_expr

cpcloud · 2018-04-14T15:16:44Z

ibis/expr/operations.py

@@ -516,6 +521,53 @@ class Log10(Logarithm):
    """Logarithm base 10"""


+# TRIGONOMETRIC OPERATIONS
+
+class TrigonometryUnary(UnaryOp):


Can you change the name from TrigonometryUnary to TrigonometricUnary, and do the same for TrigonometryBinary.

cpcloud · 2018-04-14T15:17:40Z

ibis/expr/operations.py

@@ -2183,6 +2235,14 @@ def output_type(self):
        return partial(ir.FloatingScalar, dtype=dt.float64)


+class Pi(Constant):
+    """


I guess you could saying something here like The constant pi.

cpcloud · 2018-04-14T15:24:40Z

ibis/mapd/client.py

+        elif GPUDataFrame is not None and isinstance(
+                self.cursor, GPUDataFrame
+        ):
+            result = self.cursor


Since you're executing the same code (assigning self.cursor to result) in the case that the cursor is a pandas DataFrame or a GPUDataFrame, can you remove the last two elifs? Is there a case where self.cursor is not None and it's not either a pandas DataFrame or a GPUDataFrame?

you're right! thanks!

cpcloud · 2018-04-14T15:44:38Z

ibis/mapd/compiler.py

+    # compile the argument
+    compiled_arg = translator.translate(arg)
+
+    return 'CHAR_LENGTH(%s)' % compiled_arg


Format strings.

cpcloud · 2018-04-14T15:45:38Z

ibis/mapd/tests/test_compiler.py

+import ibis.expr.datatypes as dt
+
+
+def test_timestamp_accepts_date_literals(alltypes):


This is a BigQuery specific test. Do you really need it?

cpcloud · 2018-04-14T15:48:29Z

ibis/mapd/udf.py

@@ -0,0 +1,5 @@
+"""
+User Defined Function


I wouldn't add this module unless there's support for this in MapD.

cpcloud · 2018-04-14T15:50:07Z

ibis/mapd/tests/test_client.py

+    assert result == expected
+
+'''
+def test_simple_aggregate_execute(alltypes, df):


Many of these tests are BigQuery specific. Can you remove them from here?

The preferred alternative is to add appropriate tests in ibis/tests/all/test_*.py. To do that, you'll also need to add a MapD class in ibis/tests/all/backends.py. There are many examples in that file to get you started.

cpcloud · 2018-04-14T15:54:15Z

ibis/mapd/operations.py

@@ -0,0 +1,798 @@
+from six import StringIO


It looks like a lot of the functions and objecs in here are duplicated from either the impala or bigquery backends. Can you see if you can reuse some of their functions so we have only what's needed for MapD?

Ok. I will take a look :) thanks!

cpcloud · 2018-06-06T20:53:29Z

ci/requirements-docs-3.6.yml

@@ -22,6 +22,7 @@ dependencies:
  - plumbum
  - psycopg2
  - pyarrow>=0.6.0
+  - pymapd


does this require a version constraint?

you're right! it is better to pin the version here. thanks!

cpcloud · 2018-06-06T20:54:16Z

ibis/__init__.py

@@ -71,6 +72,12 @@
    # pip install ibis-framework[bigquery]
    import ibis.bigquery.api as bigquery

+with suppress(ImportError):
+    # pip install ibis-framework[mapd]
+    if sys.version_info[0] < 3:


Use sys.version_info.major here

cpcloud · 2018-06-06T20:55:14Z

ibis/__init__.py

+with suppress(ImportError):
+    # pip install ibis-framework[mapd]
+    if sys.version_info[0] < 3:
+        raise ImportError('ibis.mapd is not allowed it for Python 2.')


typo here, should read ibis.mapd is not allowed for Python 2 or The mapd backend is not supported under Python 2

Corrections from the revision

xmnlab

Thanks @cpcloud !

I've push new changes

xmnlab · 2018-06-07T00:25:23Z

ibis/expr/operations.py

+            dtype = self.left.type().largest
+        else:
+            dtype = dt.float64
+        return dtype.scalar_type()


Oh I see. Sorry, my mistake. I was a little bit confused. I will changed that.

xmnlab · 2018-06-07T00:25:49Z

ibis/expr/operations.py

+    how = Arg(rlz.isin({'sample', 'pop'}), default=None)
+    where = Arg(rlz.boolean, default=None)
+
+    def output_type(self):


Ok! thanks!

xmnlab · 2018-06-07T01:03:21Z

ibis/expr/operations.py

+
+class Distance(ValueOp):
+    """
+    Calculates distance in meters between two WGS-84 positions.


Yes, we can do that. We just cannot test it because mapd doesn't have a Euclidean operation.
I am just not sure if common Euclidean distance function works with lat lon parameters .. so maybe would be better to remove this function and create a issue to follow this discussion.

Changed left and right params to column numeric type

xmnlab · 2018-06-07T16:21:08Z

@cpcloud thanks a lot for reviewing this PR.

this is a compilation of the main fixes here:

ci/requirements-dev-3.5.yml: I just made a rollback and works good.
Degrees and Radians: I changed the input to numeric and I changed to output to float64, I also add tests for that on mapd tests.
Correlation and Covariance: Sorry I misunderstood that, thanks for the patience :) ... I changed the output to dt.float64.scalar_type() .. I also added tests for these operations on mapd tests.
Distance: I am not sure if Euclidean distance common functions work with lat lon .. so I removed that from this PR and I will create now a issue to follow up this.

if these points are ok for you, I think it is ready for a new review. I just changed the left and right parameters from correlation to column numeric.

Again, thank you so much for your attention.

xmnlab · 2018-06-13T14:34:20Z

hi @cpcloud @kszucs

any update about this PR?

thanks a lot!

xmnlab · 2018-06-13T14:35:07Z

there is a conflict now .. I will rebase here now.

Merged from master

cpcloud

@xmnlab After you address this round of comments I wil approve and merge! Thanks for the effort!!

cpcloud · 2018-06-15T14:05:33Z

ibis/mapd/client.py

+        self, name, password=None, is_super=None, insert_access=None
+    ):
+        """
+        Create a new MapD database


This docstring looks wrong.

you're right. thanks!

cpcloud · 2018-06-15T14:06:43Z

ibis/mapd/client.py

+        statement = ddl.DropDatabase(name)
+        self._execute(statement)
+
+    def create_user(self, name, password, is_super=False):


This looks like a new API. @xmnlab can you create a follow up issue to add this API to the clients that have support for such functionality?

Ok I will do it.

cpcloud · 2018-06-15T14:06:52Z

ibis/mapd/client.py

+        )
+        self._execute(statement)
+
+    def drop_user(self, name):


Same as above.

ok I will do it! thanks!

cpcloud · 2018-06-18T14:50:04Z

ibis/tests/all/test_temporal.py

@@ -41,9 +41,11 @@ def test_timestamp_extract(backend, alltypes, df, attr):
 @pytest.mark.parametrize('unit', [
    'Y', 'M', 'D',
    param('W', marks=pytest.mark.xfail),
-    'h', 'm', 's', 'ms', 'us', 'ns'
+    'h', 'm', 's', 'ms', 'us',
+    param('ns', marks=pytest.mark.xfail)


Does this pass now that you added the skipif_backend?

Yes, for some reason when it try to test ns unit .. raise an error that the pytest breaks ...
I tried to use pytest mark for ns and it didn't work .. I needed to skip that for mapd.
for another tests here .. I could remove skipif_backend and just put some xfail for some units and works good.

Is it necessary to have both xfail on 'ns' and the skipif_backend('MapD') decorator? Shouldn't it be enough to just skip this on MapD altogether?

you're right! sorry, I will remove this now.

cpcloud · 2018-06-18T14:50:44Z

ibis/tests/all/conftest.py

+    pytest.param(Impala, marks=pytest.mark.impala)
+]
+
+if sys.version_info.major == 3:


should be > 2

cpcloud · 2018-06-18T14:51:50Z

ibis/mapd/tests/test_operations.py

+    ),
+    param(
+        lambda t: t.double_col.cov(t.float_col),
+        91.67005567565313,


Ultimately we will want to change these to use a pandas call or numpy call, so that we don't have to depend on hard coded values.

This is fine for now though.

cpcloud · 2018-06-18T14:54:17Z

ibis/mapd/ddl.py

@@ -0,0 +1,422 @@
+from ibis.sql.compiler import DDL, DML


Do you have tests for the classes in this file? If not, please add them in a follow-up PR.

No ... I had some problems adding DDL tests because it was breaking the mapd database container. I will create now a issue for that.

cpcloud · 2018-06-18T14:57:01Z

ibis/expr/api.py

@@ -1752,6 +1814,34 @@ def _string_like(self, patterns):
    )


+def _string_ilike(self, patterns):


Does this API have a test in this PR?

xmnlab

changes done. I will commit my changes.

xmnlab · 2018-06-18T15:05:57Z

ibis/mapd/tests/test_operations.py

+    ),
+    param(
+        lambda t: t.double_col.cov(t.float_col),
+        91.67005567565313,


xmnlab · 2018-06-18T15:09:55Z

ibis/tests/all/conftest.py

+    pytest.param(Impala, marks=pytest.mark.impala)
+]
+
+if sys.version_info.major == 3:


xmnlab · 2018-06-18T15:41:48Z

ibis/mapd/ddl.py

@@ -0,0 +1,422 @@
+from ibis.sql.compiler import DDL, DML


No ... I had some problems adding DDL tests because it was breaking the mapd database container. I will create now a issue for that.

xmnlab · 2018-06-18T15:49:37Z

ibis/mapd/client.py

+        statement = ddl.DropDatabase(name)
+        self._execute(statement)
+
+    def create_user(self, name, password, is_super=False):


Ok I will do it.

xmnlab · 2018-06-18T15:54:24Z

ibis/mapd/client.py

+        self, name, password=None, is_super=None, insert_access=None
+    ):
+        """
+        Create a new MapD database


you're right. thanks!

xmnlab · 2018-06-18T15:54:58Z

ibis/mapd/client.py

+        )
+        self._execute(statement)
+
+    def drop_user(self, name):


ok I will do it! thanks!

xmnlab · 2018-06-18T20:34:15Z

ibis/tests/all/test_temporal.py

@@ -41,9 +41,11 @@ def test_timestamp_extract(backend, alltypes, df, attr):
 @pytest.mark.parametrize('unit', [
    'Y', 'M', 'D',
    param('W', marks=pytest.mark.xfail),
-    'h', 'm', 's', 'ms', 'us', 'ns'
+    'h', 'm', 's', 'ms', 'us',
+    param('ns', marks=pytest.mark.xfail)


Yes, for some reason when it try to test ns unit .. raise an error that the pytest breaks ...
I tried to use pytest mark for ns and it didn't work .. I needed to skip that for mapd.
for another tests here .. I could remove skipif_backend and just put some xfail for some units and works good.

Changes from the review

Removed xfail mark for ns unit

cpcloud · 2018-06-18T21:29:29Z

Merging on green!

xmnlab · 2018-06-18T21:40:00Z

@cpcloud thank you so much for your attention and support!

cpcloud · 2018-06-18T21:55:16Z

nice! bombs away!

xmnlab added 26 commits April 5, 2018 15:23

Added mapd backend initial files.

73becc5

Changed connection

f0d09f5

MapDClient.table function worked (initial version).

b120529

Merge pull request #1 from xmnlab/master

6099ba2

Added mapd backend initial files.

Improved mapd client and compiler; Added initial documentation.

8c8df6d

Merge pull request #3 from xmnlab/master

fb8bbe3

Improved mapd client and compiler; Added initial documentation.

README updated; Initial changes to use execute method.

f381223

Merge pull request #8 from xmnlab/master

7289937

README updated; Initial changes to use execute method.

MapDCursor added

f34289d

Resolves #7

77c93a5

Resolves #4

33fc398

Double precision data type removed

a6b773a

Resolves partially #5

6592058

Partially added mapd operations.

f524894

Sin function added.

b0e0eb9

Changed TrigonometryUnary class

7b62e02

Merge pull request #9 from xmnlab/master

14bcedf

Improving ibis.mapd client and compiler

Fixed small bugs.

5a06f30

Math, trigonometric and geometric functions added

bda8e3c

Merge pull request #11 from xmnlab/master

3c4e6f7

Added Math, trigonometric and geometric operations

Added small changes

ed58893

Merge remote-tracking branch 'ibis-project/master'

120efaa

Added std, var and corr operations.

4061ea0

Resolves ibis ibis-project#1418 and resolves ibis ibis-project#893

c52a0bb

Merge pull request #15 from xmnlab/master

703b997

Resolves ibis ibis-project#1418 and resolves ibis-project#893

Added string operation; Added now and truncate operations;

0aac204

cpcloud requested changes Apr 14, 2018

View reviewed changes

xmnlab added 3 commits April 14, 2018 14:17

Added timestamp truncate and extract

bec6199

Merge remote-tracking branch 'quansight/master'

70ba0ec

Merge remote-tracking branch 'ibis-project/master'

c952e95

cpcloud reviewed Jun 6, 2018

View reviewed changes

xmnlab added 3 commits June 6, 2018 21:13

Corrections from the revision

9b6a48c

Merge remote-tracking branch 'ibis-project/master'

2cac525

Merge pull request #60 from xmnlab/master

a169dff

Corrections from the revision

xmnlab commented Jun 7, 2018

View reviewed changes

xmnlab added 2 commits June 7, 2018 12:14

Changed left and right params to column numeric type

5fec8d4

Merge pull request #61 from xmnlab/master

825d5ca

Changed left and right params to column numeric type

xmnlab added 2 commits June 13, 2018 10:40

Merged from master

fe01689

Merge pull request #62 from xmnlab/master

088f49d

Merged from master

cpcloud requested changes Jun 18, 2018

View reviewed changes

xmnlab added 3 commits June 18, 2018 11:08

Merge remote-tracking branch 'ibis-project/master'

d1e76b0

Fixed small bugs; added test for ilike operation

5a342c8

Changed parameters for mapd tests.

f9f017d

xmnlab commented Jun 18, 2018

View reviewed changes

xmnlab added 3 commits June 18, 2018 16:40

Merge pull request #63 from xmnlab/master

7043581

Changes from the review

Removed xfail mark for ns unit

8f5355f

Merge pull request #64 from xmnlab/master

f4a5dc8

Removed xfail mark for ns unit

cpcloud approved these changes Jun 18, 2018

View reviewed changes

cpcloud closed this in 037db67 Jun 18, 2018

xmnlab changed the title ~~[WiP] Adding new backend for MapD~~ Adding new backend for MapD Jun 19, 2018

This was referenced Jun 22, 2018

Correlation and Covariation operations #1432

Closed

ILIKE operation #1433

Closed

		@@ -388,6 +389,8 @@ def row_number():

		e = ops.E().to_expr()

		pi = ops.Pi().to_expr().name('pi')

		import ibis.expr.datatypes as dt


		def test_timestamp_accepts_date_literals(alltypes):

		@@ -1752,6 +1814,34 @@ def _string_like(self, patterns):
		)


		def _string_ilike(self, patterns):

Adding new backend for MapD #1419

Adding new backend for MapD #1419

Conversation

xmnlab commented Apr 13, 2018

cpcloud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab commented Jun 7, 2018

xmnlab commented Jun 13, 2018

xmnlab commented Jun 13, 2018

cpcloud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab Jun 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab Jun 18, 2018 • edited Loading

Choose a reason for hiding this comment

cpcloud commented Jun 18, 2018

xmnlab commented Jun 18, 2018

cpcloud commented Jun 18, 2018

xmnlab Jun 18, 2018 •

edited

Loading

xmnlab Jun 18, 2018 •

edited

Loading