main cuml umap #397

dcolinmorgan · 2022-09-09T09:50:01Z

a few stray prints (couldnt get logger to work) but good for testing
now main branch has cuml umap with g.umap(engine='cuml'), otherwise defaults to umap-learn (also takes that as engine arg)

lmeyerov

engine : UMAPEngine (public interfaces) + engine_resolved : UMAPEngineCOncrete (internal calls) so users can pick which engine, default to "auto". Match how featurize() does it.
print -> logger
un-delete feature threading through params (needed for memoization)

In a follow-on PR, let's add an ipynb demo!

graphistry/umap_utils.py

lmeyerov · 2022-09-11T15:47:39Z

graphistry/umap_utils.py

@@ -119,10 +132,12 @@ def umap_lazy_init(
        negative_sample_rate=5,
        n_components: int = 2,
        metric: str = "euclidean",
+        # engine: str = "auto",


need this b/c dicates which gets init'd below

lmeyerov · 2022-09-11T15:49:58Z

graphistry/umap_utils.py

@@ -134,6 +149,7 @@ def umap_lazy_init(
                local_connectivity=local_connectivity,
                repulsion_strength=repulsion_strength,
                negative_sample_rate=negative_sample_rate,
+                # engine=engine,


bring back all these engine calls, we need

lmeyerov · 2022-09-11T15:57:02Z

graphistry/umap_utils.py

+
+            # self._umap = umap.UMAP(**umap_kwargs)
+            # if self.engine == 'cuml':
+            if 'cuml' in sys.modules:


def concrete_umap_engine(engine): if engine == "auto": if 'cuml' in ...

and then here:

def lazy_init(engine : UMAPEngine): engine_concrete : UMAPEngineConcrete = concretize_umap_engine(engine) if engine_concrete == 'cuml': ...

i assume 'conrete_umap_engine' here == 'resolve_umap_engine' you described above? the example feature_util has no 'conrete_feature_engine' analog

lmeyerov · 2022-09-11T15:57:48Z

graphistry/umap_utils.py

@@ -181,6 +210,8 @@ def umap_fit(self, X: pd.DataFrame, y: Union[pd.DataFrame, None] = None):
        mins = (time() - t) / 60
        logger.info(f"-UMAP-ing took {mins:.2f} minutes total")
        logger.info(f" - or {X.shape[0]/mins:.2f} rows per minute")
+        print(f"-UMAP-ing took {mins:.2f} minutes total")
+        print(f" - or {X.shape[0]/mins:.2f} rows per minute")


revert all printf before final landing

@dcolinmorgan if you want to have logging set VERBOSE=True in constants.py https://github.com/graphistry/pygraphistry/blob/master/graphistry/constants.py

lmeyerov · 2022-09-11T15:58:22Z

graphistry/umap_utils.py

+
+        # res._umap = umap.UMAP(**umap_kwargs)
+
+        if 'cuml' in sys.modules:


pass in engine selection; should be concretized by this point

graphistry/umap_utils.py

lmeyerov · 2022-09-11T15:59:42Z

graphistry/umap_utils.py

+            # engine='umap-learn'
+        print(engine)
+        has_umap=assert_imported(engine)
+        self.umap_lazy_init(has_umap=has_umap)


engine_resolved = resolve_umap)engine(engine)

lmeyerov · 2022-09-14T10:13:59Z

graphistry/umap_utils.py

+                     "`pip install cuml`")
+        raise import_cuml_exn
+
+UMAPEngineConcrete = Literal["none", "cuml", "umap_learn"]


"none" seems like an illegal value in this case -- how do we do a umap w/ no engine?

(it's valid for feature engineering b/c that means a no-op passthrough of features)

graphistry/umap_utils.py

lmeyerov · 2022-09-14T10:23:12Z

graphistry/umap_utils.py

+            self.engine = engine
+
+            if engine_resolved == "umap_learn":
+                del umap_kwargs['engine']


confusing for dels to be here, can be part of umap_kwargs construction

lmeyerov · 2022-09-14T10:23:50Z

graphistry/umap_utils.py

            self.umap_initialized = True
+            self.engine = engine


save resolved engine here

lmeyerov · 2022-09-14T10:25:31Z

graphistry/umap_utils.py

@@ -361,7 +434,10 @@ def umap(
        :return: self, with attributes set with new data
        """
        assert_imported()
+        assert_imported_cuml()
        self.umap_lazy_init()


pass engine

graphistry/umap_utils.py

lmeyerov · 2022-09-14T10:30:42Z

graphistry/umap_utils.py

        self.umap_lazy_init()
+        engine_resolved = resolve_umap_engine(engine)
+

        self.suffix = suffix


all assignments should be on self, and after if inplace: res = self else res = self.bind()... includes lazy init which seems to mutate a bunch too

ok i missed that one point and have changed it in the wimpiest commit ever

Ah guess I was unclear

engine and suffix should be set on res , currently defined like 430, not mutate the original self

For end-user simplicity and internal optimizations (memoization, ...), an important guarantee is the API is functional/pure

lmeyerov

Fixes:

ensure def umap(self, ...) immediately switches to res = .., so subsequent calls do not mutate original self (lazy_init...) in typical case of in_place=False. RN, I think it'll self._umap = ... .
memoization seems to be dropping impactful params like metric & engine

Recommended:

surprisingly typed (and unnecessary?) has_umap : str = "True"
notes on ipynb
add to CHANGELOG and to README

Follow-on PR?

if input is a pandas df yet engine is cuml, can we auto-coerce pdf to cudf (and keep cudf after)?

lmeyerov · 2022-09-15T14:39:42Z

graphistry/umap_utils.py

@@ -167,28 +160,30 @@ def umap_lazy_init(
        negative_sample_rate=5,
        n_components: int = 2,
        metric: str = "euclidean",
-        engine: str = "auto", #this seems to override user input
-        # has_umap: str = "True",
+        has_umap: str = "True",


Can we get rid of has_umap? Or at least make a bool?

I think it is there just for throwing exns when no engine, so we can do those early vs propagating

demos/demos_databases_apis/gpu_rapids/part_iv_gpu_cuml.ipynb

lmeyerov · 2022-09-15T14:57:16Z

graphistry/umap_utils.py

        engine_resolved = resolve_umap_engine(engine)
-
+        self.engine=engine_resolved



Maybe more on a future PR, we can probably automate pandas->cudf (and back) when the engine is cuml

lmeyerov · 2022-09-19T13:48:11Z

graphistry/umap_utils.py

        self.umap_lazy_init()
+        engine_resolved = resolve_umap_engine(engine)
+

        self.suffix = suffix


Ah guess I was unclear

engine and suffix should be set on res , currently defined like 430, not mutate the original self

For end-user simplicity and internal optimizations (memoization, ...), an important guarantee is the API is functional/pure

lmeyerov · 2022-09-19T16:59:36Z

graphistry/umap_utils.py

        inplace: bool = False,
        feature_engine: str = "auto",
        memoize: bool = True,
        **featurize_kwargs,
    ):
        """
            UMAP the featurized node or edges data,
-            or pass in your own X, y (optional).
+            or pass in your own fX, y (optional).


lmeyerov · 2022-09-21T05:03:36Z

graphistry/umap_utils.py

            umap_kwargs = dict(
                n_components=n_components,
-                metric=metric,
+                # metric=metric,


why remove metric?

if a problem for cuml, we can throw a targeted exn if engine is cuml

as far as I can tell this flag is still an open issue, with some hacks to get it working rapidsai/cuml#1653 (comment). From what I can tell there is no such flag listed in the cuml UMAP function https://docs.rapids.ai/api/cuml/stable/api.html?highlight=umap#cuml.UMAP and since metric='euclidean' by default in sklearn anyway I thought we could bypass the need for it. although i can see why the user could want to keep the functionality

yeah that was my guess why you did it. it ends up big for umap users, so to support mult engines, a pattern can be like:

opts = { 'a': 'b', 'c': 'd', **({'x': 'y'} if engine='umap' else {}) }

graphistry/umap_utils.py

lmeyerov · 2022-09-22T05:40:38Z

README.md

+
+    ```python
+    g = graphistry.nodes(G)
+    g.umap(engine='cuml',metric='euclidean')


g.umap(engine='cum').plot() :)

README.md

lmeyerov · 2022-09-22T05:43:31Z

README.md

@@ -178,6 +178,13 @@ It is easy to turn arbitrary data into insightful graphs. PyGraphistry comes wit
    graphistry.edges(edges, 'src', 'dst').plot()
    ```

+* GPU [RAPIDS.ai](https://www.rapids.ai) cuml


lmeyerov · 2022-09-22T05:48:44Z

README.md

@@ -384,6 +391,12 @@ See `help(g.featurize)` for more options
      g.umap(kind='nodes', X=['col_1', ..., 'col_n'], y=['label', ..., 'other_targets'], ...)
    ```

+* UMAP supports different backend `engine`, which by default exploit an availabile GPU, resulting in marked speed increases, and falling back to CPU processing:


umap(engine="...") supports multiple implementations. It defaults to using the GPU-accelerated engine="cuml" when a GPU is available, resulting in orders-of-magnitude speedups, and falls back to CPU processing via engine="umap".

graphistry/umap_utils.py

lmeyerov · 2022-09-23T05:40:50Z

demos/demos_databases_apis/gpu_rapids/part_iv_gpu_cuml.ipynb

+    "!pip install -U pygraphistry/ --quiet\n",
+    "\n",
+    "import graphistry \n",
+    "graphistry.register(api=3,protocol=\"https\", server=\"hub.graphistry.com\", username='dcolinmorgan', password='f5UwthGEF@F@xnP')\n",


@dcolinmorgan can you scrub your pwd and reset your hub acct?

lmeyerov · 2022-09-25T01:06:55Z

FYI, a bunch of fixes on main branch in case your CI has a lot of false positives. Still a false positive from py3.10 around mypi, but that should be it.

main cuml umap

a674343

lmeyerov requested changes Sep 11, 2022

View reviewed changes

concrete cuml

a2c6fcb

lmeyerov reviewed Sep 14, 2022

View reviewed changes

graphistry/umap_utils.py Show resolved Hide resolved

lmeyerov reviewed Sep 14, 2022

View reviewed changes

graphistry/umap_utils.py Outdated

self.umap_initialized = True

self.engine = engine

Copy link

Contributor

lmeyerov Sep 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

save resolved engine here

lmeyerov reviewed Sep 14, 2022

View reviewed changes

graphistry/umap_utils.py Show resolved Hide resolved

lmeyerov reviewed Sep 14, 2022

View reviewed changes

Daniel Morgan added 2 commits September 15, 2022 15:31

functional concrete flags, demo notebook

cf7e5bc

define to self only

ffdc923

lmeyerov requested changes Sep 21, 2022

View reviewed changes

resurrect engine, embelish notebook

084aac0

lmeyerov reviewed Sep 22, 2022

View reviewed changes

README.md Outdated

```python

g = graphistry.nodes(G)

g.umap(engine='cuml',metric='euclidean')

Copy link

Contributor

lmeyerov Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

g.umap(engine='cum').plot() :)

lmeyerov reviewed Sep 22, 2022

View reviewed changes

README.md Show resolved Hide resolved

lmeyerov reviewed Sep 22, 2022

View reviewed changes

graphistry/umap_utils.py Show resolved Hide resolved

silkspace reviewed Sep 22, 2022

View reviewed changes

graphistry/umap_utils.py Show resolved Hide resolved

readme, remove res.engine def

cd611ed

lmeyerov reviewed Sep 23, 2022

View reviewed changes

Daniel Morgan added 3 commits September 27, 2022 07:58

remove accnt info from notebook

6beb45e

move umap_lazy_init after if inplace

63ab188

res.suffix location

3ec73bc

dcolinmorgan closed this by deleting the head repository Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

main cuml umap #397

main cuml umap #397

dcolinmorgan commented Sep 9, 2022

lmeyerov left a comment •

edited

Loading

lmeyerov Sep 11, 2022

lmeyerov Sep 11, 2022

lmeyerov Sep 11, 2022

dcolinmorgan Sep 13, 2022

lmeyerov Sep 11, 2022

silkspace Sep 19, 2022

lmeyerov Sep 11, 2022

lmeyerov Sep 11, 2022

lmeyerov Sep 14, 2022

lmeyerov Sep 14, 2022

lmeyerov Sep 14, 2022

lmeyerov Sep 14, 2022

lmeyerov Sep 14, 2022 •

edited

Loading

dcolinmorgan Sep 19, 2022

lmeyerov Sep 19, 2022

lmeyerov left a comment

lmeyerov Sep 15, 2022

lmeyerov Sep 15, 2022

lmeyerov Sep 19, 2022

lmeyerov Sep 19, 2022

lmeyerov Sep 21, 2022

dcolinmorgan Sep 21, 2022 •

edited

Loading

lmeyerov Sep 21, 2022 •

edited

Loading

lmeyerov Sep 22, 2022

lmeyerov Sep 22, 2022

lmeyerov Sep 22, 2022

lmeyerov Sep 23, 2022

lmeyerov commented Sep 25, 2022


		# res._umap = umap.UMAP(**umap_kwargs)

		if 'cuml' in sys.modules:

		engine_resolved = resolve_umap_engine(engine)

		self.engine=engine_resolved

main cuml umap #397

main cuml umap #397

Conversation

dcolinmorgan commented Sep 9, 2022

lmeyerov left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmeyerov Sep 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmeyerov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcolinmorgan Sep 21, 2022 • edited Loading

Choose a reason for hiding this comment

lmeyerov Sep 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmeyerov commented Sep 25, 2022

lmeyerov left a comment •

edited

Loading

lmeyerov Sep 14, 2022 •

edited

Loading

dcolinmorgan Sep 21, 2022 •

edited

Loading

lmeyerov Sep 21, 2022 •

edited

Loading