Stacked @card v1.2 : Customizing @card with current.card (#894)

* allow passing userland `MetaflowCardComponents`
-  Added api for `current.cards` that handles multiple decorators

* `current.cards` with `id` support.
- Introduced `ALLOW_USER_COMPONENTS` attribute to `MetaflowCard` class
- setting `ALLOW_USER_COMPONENTS=True` allow editable cards
- Modified logic of `CardComponentCollector`
- `CardComponentCollector.append` only accessible to default editble card

* Added `customize=True` in `@card`
- `customize=True` only allowed for one @card deco per @step
- `customize=True` sets the card to be default editable
- `current.card.append` appends to @card with `customize=True`


* Stacked @card v1.2: Read cli changes for Supporting Multiple`@card`s (#895)

* Added the `--id` option to `card view`/`card get`
- modified datastore and fixed the exception class.
- Allowing some ambiguity in pathspec argument for `get/view` command.

* Added `id` argument to `get_cards`

* Bringing `card list` cli functionality from test-suite branch
- added functionality to list cards about a task and even list it as JSON.

* changed hash checking logic.
- instead of equating we check `startswith`

* Added list many feature for `card list`
- listing all cards from the latest run when card list is called with no argument

* Added `card list` print formatting changes
- `--as-json` works for many cards and single cards


* Stacked @card v1.2 : New `MetaflowCardComponent`s  (#896)

* user-facing `MetaflowCardComponent`s
- import via `from metaflow.cards import Artifact,..`

* 7 Major Changes:
- Removed `Section` component from `metaflow.cards`
- Making user-facing component inherent to `UserComponent` which in turn inherits `MetaflowCardComponent`
- Wrap all `UserComponents` inside a section after rendering everything per card
- created a `render_safely` decorator to ensure fail-safe render of `UserComponent`s
- removed code from component serializer which used internal components
- Refactored some components that return render
- Added docstrings to all components.

* JS + CSS + Cards UI Build

* Stacked @card v1.2 : Graph Related Changes to card cli (#911)

* accomodating changes from #833
- Minor tweaks to `graph.py`

* Stacked @card v1.2 : Namespace Packages with `@card` (#897)

* setup import of cards from `metaflow_extensions` using `metaflow.extension_support`
- Added import modules in `card_modules`
- Added hook in card decorators to add custom packages

* Added some debug statements for external imports.

* Stacked @card v1.2 : Test cases for Multiple `@card`s (#898)

* Multiple Cards Test Suite Mods (#27)

- Added `list_cards` to `CliCheck` and `MetadataCheck`
- Bringing #875 into the code
- Added a card that prints taskspec with random number

* Added test case for multiple cards

* Added tests card, summary :
- `current.cards['myid']` should be accessible when cards have an `id` argument in decorator
- `current.cards.append` should not work when there are no single default editable card.
- if a card has `ALLOW_USER_COMPONENTS=False` then it can still be edited via accessing it with `id` property.
- adding arbitrary information to `current.cards.append` should not break user code.
- Only cards with `ALLOW_USER_COMPONENTS=True` are considered default editable.
- If a single @card decorator is present with `id` then it `current.cards.append` should still work
- Access of `current.cards` with non existant id should not fail.
- `current.cards.append` should be accessible to the card with `customize=True`.
-

* fixed `DefaultEditableCardTest` test case
- Fixed comment
- Fixed the `customize=True` test case.

* ensure `test_pathspec_card` has no duplicates
- ensure entropy of rendered information is high enough to not overwrite a file.

* test case fix : `current.cards` to `current.card`

* Added Test case to support import of cards.
- Test case validates that broken card modules don't break metaflow
- test case validates that we are able to import cards from metaflow_extensions
- Test case validate that cards can be editable if they are importable.

* Added Env var to tests to avoid warnings added to cards.

* Added Test for card resume.

* Stacked @card v1.2: Card Dev Docs (#899)

Co-authored-by: Brendan Gibson <brendan@outerbounds.co>
Co-authored-by: Brendan Gibson <93726128+obgibson@users.noreply.github.com>
Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com>
Co-authored-by: Romain Cledat <rcledat@netflix.com>

metaflow/cards.py

@@ -1,21 +1,19 @@
 from metaflow.plugins.cards.card_client import get_cards
 from metaflow.plugins.cards.card_modules.card import MetaflowCardComponent, MetaflowCard
+from metaflow.plugins.cards.card_modules.components import (
+    Artifact,
+    Table,
+    Image,
+    Error,
+    Markdown,
+)
 from metaflow.plugins.cards.card_modules.basic import (
     DefaultCard,
-    TitleComponent,
-    SubTitleComponent,
-    SectionComponent,
-    ImageComponent,
-    BarChartComponent,
-    LineChartComponent,
-    TableComponent,
-    DagComponent,
     PageComponent,
-    ArtifactsComponent,
     RENDER_TEMPLATE_PATH,
     TaskToDict,
     DefaultComponent,
-    ChartComponent,
     TaskInfoComponent,
     ErrorCard,
+    BlankCard,
 )

metaflow/graph.py

@@ -298,6 +298,8 @@ def node_to_dict(name, node):
                 d["foreach_artifact"] = node.foreach_param
             elif d["type"] == "split-parallel":
                 d["num_parallel"] = node.num_parallel
+            if node.matching_join:
+                d["matching_join"] = node.matching_join
             return d
 
         def populate_block(start_name, end_name):

metaflow/metaflow_config.py

@@ -64,26 +64,29 @@ def from_conf(name, default=None):
 DATATOOLS_SUFFIX = from_conf("METAFLOW_DATATOOLS_SUFFIX", "data")
 DATATOOLS_S3ROOT = from_conf(
     "METAFLOW_DATATOOLS_S3ROOT",
-    "%s/%s" % (from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATATOOLS_SUFFIX)
+    os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATATOOLS_SUFFIX)
     if from_conf("METAFLOW_DATASTORE_SYSROOT_S3")
     else None,
 )
 # Local datatools root location
 DATATOOLS_LOCALROOT = from_conf(
     "METAFLOW_DATATOOLS_LOCALROOT",
-    "%s/%s" % (from_conf("METAFLOW_DATASTORE_SYSROOT_LOCAL"), DATATOOLS_SUFFIX)
+    os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_LOCAL"), DATATOOLS_SUFFIX)
     if from_conf("METAFLOW_DATASTORE_SYSROOT_LOCAL")
     else None,
 )
 
+# Cards related config variables
 DATASTORE_CARD_SUFFIX = "mf.cards"
 DATASTORE_CARD_LOCALROOT = from_conf("METAFLOW_CARD_LOCALROOT")
 DATASTORE_CARD_S3ROOT = from_conf(
     "METAFLOW_CARD_S3ROOT",
-    "%s/%s" % (from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATASTORE_CARD_SUFFIX)
+    os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATASTORE_CARD_SUFFIX)
     if from_conf("METAFLOW_DATASTORE_SYSROOT_S3")
     else None,
 )
+CARD_NO_WARNING = from_conf("METAFLOW_CARD_NO_WARNING", False)
+
 # S3 endpoint url
 S3_ENDPOINT_URL = from_conf("METAFLOW_S3_ENDPOINT_URL", None)
 S3_VERIFY_CERTIFICATE = from_conf("METAFLOW_S3_VERIFY_CERTIFICATE", None)

metaflow/plugins/__init__.py

@@ -150,17 +150,32 @@ def get_plugin_cli():
 _merge_lists(FLOW_DECORATORS, _ext_plugins["FLOW_DECORATORS"], "name")
 
 # Cards
-from .cards.card_modules.basic import DefaultCard, TaskSpecCard, ErrorCard
-from .cards.card_modules.test_cards import TestErrorCard, TestTimeoutCard, TestMockCard
+from .cards.card_modules.basic import DefaultCard, TaskSpecCard, ErrorCard, BlankCard
+from .cards.card_modules.test_cards import (
+    TestErrorCard,
+    TestTimeoutCard,
+    TestMockCard,
+    TestPathSpecCard,
+    TestEditableCard,
+    TestEditableCard2,
+    TestNonEditableCard,
+)
+from .cards.card_modules import MF_EXTERNAL_CARDS
 
 CARDS = [
     DefaultCard,
     TaskSpecCard,
     ErrorCard,
+    BlankCard,
     TestErrorCard,
     TestTimeoutCard,
     TestMockCard,
-]
+    TestPathSpecCard,
+    TestEditableCard,
+    TestEditableCard2,
+    TestNonEditableCard,
+    BlankCard,
+] + MF_EXTERNAL_CARDS
 # Sidecars
 from ..mflog.save_logs_periodically import SaveLogsPeriodicallySidecar
 from metaflow.metadata.heartbeat import MetadataHeartBeat