Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: iterative/dvc
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 2.33.2
Choose a base ref
...
head repository: iterative/dvc
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 2.34.0
Choose a head ref

Commits on Nov 3, 2022

  1. Copy the full SHA
    08a643e View commit details
  2. Copy the full SHA
    1cad8ea View commit details
  3. Copy the full SHA
    f9b0de8 View commit details
  4. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    fe1d805 View commit details
  5. Copy the full SHA
    c02bd7f View commit details
  6. Copy the full SHA
    bda1388 View commit details
  7. Copy the full SHA
    ecd371f View commit details
  8. Copy the full SHA
    c27005a View commit details
  9. tests: use mocker

    skshetry committed Nov 3, 2022
    Copy the full SHA
    4d17c17 View commit details
  10. fix windows

    skshetry committed Nov 3, 2022
    Copy the full SHA
    15f35e8 View commit details

Commits on Nov 4, 2022

  1. build(deps-dev): Bump dvc-s3 from 2.20.1 to 2.21.0

    Bumps [dvc-s3](https://github.com/iterative/dvc-s3) from 2.20.1 to 2.21.0.
    - [Release notes](https://github.com/iterative/dvc-s3/releases)
    - [Commits](iterative/dvc-s3@2.20.1...2.21.0)
    
    ---
    updated-dependencies:
    - dependency-name: dvc-s3
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored and skshetry committed Nov 4, 2022
    Copy the full SHA
    43a8eab View commit details
  2. plots: allow top-level strings

    dberenbaum authored and skshetry committed Nov 4, 2022
    Copy the full SHA
    ab69b88 View commit details
  3. Update dvc/repo/plots/__init__.py

    pared authored and skshetry committed Nov 4, 2022
    Copy the full SHA
    d7152e3 View commit details
  4. Copy the full SHA
    76b1f8c View commit details
  5. Copy the full SHA
    33d4ead View commit details
  6. Copy the full SHA
    fb70315 View commit details
  7. Copy the full SHA
    bb62833 View commit details
  8. Copy the full SHA
    3891e81 View commit details
  9. Copy the full SHA
    7bb8428 View commit details
  10. Update README.rst

    skshetry committed Nov 4, 2022
    Copy the full SHA
    740fe50 View commit details
  11. Copy the full SHA
    69f78d3 View commit details
  12. fix hydra_sweeps referenced before assignment

    dberenbaum authored and daavoo committed Nov 4, 2022
    Copy the full SHA
    240f0eb View commit details

Commits on Nov 7, 2022

  1. exp init: Update DVCLive paths.

    Closes #8531
    daavoo committed Nov 7, 2022
    Copy the full SHA
    78e2574 View commit details
  2. deps: Bump dvclive to >= 1.0

    daavoo committed Nov 7, 2022
    Copy the full SHA
    eb20a8e View commit details
  3. Remove live section support.

    daavoo committed Nov 7, 2022
    Copy the full SHA
    1de983f View commit details
  4. tests: ignore versions for subprojects

    skshetry authored and daavoo committed Nov 7, 2022
    Copy the full SHA
    a48a19c View commit details
  5. exp init: fix live metrics path

    skshetry authored and daavoo committed Nov 7, 2022
    Copy the full SHA
    2c3956b View commit details
Showing with 2,452 additions and 3,643 deletions.
  1. +1 −1 .github/workflows/benchmarks.yaml
  2. +6 −2 README.rst
  3. +0 −20 dvc/commands/stage.py
  4. +6 −4 dvc/dependency/__init__.py
  5. +5 −0 dvc/dependency/base.py
  6. +0 −3 dvc/env.py
  7. +56 −64 dvc/output.py
  8. +1 −1 dvc/repo/__init__.py
  9. +10 −8 dvc/repo/experiments/init.py
  10. +2 −10 dvc/repo/experiments/queue/base.py
  11. +14 −20 dvc/repo/experiments/run.py
  12. +9 −12 dvc/repo/index.py
  13. +0 −19 dvc/repo/live.py
  14. +1 −6 dvc/repo/metrics/show.py
  15. +13 −6 dvc/repo/plots/__init__.py
  16. +0 −6 dvc/repo/reproduce.py
  17. +1 −9 dvc/schema.py
  18. +2 −17 dvc/stage/__init__.py
  19. +0 −1 dvc/stage/loader.py
  20. +0 −4 dvc/stage/monitor.py
  21. +0 −1 dvc/stage/params.py
  22. +3 −10 dvc/stage/serialize.py
  23. +1 −43 dvc/stage/utils.py
  24. +37 −8 dvc/utils/hydra.py
  25. +5 −4 pyproject.toml
  26. +0 −173 tests/basic_env.py
  27. +2 −5 tests/conftest.py
  28. +2 −29 tests/dir_helpers.py
  29. +6 −7 tests/func/experiments/test_experiments.py
  30. +4 −10 tests/func/experiments/test_gc.py
  31. +8 −6 tests/func/experiments/test_init.py
  32. +5 −0 tests/func/experiments/test_set_params.py
  33. +7 −10 tests/func/experiments/test_show.py
  34. +141 −190 tests/func/test_add.py
  35. +2 −2 tests/func/test_analytics.py
  36. +277 −381 tests/func/test_checkout.py
  37. +137 −147 tests/func/test_cli.py
  38. +3 −0 tests/func/test_dvcfile.py
  39. +56 −150 tests/func/test_gc.py
  40. +9 −9 tests/func/test_import.py
  41. +32 −41 tests/func/test_import_url.py
  42. +1 −1 tests/func/test_init.py
  43. +0 −228 tests/func/test_live.py
  44. +87 −156 tests/func/test_move.py
  45. +56 −65 tests/func/test_remote.py
  46. +2 −2 tests/func/test_repo_index.py
  47. +667 −676 tests/func/test_repro.py
  48. +14 −156 tests/func/test_repro_multistage.py
  49. +520 −627 tests/func/test_run_single_stage.py
  50. +54 −73 tests/func/test_stage.py
  51. +24 −34 tests/func/test_unprotect.py
  52. +1 −1 tests/func/test_version.py
  53. +59 −0 tests/scripts.py
  54. +8 −8 tests/unit/command/test_diff.py
  55. +0 −18 tests/unit/command/test_run.py
  56. +0 −7 tests/unit/command/test_stage.py
  57. +3 −3 tests/unit/dependency/test_dependency.py
  58. +15 −20 tests/unit/output/test_local.py
  59. +7 −7 tests/unit/output/test_output.py
  60. +2 −2 tests/unit/proc/test_process.py
  61. +45 −55 tests/unit/remote/test_base.py
  62. +1 −2 tests/unit/remote/test_http.py
  63. +4 −5 tests/unit/remote/test_webdav.py
  64. +5 −5 tests/unit/remote/test_webhdfs.py
  65. +2 −5 tests/unit/repo/experiments/conftest.py
  66. +3 −3 tests/unit/stage/test_cache.py
  67. +3 −3 tests/unit/test_analytics.py
  68. +1 −1 tests/unit/test_info.py
  69. +4 −5 tests/unit/utils/test_collections.py
  70. +0 −36 tests/utils/scriptify.py
2 changes: 1 addition & 1 deletion .github/workflows/benchmarks.yaml
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ jobs:
steps:
- uses: actions/setup-python@v4
with:
python-version: "3.10"
python-version: "3.11"
- uses: iterative/dvc-bench@main
with:
pytest_options: "-k 'test_init or test_help'"
8 changes: 6 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@

|CI| |Python Version| |Coverage| |VS Code| |DOI|

|PyPI| |Packages| |Brew| |Conda| |Choco| |Snap|
|PyPI| |PyPI Downloads| |Packages| |Brew| |Conda| |Choco| |Snap|

|
@@ -276,6 +276,10 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
:target: https://pypi.org/project/dvc
:alt: PyPI

.. |PyPI Downloads| image:: https://img.shields.io/pypi/dm/dvc.svg?color=blue&label=Downloads&logo=pypi&logoColor=gold
:target: https://pypi.org/project/dvc
:alt: PyPI Downloads

.. |Packages| image:: https://img.shields.io/github/v/release/iterative/dvc?label=deb|pkg|rpm|exe&logo=GitHub
:target: https://github.com/iterative/dvc/releases/latest
:alt: deb|pkg|rpm|exe
@@ -292,6 +296,6 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
:target: https://github.com/iterative/dvc/graphs/contributors
:alt: Contributors

.. |VS Code| image:: https://vsmarketplacebadge.apphb.com/version/Iterative.dvc.svg
.. |VS Code| image:: https://img.shields.io/visual-studio-marketplace/v/Iterative.dvc?color=blue&label=VSCode&logo=visualstudiocode&logoColor=blue
:target: https://marketplace.visualstudio.com/items?itemName=Iterative.dvc
:alt: VS Code Extension
20 changes: 0 additions & 20 deletions dvc/commands/stage.py
Original file line number Diff line number Diff line change
@@ -238,26 +238,6 @@ def _add_common_args(parser):
help="Declare output plot file (do not put into DVC cache).",
metavar="<path>",
)
parser.add_argument(
"--live", help="Declare output as dvclive.", metavar="<path>"
)
parser.add_argument(
"--live-no-cache",
help="Declare output as dvclive (do not put into DVC cache).",
metavar="<path>",
)
parser.add_argument(
"--live-no-summary",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
parser.add_argument(
"--live-no-html",
action="store_true",
default=False,
help="Signal dvclive logger to not produce training report.",
)
parser.add_argument(
"-w",
"--wdir",
10 changes: 6 additions & 4 deletions dvc/dependency/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from collections import defaultdict
from typing import Any, Mapping

from dvc.output import ARTIFACT_SCHEMA, Output
from dvc.output import ARTIFACT_SCHEMA, DIR_FILES_SCHEMA, Output

from .base import Dependency
from .param import ParamsDependency
@@ -15,10 +15,11 @@
**ARTIFACT_SCHEMA,
**RepoDependency.REPO_SCHEMA,
**ParamsDependency.PARAM_SCHEMA,
Output.PARAM_FILES: [DIR_FILES_SCHEMA],
}


def _get(stage, p, info, fs_config=None):
def _get(stage, p, info, **kwargs):
if info and info.get(RepoDependency.PARAM_REPO):
repo = info.pop(RepoDependency.PARAM_REPO)
return RepoDependency(repo, stage, p, info)
@@ -27,14 +28,15 @@ def _get(stage, p, info, fs_config=None):
params = info.pop(ParamsDependency.PARAM_PARAMS)
return ParamsDependency(stage, p, params)

return Dependency(stage, p, info, fs_config=fs_config)
return Dependency(stage, p, info, **kwargs)


def loadd_from(stage, d_list):
ret = []
for d in d_list:
p = d.pop(Output.PARAM_PATH, None)
ret.append(_get(stage, p, d))
files = d.pop(Output.PARAM_FILES, None)
ret.append(_get(stage, p, d, files=files))
return ret


5 changes: 5 additions & 0 deletions dvc/dependency/base.py
Original file line number Diff line number Diff line change
@@ -62,3 +62,8 @@ def save(self):
self.fs_path = self.fs.path.version_path(
self.fs_path, self.meta.version_id
)

def dumpd(self, **kwargs):
if self.fs.version_aware:
kwargs["with_files"] = True
return super().dumpd(**kwargs)
3 changes: 0 additions & 3 deletions dvc/env.py
Original file line number Diff line number Diff line change
@@ -2,9 +2,6 @@
DVC_DAEMON = "DVC_DAEMON"
DVC_PAGER = "DVC_PAGER"
DVC_ROOT = "DVC_ROOT"
DVCLIVE_PATH = "DVCLIVE_PATH"
DVCLIVE_SUMMARY = "DVCLIVE_SUMMARY"
DVCLIVE_HTML = "DVCLIVE_HTML"
DVCLIVE_RESUME = "DVCLIVE_RESUME"
DVC_IGNORE_ISATTY = "DVC_IGNORE_ISATTY"
DVC_EXP_GIT_REMOTE = "DVC_EXP_GIT_REMOTE"
120 changes: 56 additions & 64 deletions dvc/output.py
Original file line number Diff line number Diff line change
@@ -87,7 +87,6 @@ def loadd_from(stage, d_list):
plot = d.pop(Output.PARAM_PLOT, False)
persist = d.pop(Output.PARAM_PERSIST, False)
checkpoint = d.pop(Output.PARAM_CHECKPOINT, False)
live = d.pop(Output.PARAM_LIVE, False)
remote = d.pop(Output.PARAM_REMOTE, None)
annot = {field: d.pop(field, None) for field in ANNOTATION_FIELDS}
files = d.pop(Output.PARAM_FILES, None)
@@ -101,7 +100,6 @@ def loadd_from(stage, d_list):
plot=plot,
persist=persist,
checkpoint=checkpoint,
live=live,
remote=remote,
**annot,
files=files,
@@ -118,7 +116,6 @@ def loads_from(
plot=False,
persist=False,
checkpoint=False,
live=False,
remote=None,
):
return [
@@ -131,7 +128,6 @@ def loads_from(
plot=plot,
persist=persist,
checkpoint=checkpoint,
live=live,
remote=remote,
)
for s in s_list
@@ -166,29 +162,21 @@ def load_from_pipeline(stage, data, typ="outs"):
stage.PARAM_OUTS,
stage.PARAM_METRICS,
stage.PARAM_PLOTS,
stage.PARAM_LIVE,
):
raise ValueError(f"'{typ}' key is not allowed for pipeline files.")

metric = typ == stage.PARAM_METRICS
plot = typ == stage.PARAM_PLOTS
live = typ == stage.PARAM_LIVE
if live:
# `live` is single object
data = [data]

d = _merge_data(data)

for path, flags in d.items():
plt_d, live_d = {}, {}
plt_d = {}
if plot:
from dvc.schema import PLOT_PROPS

plt_d, flags = _split_dict(flags, keys=PLOT_PROPS.keys())
if live:
from dvc.schema import LIVE_PROPS

live_d, flags = _split_dict(flags, keys=LIVE_PROPS.keys())
extra = project(
flags,
[
@@ -206,7 +194,6 @@ def load_from_pipeline(stage, data, typ="outs"):
info={},
plot=plt_d or plot,
metric=metric,
live=live_d or live,
**extra,
)

@@ -263,9 +250,6 @@ class Output:
PARAM_PLOT_TITLE = "title"
PARAM_PLOT_HEADER = "header"
PARAM_PERSIST = "persist"
PARAM_LIVE = "live"
PARAM_LIVE_SUMMARY = "summary"
PARAM_LIVE_HTML = "html"
PARAM_REMOTE = "remote"

METRIC_SCHEMA = Any(
@@ -292,7 +276,6 @@ def __init__(
plot=False,
persist=False,
checkpoint=False,
live=False,
desc=None,
type=None, # pylint: disable=redefined-builtin
labels=None,
@@ -355,7 +338,6 @@ def __init__(
self.plot = False if self.IS_DEPENDENCY else plot
self.persist = persist
self.checkpoint = checkpoint
self.live = live

self.fs_path = self._parse_path(self.fs, fs_path)
self.obj: Optional["HashFile"] = None
@@ -377,8 +359,12 @@ def __init__(
name=self.hash_name,
value=getattr(self.meta, self.hash_name, None),
)
if self.hash_info and self.hash_info.isdir:
if self.meta.nfiles or self.hash_info and self.hash_info.isdir:
self.meta.isdir = True
if not self.hash_info and self.hash_name != "md5":
md5 = getattr(self.meta, "md5", None)
if md5:
self.hash_info = HashInfo("md5", md5)

def _parse_path(self, fs, fs_path):
parsed = urlparse(self.def_path)
@@ -523,14 +509,27 @@ def get_entry(self) -> "DataIndexEntry":
except NoRemoteError:
remote = None

return DataIndexEntry(
if self.files and not self.obj:
self.obj = self.get_obj()

entry = DataIndexEntry(
meta=self.meta,
obj=self.obj,
hash_info=self.hash_info,
odb=self.odb,
cache=self.odb,
remote=remote,
)
if self.stage.is_import and not self.stage.is_repo_import:
dep = self.stage.deps[0]
entry.fs = dep.fs
entry.path = dep.fs_path
entry.meta = dep.meta
if not entry.obj:
if not dep.obj:
dep.obj = dep.get_obj()
entry.obj = dep.obj
return entry

def changed_checksum(self):
return self.hash_info != self.get_hash()
@@ -631,32 +630,28 @@ def save(self):
if self.metric:
self.verify_metric()

if not self.use_cache:
_, self.meta, obj = build(
if self.use_cache:
_, self.meta, self.obj = build(
self.odb,
self.fs_path,
self.fs,
self.hash_name,
ignore=self.dvcignore,
)
else:
_, self.meta, self.obj = build(
self.repo.odb.local,
self.fs_path,
self.fs,
self.hash_name,
ignore=self.dvcignore,
dry_run=True,
)
self.hash_info = obj.hash_info
self.files = None
if not self.IS_DEPENDENCY:
logger.debug(
"Output '%s' doesn't use cache. Skipping saving.", self
)
return

assert not self.IS_DEPENDENCY

_, self.meta, self.obj = build(
self.odb,
self.fs_path,
self.fs,
self.hash_name,
ignore=self.dvcignore,
)
self.hash_info = self.obj.hash_info
self.files = None

@@ -754,40 +749,37 @@ def dumpd(self, **kwargs):

ret[self.PARAM_PATH] = path

if self.IS_DEPENDENCY:
return ret

ret.update(self.annot.to_dict())
if not self.use_cache:
ret[self.PARAM_CACHE] = self.use_cache

if isinstance(self.metric, dict):
if (
self.PARAM_METRIC_XPATH in self.metric
and not self.metric[self.PARAM_METRIC_XPATH]
):
del self.metric[self.PARAM_METRIC_XPATH]
if not self.IS_DEPENDENCY:
ret.update(self.annot.to_dict())
if not self.use_cache:
ret[self.PARAM_CACHE] = self.use_cache

if self.metric:
ret[self.PARAM_METRIC] = self.metric
if isinstance(self.metric, dict):
if (
self.PARAM_METRIC_XPATH in self.metric
and not self.metric[self.PARAM_METRIC_XPATH]
):
del self.metric[self.PARAM_METRIC_XPATH]

if self.plot:
ret[self.PARAM_PLOT] = self.plot
if self.metric:
ret[self.PARAM_METRIC] = self.metric

if self.persist:
ret[self.PARAM_PERSIST] = self.persist
if self.plot:
ret[self.PARAM_PLOT] = self.plot

if self.checkpoint:
ret[self.PARAM_CHECKPOINT] = self.checkpoint
if self.persist:
ret[self.PARAM_PERSIST] = self.persist

if self.live:
ret[self.PARAM_LIVE] = self.live
if self.checkpoint:
ret[self.PARAM_CHECKPOINT] = self.checkpoint

if self.remote:
ret[self.PARAM_REMOTE] = self.remote
if self.remote:
ret[self.PARAM_REMOTE] = self.remote

if self.hash_info.isdir and (
kwargs.get("with_files") or self.files is not None
if (
(not self.IS_DEPENDENCY or self.stage.is_import)
and self.hash_info.isdir
and (kwargs.get("with_files") or self.files is not None)
):
if self.obj:
obj = self.obj
@@ -1182,11 +1174,11 @@ def is_decorated(self) -> bool:

@property
def is_metric(self) -> bool:
return bool(self.metric) or bool(self.live)
return bool(self.metric)

@property
def is_plot(self) -> bool:
return bool(self.plot) or bool(self.live)
return bool(self.plot)


META_SCHEMA = {
2 changes: 1 addition & 1 deletion dvc/repo/__init__.py
Original file line number Diff line number Diff line change
@@ -269,7 +269,7 @@ def scm(self):
except SCMError:
if self._uninitialized:
# might not be a git/dvc repo at all
# used in `params/metrics/plots/live` targets
# used in `params/metrics/plots` targets
return SCM(self.root_dir, no_scm=True)
raise

Loading