Fix strictyaml line/column numbering #10110
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We understand that certain data types may lack a `lc` property, so we utilize the `key()` and `item()` methods of the `lc` property to obtain the line and column details for such types. Even though we still need to navigate to the parent mapping or sequence to access the `lc` property, these methods enable us to handle diverse data types seamlessly and display accurate line and column information. Fixes iterative#10109 Signed-off-by: hqdncw <hqdncw@gmail.com>
Signed-off-by: hqdncw <hqdncw@gmail.com>
Signed-off-by: hqdncw <hqdncw@gmail.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #10110 +/- ##
==========================================
- Coverage 90.63% 90.53% -0.10%
==========================================
Files 496 500 +4
Lines 37751 37964 +213
Branches 5490 5519 +29
==========================================
+ Hits 34215 34372 +157
- Misses 2900 2947 +47
- Partials 636 645 +9 ☔ View full report in Codecov by Sentry. |
Signed-off-by: hqdncw <hqdncw@gmail.com>
skshetry
reviewed
Nov 27, 2023
|
Any updates @hqdncw? |
Before this pull request can be approved, we need to address two flaky tests in the codebase. The root cause of the issue is that the current implementation doesn't correctly handle cases where the glob pattern is specified for a key that doesn't exist in the given data, leading to a KeyError. This error is being caught by the try/except block on lines 148-153, but it's causing the unit tests to report a false positive (i.e., passing) status. I'll work on fixing this problem tomorrow. Output/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pytest_benchmark/logger.py:46: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
warner(PytestBenchmarkWarning(text))
============================= test session starts ==============================
platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.3.0 -- /home/sid/workspace/dvc/.venv/bin/python3
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/sid/workspace/dvc
configfile: pyproject.toml
plugins: dvc-3.30.1, docker-2.0.1, cases-3.8.1, xdist-3.4.0, mock-3.12.0, benchmark-4.0.0, virtualenv-1.7.0, timeout-2.2.0, flaky-3.7.0, lazy-fixture-0.6.3, hydra-core-1.3.2, shutil-1.7.0, anyio-4.0.0, test-utils-0.0.8, cov-4.1.0
collecting ... collected 20 items
tests/func/utils/test_strict_yaml.py::test_exceptions[duplicate_keys] PASSED [ 5%]
tests/func/utils/test_strict_yaml.py::test_exceptions[mapping_values_not_allowed] PASSED [ 10%]
tests/func/utils/test_strict_yaml.py::test_exceptions[no_hyphen_block] PASSED [ 15%]
tests/func/utils/test_strict_yaml.py::test_exceptions[unclosed_scalar] PASSED [ 20%]
tests/func/utils/test_strict_yaml.py::test_exceptions[not_a_dict] PASSED [ 25%]
tests/func/utils/test_strict_yaml.py::test_exceptions[empty_stage] PASSED [ 30%]
tests/func/utils/test_strict_yaml.py::test_exceptions[missing_cmd] PASSED [ 35%]
tests/func/utils/test_strict_yaml.py::test_exceptions[deps_as_dict] PASSED [ 40%]
tests/func/utils/test_strict_yaml.py::test_exceptions[outs_as_str] PASSED [ 45%]
tests/func/utils/test_strict_yaml.py::test_exceptions[null_value_on_outs] PASSED [ 50%]
tests/func/utils/test_strict_yaml.py::test_exceptions[additional_key_on_outs] PASSED [ 55%]
tests/func/utils/test_strict_yaml.py::test_exceptions[foreach_scalar] PASSED [ 60%]
tests/func/utils/test_strict_yaml.py::test_exceptions[foreach_do_do_null] PASSED [ 65%]
tests/func/utils/test_strict_yaml.py::test_exceptions[foreach_do_missing_cmd] FAILED [ 70%]
tests/func/utils/test_strict_yaml.py::test_exceptions[foreach_unknown_cmd_missing_do] FAILED [ 75%]
tests/func/utils/test_strict_yaml.py::test_exceptions[merge_conflicts] PASSED [ 80%]
tests/func/utils/test_strict_yaml.py::test_on_revision[stages:\n stage1:\n cmd: python train.py\n cmd: python train.py-'./dvc.yaml' is invalid in revision '{short_rev}'.] PASSED [ 85%]
tests/func/utils/test_strict_yaml.py::test_on_revision[stages:\n stage1:\n cmd: {}-'./dvc.yaml' validation failed in revision '{short_rev}'.] PASSED [ 90%]
tests/func/utils/test_strict_yaml.py::test_make_relpath PASSED [ 95%]
tests/func/utils/test_strict_yaml.py::test_fallback_exception_message FAILED [100%]
=================================== FAILURES ===================================
___________________ test_exceptions[foreach_do_missing_cmd] ____________________
data = {'stages': {'stage1': {'do': {'outs': ['${item}']}, 'foreach': [1, 2, 3]}}}
schema = <Schema({'plots': [Any(<class 'str'>, {<class 'str'>: Any({'x': Any(<class 'str'>, {<class 'str'>: <class 'str'>}, msg...EVENT_EXTRA, required=False) object at 0x7f3868d456d0>}, extra=PREVENT_EXTRA, required=False) object at 0x7f3868d6dd90>
text = 'stages:\n stage1:\n foreach: [1,2,3]\n do:\n outs:\n - ${item}'
path = '/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_do_mis0/dvc.yaml'
rev = None
def validate(
data: _T,
schema: Callable[[_T], _T],
text: Optional[str] = None,
path: Optional[str] = None,
rev: Optional[str] = None,
) -> _T:
from voluptuous import MultipleInvalid
try:
> return schema(data)
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:270:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py:287: in __call__
return self._compiled([], data)
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py:611: in validate_dict
return base_validate(path, iteritems(data), out)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = []
iterable = dict_items([('stages', {'stage1': {'foreach': [1, 2, 3], 'do': {'outs': ['${item}']}}})])
out = {}
def validate_mapping(path, iterable, out):
required_keys = all_required_keys.copy()
# Build a map of all provided key-value pairs.
# The type(out) is used to retain ordering in case a ordered
# map type is provided as input.
key_value_map = type(out)()
for key, value in iterable:
key_value_map[key] = value
# Insert default values for non-existing keys.
for key in all_default_keys:
if not isinstance(key.default, Undefined) and \
key.schema not in key_value_map:
# A default value has been specified for this missing
# key, insert it.
key_value_map[key.schema] = key.default()
errors = []
for key, value in key_value_map.items():
key_path = path + [key]
remove_key = False
# Optimization. Validate against the matching key first, then fallback to the rest
relevant_candidates = itertools.chain(candidates_by_key.get(key, []), additional_candidates)
# compare each given key/value against all compiled key/values
# schema key, (compiled key, compiled value)
error = None
for skey, (ckey, cvalue) in relevant_candidates:
try:
new_key = ckey(key_path, key)
except er.Invalid as e:
if len(e.path) > len(key_path):
raise
if not error or len(e.path) > len(error.path):
error = e
continue
# Backtracking is not performed once a key is selected, so if
# the value is invalid we immediately throw an exception.
exception_errors = []
# check if the key is marked for removal
is_remove = new_key is Remove
try:
cval = cvalue(key_path, value)
# include if it's not marked for removal
if not is_remove:
out[new_key] = cval
else:
remove_key = True
continue
except er.MultipleInvalid as e:
exception_errors.extend(e.errors)
except er.Invalid as e:
exception_errors.append(e)
if exception_errors:
if is_remove or remove_key:
continue
for err in exception_errors:
if len(err.path) <= len(key_path):
err.error_type = invalid_msg
errors.append(err)
# If there is a validation error for a required
# key, this means that the key was provided.
# Discard the required key so it does not
# create an additional, noisy exception.
required_keys.discard(skey)
break
# Key and value okay, mark as found in case it was
# a Required() field.
required_keys.discard(skey)
break
else:
if error:
errors.append(error)
elif remove_key:
# remove key
continue
elif self.extra == ALLOW_EXTRA:
out[key] = value
elif self.extra != REMOVE_EXTRA:
errors.append(er.Invalid('extra keys not allowed', key_path))
# else REMOVE_EXTRA: ignore the key so it's removed from output
# for any required keys left that weren't found and don't have defaults:
for key in required_keys:
msg = key.msg if hasattr(key, 'msg') and key.msg else 'required key not provided'
errors.append(er.RequiredFieldInvalid(msg, path + [key]))
if errors:
> raise er.MultipleInvalid(errors)
E voluptuous.error.MultipleInvalid: required key not provided @ data['stages']['stage1']['do']['cmd']
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py:449: MultipleInvalid
The above exception was the direct cause of the following exception:
argv = ['stage', 'list']
def main(argv=None): # noqa: C901, PLR0912, PLR0915
"""Main entry point for dvc CLI.
Args:
argv: optional list of arguments to parse. sys.argv is used by default.
Returns:
int: command's return code.
"""
from dvc._debug import debugtools
from dvc.config import ConfigError
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.logger import set_loggers_level
# NOTE: stderr/stdout may be closed if we are running from dvc.daemon.
# On Linux we directly call cli.main after double forking and closing
# the copied parent's standard file descriptors. If we make any logging
# calls in this state it will cause an exception due to writing to a closed
# file descriptor.
if not sys.stderr or sys.stderr.closed:
logging.disable()
elif not sys.stdout or sys.stdout.closed:
logging.disable(logging.INFO)
args = None
outer_log_level = logger.level
level = None
try:
args = parse_args(argv)
if args.quiet:
level = logging.CRITICAL
elif args.verbose == 1:
level = logging.DEBUG
elif args.verbose > 1:
level = logging.TRACE # type: ignore[attr-defined]
if level is not None:
set_loggers_level(level)
if level and level <= logging.DEBUG:
from platform import platform, python_implementation, python_version
from dvc import PKG, __version__
pyv = f"{python_implementation()} {python_version()}"
pkg = f" ({PKG})" if PKG else ""
logger.debug("v%s%s, %s on %s", __version__, pkg, pyv, platform())
logger.debug("command: %s", " ".join(argv or sys.argv))
logger.trace(args)
if sys.stdout and not sys.stdout.closed and not args.quiet:
from dvc.ui import ui
ui.enable()
with debugtools(args):
cmd = args.func(args)
> ret = cmd.do_run()
/home/sid/workspace/dvc/dvc/cli/__init__.py:211:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/cli/command.py:27: in do_run
return self.run()
/home/sid/workspace/dvc/dvc/commands/stage.py:92: in run
stages = self._get_stages()
/home/sid/workspace/dvc/dvc/commands/stage.py:79: in _get_stages
return dict.fromkeys(collected).keys()
/home/sid/workspace/dvc/dvc/commands/stage.py:76: in <genexpr>
self.repo.stage.collect(target=target, recursive=self.args.recursive)
/home/sid/workspace/dvc/dvc/repo/stage.py:354: in collect
stages = self.from_target(target, glob=glob)
/home/sid/workspace/dvc/dvc/repo/stage.py:208: in from_target
return self.load_all(path=path, name=name, accept_group=accept_group, glob=glob)
/home/sid/workspace/dvc/dvc/repo/stage.py:275: in load_all
stages = dvcfile.stages # type: ignore[attr-defined]
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py:25: in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
/home/sid/workspace/dvc/dvc/dvcfile.py:302: in stages
return self.LOADER(self, self.contents, self.lockfile_contents)
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py:25: in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
/home/sid/workspace/dvc/dvc/dvcfile.py:287: in contents
return self._load()[0]
/home/sid/workspace/dvc/dvc/dvcfile.py:151: in _load
return self._load_yaml(**kwargs)
/home/sid/workspace/dvc/dvc/dvcfile.py:162: in _load_yaml
return strictyaml.load(
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:298: in load
validate(data, schema, text=text, path=path, rev=rev)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
data = {'stages': {'stage1': {'do': {'outs': ['${item}']}, 'foreach': [1, 2, 3]}}}
schema = <Schema({'plots': [Any(<class 'str'>, {<class 'str'>: Any({'x': Any(<class 'str'>, {<class 'str'>: <class 'str'>}, msg...EVENT_EXTRA, required=False) object at 0x7f3868d456d0>}, extra=PREVENT_EXTRA, required=False) object at 0x7f3868d6dd90>
text = 'stages:\n stage1:\n foreach: [1,2,3]\n do:\n outs:\n - ${item}'
path = '/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_do_mis0/dvc.yaml'
rev = None
def validate(
data: _T,
schema: Callable[[_T], _T],
text: Optional[str] = None,
path: Optional[str] = None,
rev: Optional[str] = None,
) -> _T:
from voluptuous import MultipleInvalid
try:
return schema(data)
except MultipleInvalid as exc:
> raise YAMLValidationError(exc, path, text, rev=rev) from exc
E dvc.utils.strictyaml.YAMLValidationError: './dvc.yaml' validation failed
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:272: YAMLValidationError
During handling of the above exception, another exception occurred:
self = <LoggerHandler <stderr> (WARNING)>
record = <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
def emit(self, record):
"""Write to Tqdm's stream so as to not break progress-bars"""
try:
if record.exc_info:
_, exc, *_ = record.exc_info
if hasattr(exc, "__pretty_exc__"):
# try:
> self.emit_pretty_exception(exc, verbose=_is_verbose())
/home/sid/workspace/dvc/dvc/logger.py:149:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/logger.py:140: in emit_pretty_exception
return exc.__pretty_exc__(verbose=verbose)
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:243: in __pretty_exc__
lines.extend(self._prepare_context(data))
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:223: in _prepare_context
line, col = determine_linecol(data, error.path)
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:184: in determine_linecol
if isinstance((obj := get(data, location)), (CommentedSeq, CommentedMap)):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
obj = {'stages': {'stage1': {'foreach': [1, 2, 3], 'do': {'outs': ['${item}']}}}}
glob = ['stages', 'stage1', 'do', 'cmd'], separator = '/'
default = <object object at 0x7f386a6a5d60>
def get(
obj: MutableMapping,
glob: Glob,
separator="/",
default: Any = _DEFAULT_SENTINEL
) -> Union[MutableMapping, object, Callable]:
"""
Given an object which contains only one possible match for the given glob,
return the value for the leaf matching the given glob.
If the glob is not found and a default is provided,
the default is returned.
If more than one leaf matches the glob, ValueError is raised. If the glob is
not found and a default is not provided, KeyError is raised.
"""
if glob == "/":
return obj
globlist = _split_path(glob, separator)
def f(_, pair, results):
(path_segments, found) = pair
if segments.match(path_segments, globlist):
results.append(found)
if len(results) > 1:
return False
results = segments.fold(obj, f, [])
if len(results) == 0:
if default is not _DEFAULT_SENTINEL:
return default
> raise KeyError(glob)
E KeyError: ['stages', 'stage1', 'do', 'cmd']
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dpath/__init__.py:189: KeyError
During handling of the above exception, another exception occurred:
tmp_dir = PosixTmpDir('/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_do_mis0')
dvc = Repo: '/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_do_mis0'
capsys = <_pytest.capture.CaptureFixture object at 0x7f38682d4750>
force_posixpath = None, fixed_width_term = None
text = 'stages:\n stage1:\n foreach: [1,2,3]\n do:\n outs:\n - ${item}'
expected = "'./dvc.yaml' validation failed.\n\nrequired key not provided, in stages -> stage1 -> do -> cmd, line 5, column 7\n 4 │ do:\n 5 │ outs:\n 6 │ - ${item}"
@pytest.mark.parametrize("text, expected", examples.values(), ids=examples.keys())
def test_exceptions(
tmp_dir,
dvc,
capsys,
force_posixpath,
fixed_width_term,
text,
expected,
):
tmp_dir.gen("dvc.yaml", text)
capsys.readouterr() # clear outputs
> assert main(["stage", "list"]) != 0
/home/sid/workspace/dvc/tests/func/utils/test_strict_yaml.py:340:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/cli/__init__.py:232: in main
logger.exception("")
/usr/lib/python3.11/logging/__init__.py:1524: in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
/usr/lib/python3.11/logging/__init__.py:1518: in error
self._log(ERROR, msg, args, **kwargs)
/usr/lib/python3.11/logging/__init__.py:1634: in _log
self.handle(record)
/usr/lib/python3.11/logging/__init__.py:1644: in handle
self.callHandlers(record)
/usr/lib/python3.11/logging/__init__.py:1706: in callHandlers
hdlr.handle(record)
/usr/lib/python3.11/logging/__init__.py:978: in handle
self.emit(record)
/home/sid/workspace/dvc/dvc/logger.py:161: in emit
self.handleError(record)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <LoggerHandler <stderr> (WARNING)>
record = <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
def handleError(self, record): # noqa: N802
super().handleError(record)
> raise LoggingException(record)
E dvc.logger.LoggingException: failed to log <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
/home/sid/workspace/dvc/dvc/logger.py:137: LoggingException
----------------------------- Captured stderr call -----------------------------
--- Logging error ---
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 270, in validate
return schema(data)
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 287, in __call__
return self._compiled([], data)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 611, in validate_dict
return base_validate(path, iteritems(data), out)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 449, in validate_mapping
raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: required key not provided @ data['stages']['stage1']['do']['cmd']
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/cli/command.py", line 27, in do_run
return self.run()
^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 92, in run
stages = self._get_stages()
^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 79, in _get_stages
return dict.fromkeys(collected).keys()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 76, in <genexpr>
self.repo.stage.collect(target=target, recursive=self.args.recursive)
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 354, in collect
stages = self.from_target(target, glob=glob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 208, in from_target
return self.load_all(path=path, name=name, accept_group=accept_group, glob=glob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 275, in load_all
stages = dvcfile.stages # type: ignore[attr-defined]
^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 302, in stages
return self.LOADER(self, self.contents, self.lockfile_contents)
^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 287, in contents
return self._load()[0]
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 151, in _load
return self._load_yaml(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 162, in _load_yaml
return strictyaml.load(
^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 298, in load
validate(data, schema, text=text, path=path, rev=rev)
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 272, in validate
raise YAMLValidationError(exc, path, text, rev=rev) from exc
dvc.utils.strictyaml.YAMLValidationError: './dvc.yaml' validation failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/logger.py", line 149, in emit
self.emit_pretty_exception(exc, verbose=_is_verbose())
File "/home/sid/workspace/dvc/dvc/logger.py", line 140, in emit_pretty_exception
return exc.__pretty_exc__(verbose=verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 243, in __pretty_exc__
lines.extend(self._prepare_context(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 223, in _prepare_context
line, col = determine_linecol(data, error.path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 184, in determine_linecol
if isinstance((obj := get(data, location)), (CommentedSeq, CommentedMap)):
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dpath/__init__.py", line 189, in get
raise KeyError(glob)
KeyError: ['stages', 'stage1', 'do', 'cmd']
Call stack:
File "/home/sid/workspace/dvc/.venv/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
config.hook.pytest_runtestloop(session=session)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/flaky/flaky_pytest_plugin.py", line 94, in pytest_runtest_protocol
self.runner.pytest_runtest_protocol(item, nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/flaky/flaky_pytest_plugin.py", line 138, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261, in call_runtest_hook
return CallInfo.from_call(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/sid/workspace/dvc/tests/func/utils/test_strict_yaml.py", line 340, in test_exceptions
assert main(["stage", "list"]) != 0
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 232, in main
logger.exception("")
File "/usr/lib/python3.11/logging/__init__.py", line 1524, in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
File "/usr/lib/python3.11/logging/__init__.py", line 1518, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/usr/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/usr/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.11/logging/__init__.py", line 978, in handle
self.emit(record)
File "/home/sid/workspace/dvc/dvc/logger.py", line 161, in emit
self.handleError(record)
Message: ''
Arguments: ()
_______________ test_exceptions[foreach_unknown_cmd_missing_do] ________________
data = {'stages': {'stage1': {'cmd': 'python script${item}.py', 'foreach': [1, 2, 3]}}}
schema = <Schema({'plots': [Any(<class 'str'>, {<class 'str'>: Any({'x': Any(<class 'str'>, {<class 'str'>: <class 'str'>}, msg...EVENT_EXTRA, required=False) object at 0x7f3868d456d0>}, extra=PREVENT_EXTRA, required=False) object at 0x7f3868d6dd90>
text = 'stages:\n stage1:\n foreach: [1,2,3]\n cmd: python script${item}.py'
path = '/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_unknow0/dvc.yaml'
rev = None
def validate(
data: _T,
schema: Callable[[_T], _T],
text: Optional[str] = None,
path: Optional[str] = None,
rev: Optional[str] = None,
) -> _T:
from voluptuous import MultipleInvalid
try:
> return schema(data)
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:270:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py:287: in __call__
return self._compiled([], data)
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py:611: in validate_dict
return base_validate(path, iteritems(data), out)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = []
iterable = dict_items([('stages', {'stage1': {'foreach': [1, 2, 3], 'cmd': 'python script${item}.py'}})])
out = {}
def validate_mapping(path, iterable, out):
required_keys = all_required_keys.copy()
# Build a map of all provided key-value pairs.
# The type(out) is used to retain ordering in case a ordered
# map type is provided as input.
key_value_map = type(out)()
for key, value in iterable:
key_value_map[key] = value
# Insert default values for non-existing keys.
for key in all_default_keys:
if not isinstance(key.default, Undefined) and \
key.schema not in key_value_map:
# A default value has been specified for this missing
# key, insert it.
key_value_map[key.schema] = key.default()
errors = []
for key, value in key_value_map.items():
key_path = path + [key]
remove_key = False
# Optimization. Validate against the matching key first, then fallback to the rest
relevant_candidates = itertools.chain(candidates_by_key.get(key, []), additional_candidates)
# compare each given key/value against all compiled key/values
# schema key, (compiled key, compiled value)
error = None
for skey, (ckey, cvalue) in relevant_candidates:
try:
new_key = ckey(key_path, key)
except er.Invalid as e:
if len(e.path) > len(key_path):
raise
if not error or len(e.path) > len(error.path):
error = e
continue
# Backtracking is not performed once a key is selected, so if
# the value is invalid we immediately throw an exception.
exception_errors = []
# check if the key is marked for removal
is_remove = new_key is Remove
try:
cval = cvalue(key_path, value)
# include if it's not marked for removal
if not is_remove:
out[new_key] = cval
else:
remove_key = True
continue
except er.MultipleInvalid as e:
exception_errors.extend(e.errors)
except er.Invalid as e:
exception_errors.append(e)
if exception_errors:
if is_remove or remove_key:
continue
for err in exception_errors:
if len(err.path) <= len(key_path):
err.error_type = invalid_msg
errors.append(err)
# If there is a validation error for a required
# key, this means that the key was provided.
# Discard the required key so it does not
# create an additional, noisy exception.
required_keys.discard(skey)
break
# Key and value okay, mark as found in case it was
# a Required() field.
required_keys.discard(skey)
break
else:
if error:
errors.append(error)
elif remove_key:
# remove key
continue
elif self.extra == ALLOW_EXTRA:
out[key] = value
elif self.extra != REMOVE_EXTRA:
errors.append(er.Invalid('extra keys not allowed', key_path))
# else REMOVE_EXTRA: ignore the key so it's removed from output
# for any required keys left that weren't found and don't have defaults:
for key in required_keys:
msg = key.msg if hasattr(key, 'msg') and key.msg else 'required key not provided'
errors.append(er.RequiredFieldInvalid(msg, path + [key]))
if errors:
> raise er.MultipleInvalid(errors)
E voluptuous.error.MultipleInvalid: extra keys not allowed @ data['stages']['stage1']['cmd']
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py:449: MultipleInvalid
The above exception was the direct cause of the following exception:
argv = ['stage', 'list']
def main(argv=None): # noqa: C901, PLR0912, PLR0915
"""Main entry point for dvc CLI.
Args:
argv: optional list of arguments to parse. sys.argv is used by default.
Returns:
int: command's return code.
"""
from dvc._debug import debugtools
from dvc.config import ConfigError
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.logger import set_loggers_level
# NOTE: stderr/stdout may be closed if we are running from dvc.daemon.
# On Linux we directly call cli.main after double forking and closing
# the copied parent's standard file descriptors. If we make any logging
# calls in this state it will cause an exception due to writing to a closed
# file descriptor.
if not sys.stderr or sys.stderr.closed:
logging.disable()
elif not sys.stdout or sys.stdout.closed:
logging.disable(logging.INFO)
args = None
outer_log_level = logger.level
level = None
try:
args = parse_args(argv)
if args.quiet:
level = logging.CRITICAL
elif args.verbose == 1:
level = logging.DEBUG
elif args.verbose > 1:
level = logging.TRACE # type: ignore[attr-defined]
if level is not None:
set_loggers_level(level)
if level and level <= logging.DEBUG:
from platform import platform, python_implementation, python_version
from dvc import PKG, __version__
pyv = f"{python_implementation()} {python_version()}"
pkg = f" ({PKG})" if PKG else ""
logger.debug("v%s%s, %s on %s", __version__, pkg, pyv, platform())
logger.debug("command: %s", " ".join(argv or sys.argv))
logger.trace(args)
if sys.stdout and not sys.stdout.closed and not args.quiet:
from dvc.ui import ui
ui.enable()
with debugtools(args):
cmd = args.func(args)
> ret = cmd.do_run()
/home/sid/workspace/dvc/dvc/cli/__init__.py:211:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/cli/command.py:27: in do_run
return self.run()
/home/sid/workspace/dvc/dvc/commands/stage.py:92: in run
stages = self._get_stages()
/home/sid/workspace/dvc/dvc/commands/stage.py:79: in _get_stages
return dict.fromkeys(collected).keys()
/home/sid/workspace/dvc/dvc/commands/stage.py:76: in <genexpr>
self.repo.stage.collect(target=target, recursive=self.args.recursive)
/home/sid/workspace/dvc/dvc/repo/stage.py:354: in collect
stages = self.from_target(target, glob=glob)
/home/sid/workspace/dvc/dvc/repo/stage.py:208: in from_target
return self.load_all(path=path, name=name, accept_group=accept_group, glob=glob)
/home/sid/workspace/dvc/dvc/repo/stage.py:275: in load_all
stages = dvcfile.stages # type: ignore[attr-defined]
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py:25: in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
/home/sid/workspace/dvc/dvc/dvcfile.py:302: in stages
return self.LOADER(self, self.contents, self.lockfile_contents)
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py:25: in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
/home/sid/workspace/dvc/dvc/dvcfile.py:287: in contents
return self._load()[0]
/home/sid/workspace/dvc/dvc/dvcfile.py:151: in _load
return self._load_yaml(**kwargs)
/home/sid/workspace/dvc/dvc/dvcfile.py:162: in _load_yaml
return strictyaml.load(
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:298: in load
validate(data, schema, text=text, path=path, rev=rev)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
data = {'stages': {'stage1': {'cmd': 'python script${item}.py', 'foreach': [1, 2, 3]}}}
schema = <Schema({'plots': [Any(<class 'str'>, {<class 'str'>: Any({'x': Any(<class 'str'>, {<class 'str'>: <class 'str'>}, msg...EVENT_EXTRA, required=False) object at 0x7f3868d456d0>}, extra=PREVENT_EXTRA, required=False) object at 0x7f3868d6dd90>
text = 'stages:\n stage1:\n foreach: [1,2,3]\n cmd: python script${item}.py'
path = '/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_unknow0/dvc.yaml'
rev = None
def validate(
data: _T,
schema: Callable[[_T], _T],
text: Optional[str] = None,
path: Optional[str] = None,
rev: Optional[str] = None,
) -> _T:
from voluptuous import MultipleInvalid
try:
return schema(data)
except MultipleInvalid as exc:
> raise YAMLValidationError(exc, path, text, rev=rev) from exc
E dvc.utils.strictyaml.YAMLValidationError: './dvc.yaml' validation failed: 2 errors
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:272: YAMLValidationError
During handling of the above exception, another exception occurred:
self = <LoggerHandler <stderr> (WARNING)>
record = <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
def emit(self, record):
"""Write to Tqdm's stream so as to not break progress-bars"""
try:
if record.exc_info:
_, exc, *_ = record.exc_info
if hasattr(exc, "__pretty_exc__"):
# try:
> self.emit_pretty_exception(exc, verbose=_is_verbose())
/home/sid/workspace/dvc/dvc/logger.py:149:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/logger.py:140: in emit_pretty_exception
return exc.__pretty_exc__(verbose=verbose)
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:243: in __pretty_exc__
lines.extend(self._prepare_context(data))
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:223: in _prepare_context
line, col = determine_linecol(data, error.path)
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:184: in determine_linecol
if isinstance((obj := get(data, location)), (CommentedSeq, CommentedMap)):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
obj = {'stages': {'stage1': {'foreach': [1, 2, 3], 'cmd': 'python script${item}.py'}}}
glob = ['stages', 'stage1', 'do'], separator = '/'
default = <object object at 0x7f386a6a5d60>
def get(
obj: MutableMapping,
glob: Glob,
separator="/",
default: Any = _DEFAULT_SENTINEL
) -> Union[MutableMapping, object, Callable]:
"""
Given an object which contains only one possible match for the given glob,
return the value for the leaf matching the given glob.
If the glob is not found and a default is provided,
the default is returned.
If more than one leaf matches the glob, ValueError is raised. If the glob is
not found and a default is not provided, KeyError is raised.
"""
if glob == "/":
return obj
globlist = _split_path(glob, separator)
def f(_, pair, results):
(path_segments, found) = pair
if segments.match(path_segments, globlist):
results.append(found)
if len(results) > 1:
return False
results = segments.fold(obj, f, [])
if len(results) == 0:
if default is not _DEFAULT_SENTINEL:
return default
> raise KeyError(glob)
E KeyError: ['stages', 'stage1', 'do']
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dpath/__init__.py:189: KeyError
During handling of the above exception, another exception occurred:
tmp_dir = PosixTmpDir('/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_unknow0')
dvc = Repo: '/tmp/pytest-of-sid/pytest-468/test_exceptions_foreach_unknow0'
capsys = <_pytest.capture.CaptureFixture object at 0x7f3867f8ff10>
force_posixpath = None, fixed_width_term = None
text = 'stages:\n stage1:\n foreach: [1,2,3]\n cmd: python script${item}.py'
expected = "'./dvc.yaml' validation failed: 2 errors.\n\nextra keys not allowed, in stages -> stage1 -> cmd, line 3, column 5\n ...tages -> stage1 -> do, line 3, column 5\n 2 stage1:\n 3 │ foreach: [1,2,3]\n 4 │ cmd: python script${item}.py"
@pytest.mark.parametrize("text, expected", examples.values(), ids=examples.keys())
def test_exceptions(
tmp_dir,
dvc,
capsys,
force_posixpath,
fixed_width_term,
text,
expected,
):
tmp_dir.gen("dvc.yaml", text)
capsys.readouterr() # clear outputs
> assert main(["stage", "list"]) != 0
/home/sid/workspace/dvc/tests/func/utils/test_strict_yaml.py:340:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/cli/__init__.py:232: in main
logger.exception("")
/usr/lib/python3.11/logging/__init__.py:1524: in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
/usr/lib/python3.11/logging/__init__.py:1518: in error
self._log(ERROR, msg, args, **kwargs)
/usr/lib/python3.11/logging/__init__.py:1634: in _log
self.handle(record)
/usr/lib/python3.11/logging/__init__.py:1644: in handle
self.callHandlers(record)
/usr/lib/python3.11/logging/__init__.py:1706: in callHandlers
hdlr.handle(record)
/usr/lib/python3.11/logging/__init__.py:978: in handle
self.emit(record)
/home/sid/workspace/dvc/dvc/logger.py:161: in emit
self.handleError(record)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <LoggerHandler <stderr> (WARNING)>
record = <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
def handleError(self, record): # noqa: N802
super().handleError(record)
> raise LoggingException(record)
E dvc.logger.LoggingException: failed to log <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
/home/sid/workspace/dvc/dvc/logger.py:137: LoggingException
----------------------------- Captured stderr call -----------------------------
--- Logging error ---
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 270, in validate
return schema(data)
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 287, in __call__
return self._compiled([], data)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 611, in validate_dict
return base_validate(path, iteritems(data), out)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 449, in validate_mapping
raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: extra keys not allowed @ data['stages']['stage1']['cmd']
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/cli/command.py", line 27, in do_run
return self.run()
^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 92, in run
stages = self._get_stages()
^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 79, in _get_stages
return dict.fromkeys(collected).keys()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 76, in <genexpr>
self.repo.stage.collect(target=target, recursive=self.args.recursive)
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 354, in collect
stages = self.from_target(target, glob=glob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 208, in from_target
return self.load_all(path=path, name=name, accept_group=accept_group, glob=glob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 275, in load_all
stages = dvcfile.stages # type: ignore[attr-defined]
^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 302, in stages
return self.LOADER(self, self.contents, self.lockfile_contents)
^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 287, in contents
return self._load()[0]
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 151, in _load
return self._load_yaml(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 162, in _load_yaml
return strictyaml.load(
^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 298, in load
validate(data, schema, text=text, path=path, rev=rev)
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 272, in validate
raise YAMLValidationError(exc, path, text, rev=rev) from exc
dvc.utils.strictyaml.YAMLValidationError: './dvc.yaml' validation failed: 2 errors
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/logger.py", line 149, in emit
self.emit_pretty_exception(exc, verbose=_is_verbose())
File "/home/sid/workspace/dvc/dvc/logger.py", line 140, in emit_pretty_exception
return exc.__pretty_exc__(verbose=verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 243, in __pretty_exc__
lines.extend(self._prepare_context(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 223, in _prepare_context
line, col = determine_linecol(data, error.path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 184, in determine_linecol
if isinstance((obj := get(data, location)), (CommentedSeq, CommentedMap)):
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dpath/__init__.py", line 189, in get
raise KeyError(glob)
KeyError: ['stages', 'stage1', 'do']
Call stack:
File "/home/sid/workspace/dvc/.venv/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
config.hook.pytest_runtestloop(session=session)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/flaky/flaky_pytest_plugin.py", line 94, in pytest_runtest_protocol
self.runner.pytest_runtest_protocol(item, nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/flaky/flaky_pytest_plugin.py", line 138, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261, in call_runtest_hook
return CallInfo.from_call(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/sid/workspace/dvc/tests/func/utils/test_strict_yaml.py", line 340, in test_exceptions
assert main(["stage", "list"]) != 0
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 232, in main
logger.exception("")
File "/usr/lib/python3.11/logging/__init__.py", line 1524, in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
File "/usr/lib/python3.11/logging/__init__.py", line 1518, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/usr/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/usr/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.11/logging/__init__.py", line 978, in handle
self.emit(record)
File "/home/sid/workspace/dvc/dvc/logger.py", line 161, in emit
self.handleError(record)
Message: ''
Arguments: ()
_______________________ test_fallback_exception_message ________________________
errors = <class 'ruamel.yaml.error.YAMLError'>
into = YAMLFileCorruptedError("unable to read: 'dvc.yaml', YAML file structure is corrupted")
@contextmanager
def reraise(errors, into):
"""Reraises errors as other exception."""
errors = _ensure_exceptable(errors)
try:
> yield
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/flow.py:84:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/utils/serialize/_yaml.py:30: in parse_yaml
return yaml.load(text) or {}
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/ruamel/yaml/main.py:451: in load
return constructor.get_single_data()
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/ruamel/yaml/constructor.py:112: in get_single_data
node = self.composer.get_single_node()
_ruamel_yaml.pyx:706: in ruamel.yaml.clib._ruamel_yaml.CParser.get_single_node
???
_ruamel_yaml.pyx:724: in ruamel.yaml.clib._ruamel_yaml.CParser._compose_document
???
_ruamel_yaml.pyx:775: in ruamel.yaml.clib._ruamel_yaml.CParser._compose_node
???
_ruamel_yaml.pyx:891: in ruamel.yaml.clib._ruamel_yaml.CParser._compose_mapping_node
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E ruamel.yaml.scanner.ScannerError: mapping values are not allowed in this context
E in "<unicode string>", line 3, column 8
_ruamel_yaml.pyx:904: ScannerError
The above exception was the direct cause of the following exception:
argv = ['stage', 'list']
def main(argv=None): # noqa: C901, PLR0912, PLR0915
"""Main entry point for dvc CLI.
Args:
argv: optional list of arguments to parse. sys.argv is used by default.
Returns:
int: command's return code.
"""
from dvc._debug import debugtools
from dvc.config import ConfigError
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.logger import set_loggers_level
# NOTE: stderr/stdout may be closed if we are running from dvc.daemon.
# On Linux we directly call cli.main after double forking and closing
# the copied parent's standard file descriptors. If we make any logging
# calls in this state it will cause an exception due to writing to a closed
# file descriptor.
if not sys.stderr or sys.stderr.closed:
logging.disable()
elif not sys.stdout or sys.stdout.closed:
logging.disable(logging.INFO)
args = None
outer_log_level = logger.level
level = None
try:
args = parse_args(argv)
if args.quiet:
level = logging.CRITICAL
elif args.verbose == 1:
level = logging.DEBUG
elif args.verbose > 1:
level = logging.TRACE # type: ignore[attr-defined]
if level is not None:
set_loggers_level(level)
if level and level <= logging.DEBUG:
from platform import platform, python_implementation, python_version
from dvc import PKG, __version__
pyv = f"{python_implementation()} {python_version()}"
pkg = f" ({PKG})" if PKG else ""
logger.debug("v%s%s, %s on %s", __version__, pkg, pyv, platform())
logger.debug("command: %s", " ".join(argv or sys.argv))
logger.trace(args)
if sys.stdout and not sys.stdout.closed and not args.quiet:
from dvc.ui import ui
ui.enable()
with debugtools(args):
cmd = args.func(args)
> ret = cmd.do_run()
/home/sid/workspace/dvc/dvc/cli/__init__.py:211:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/cli/command.py:27: in do_run
return self.run()
/home/sid/workspace/dvc/dvc/commands/stage.py:92: in run
stages = self._get_stages()
/home/sid/workspace/dvc/dvc/commands/stage.py:79: in _get_stages
return dict.fromkeys(collected).keys()
/home/sid/workspace/dvc/dvc/commands/stage.py:76: in <genexpr>
self.repo.stage.collect(target=target, recursive=self.args.recursive)
/home/sid/workspace/dvc/dvc/repo/stage.py:354: in collect
stages = self.from_target(target, glob=glob)
/home/sid/workspace/dvc/dvc/repo/stage.py:208: in from_target
return self.load_all(path=path, name=name, accept_group=accept_group, glob=glob)
/home/sid/workspace/dvc/dvc/repo/stage.py:275: in load_all
stages = dvcfile.stages # type: ignore[attr-defined]
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py:25: in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
/home/sid/workspace/dvc/dvc/dvcfile.py:302: in stages
return self.LOADER(self, self.contents, self.lockfile_contents)
/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py:25: in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
/home/sid/workspace/dvc/dvc/dvcfile.py:287: in contents
return self._load()[0]
/home/sid/workspace/dvc/dvc/dvcfile.py:151: in _load
return self._load_yaml(**kwargs)
/home/sid/workspace/dvc/dvc/dvcfile.py:162: in _load_yaml
return strictyaml.load(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = '/tmp/pytest-of-sid/pytest-468/test_fallback_exception_messag0/dvc.yaml'
schema = <Schema({'plots': [Any(<class 'str'>, {<class 'str'>: Any({'x': Any(<class 'str'>, {<class 'str'>: <class 'str'>}, msg...EVENT_EXTRA, required=False) object at 0x7f3868d456d0>}, extra=PREVENT_EXTRA, required=False) object at 0x7f3868d6dd90>
fs = <dvc_objects.fs.local.LocalFileSystem object at 0x7f386a7ce450>
encoding = 'utf-8', round_trip = False
def load(
path: str,
schema: Optional[Callable[[_T], _T]] = None,
fs: Optional["FileSystem"] = None,
encoding: str = "utf-8",
round_trip: bool = False,
) -> Any:
open_fn = fs.open if fs else open
rev = getattr(fs, "rev", None)
try:
with open_fn(path, encoding=encoding) as fd: # type: ignore[operator]
text = fd.read()
data = parse_yaml(text, path, typ="rt" if round_trip else "safe")
except UnicodeDecodeError as exc:
raise EncodingError(path, encoding) from exc
except YAMLFileCorruptedError as exc:
cause = exc.__cause__
> raise YAMLSyntaxError(path, text, exc, rev=rev) from cause
E dvc.utils.strictyaml.YAMLSyntaxError: unable to read: 'dvc.yaml', YAML file structure is corrupted
/home/sid/workspace/dvc/dvc/utils/strictyaml.py:293: YAMLSyntaxError
During handling of the above exception, another exception occurred:
self = <LoggerHandler <stderr> (WARNING)>
record = <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
def emit(self, record):
"""Write to Tqdm's stream so as to not break progress-bars"""
try:
if record.exc_info:
_, exc, *_ = record.exc_info
if hasattr(exc, "__pretty_exc__"):
# try:
> self.emit_pretty_exception(exc, verbose=_is_verbose())
/home/sid/workspace/dvc/dvc/logger.py:149:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/logger.py:140: in emit_pretty_exception
return exc.__pretty_exc__(verbose=verbose)
/usr/lib/python3.11/unittest/mock.py:1118: in __call__
return self._mock_call(*args, **kwargs)
/usr/lib/python3.11/unittest/mock.py:1122: in _mock_call
return self._execute_mock_call(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <MagicMock name='__pretty_exc__' id='139880243567120'>, args = ()
kwargs = {'verbose': True}, effect = <class 'ValueError'>
def _execute_mock_call(self, /, *args, **kwargs):
# separate from _increment_mock_call so that awaited functions are
# executed separately from their call, also AsyncMock overrides this method
effect = self.side_effect
if effect is not None:
if _is_exception(effect):
> raise effect
E ValueError
/usr/lib/python3.11/unittest/mock.py:1177: ValueError
During handling of the above exception, another exception occurred:
tmp_dir = PosixTmpDir('/tmp/pytest-of-sid/pytest-468/test_fallback_exception_messag0')
dvc = Repo: '/tmp/pytest-of-sid/pytest-468/test_fallback_exception_messag0'
mocker = <pytest_mock.plugin.MockerFixture object at 0x7f3867cc3850>
caplog = <_pytest.logging.LogCaptureFixture object at 0x7f3867f8e990>
def test_fallback_exception_message(tmp_dir, dvc, mocker, caplog):
# When trying to pretty print exception messages, we fallback to old way
# of printing things.
mocker.patch(
"dvc.utils.strictyaml.YAMLSyntaxError.__pretty_exc__",
side_effect=ValueError,
)
mocker.patch(
"dvc.utils.strictyaml.YAMLValidationError.__pretty_exc__",
side_effect=ValueError,
)
# syntax errors
dvc_file = tmp_dir / "dvc.yaml"
dvc_file.write_text(MAPPING_VALUES_NOT_ALLOWED)
> assert main(["stage", "list"]) != 0
/home/sid/workspace/dvc/tests/func/utils/test_strict_yaml.py:416:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/sid/workspace/dvc/dvc/cli/__init__.py:232: in main
logger.exception("")
/usr/lib/python3.11/logging/__init__.py:1524: in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
/usr/lib/python3.11/logging/__init__.py:1518: in error
self._log(ERROR, msg, args, **kwargs)
/usr/lib/python3.11/logging/__init__.py:1634: in _log
self.handle(record)
/usr/lib/python3.11/logging/__init__.py:1644: in handle
self.callHandlers(record)
/usr/lib/python3.11/logging/__init__.py:1706: in callHandlers
hdlr.handle(record)
/usr/lib/python3.11/logging/__init__.py:978: in handle
self.emit(record)
/home/sid/workspace/dvc/dvc/logger.py:161: in emit
self.handleError(record)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <LoggerHandler <stderr> (WARNING)>
record = <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
def handleError(self, record): # noqa: N802
super().handleError(record)
> raise LoggingException(record)
E dvc.logger.LoggingException: failed to log <LogRecord: dvc.cli, 40, /home/sid/workspace/dvc/dvc/cli/__init__.py, 232, "">
/home/sid/workspace/dvc/dvc/logger.py:137: LoggingException
----------------------------- Captured stderr call -----------------------------
--- Logging error ---
Traceback (most recent call last):
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/flow.py", line 84, in reraise
yield
File "/home/sid/workspace/dvc/dvc/utils/serialize/_yaml.py", line 30, in parse_yaml
return yaml.load(text) or {}
^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/ruamel/yaml/main.py", line 451, in load
return constructor.get_single_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/ruamel/yaml/constructor.py", line 112, in get_single_data
node = self.composer.get_single_node()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "_ruamel_yaml.pyx", line 706, in ruamel.yaml.clib._ruamel_yaml.CParser.get_single_node
File "_ruamel_yaml.pyx", line 724, in ruamel.yaml.clib._ruamel_yaml.CParser._compose_document
File "_ruamel_yaml.pyx", line 775, in ruamel.yaml.clib._ruamel_yaml.CParser._compose_node
File "_ruamel_yaml.pyx", line 891, in ruamel.yaml.clib._ruamel_yaml.CParser._compose_mapping_node
File "_ruamel_yaml.pyx", line 904, in ruamel.yaml.clib._ruamel_yaml.CParser._parse_next_event
ruamel.yaml.scanner.ScannerError: mapping values are not allowed in this context
in "<unicode string>", line 3, column 8
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/cli/command.py", line 27, in do_run
return self.run()
^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 92, in run
stages = self._get_stages()
^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 79, in _get_stages
return dict.fromkeys(collected).keys()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/stage.py", line 76, in <genexpr>
self.repo.stage.collect(target=target, recursive=self.args.recursive)
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 354, in collect
stages = self.from_target(target, glob=glob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 208, in from_target
return self.load_all(path=path, name=name, accept_group=accept_group, glob=glob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/stage.py", line 275, in load_all
stages = dvcfile.stages # type: ignore[attr-defined]
^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 302, in stages
return self.LOADER(self, self.contents, self.lockfile_contents)
^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 287, in contents
return self._load()[0]
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 151, in _load
return self._load_yaml(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/dvcfile.py", line 162, in _load_yaml
return strictyaml.load(
^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/utils/strictyaml.py", line 293, in load
raise YAMLSyntaxError(path, text, exc, rev=rev) from cause
dvc.utils.strictyaml.YAMLSyntaxError: unable to read: 'dvc.yaml', YAML file structure is corrupted
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/logger.py", line 149, in emit
self.emit_pretty_exception(exc, verbose=_is_verbose())
File "/home/sid/workspace/dvc/dvc/logger.py", line 140, in emit_pretty_exception
return exc.__pretty_exc__(verbose=verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/unittest/mock.py", line 1118, in __call__
return self._mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/unittest/mock.py", line 1122, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/unittest/mock.py", line 1177, in _execute_mock_call
raise effect
ValueError
Call stack:
File "/home/sid/workspace/dvc/.venv/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
config.hook.pytest_runtestloop(session=session)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/flaky/flaky_pytest_plugin.py", line 94, in pytest_runtest_protocol
self.runner.pytest_runtest_protocol(item, nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/flaky/flaky_pytest_plugin.py", line 138, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261, in call_runtest_hook
return CallInfo.from_call(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/sid/workspace/dvc/tests/func/utils/test_strict_yaml.py", line 416, in test_fallback_exception_message
assert main(["stage", "list"]) != 0
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 232, in main
logger.exception("")
File "/usr/lib/python3.11/logging/__init__.py", line 1524, in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
File "/usr/lib/python3.11/logging/__init__.py", line 1518, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/usr/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/usr/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.11/logging/__init__.py", line 978, in handle
self.emit(record)
File "/home/sid/workspace/dvc/dvc/logger.py", line 161, in emit
self.handleError(record)
Message: ''
Arguments: ()
=========================== short test summary info ============================
FAILED tests/func/utils/test_strict_yaml.py::test_exceptions[foreach_do_missing_cmd]
FAILED tests/func/utils/test_strict_yaml.py::test_exceptions[foreach_unknown_cmd_missing_do]
FAILED tests/func/utils/test_strict_yaml.py::test_fallback_exception_message
========================= 3 failed, 17 passed in 2.72s ========================= |
Signed-off-by: hqdncw <hqdncw@gmail.com>
Signed-off-by: hqdncw <hqdncw@gmail.com>
Signed-off-by: hqdncw <hqdncw@gmail.com>
hqdncw
changed the title
skshetry
reviewed
Dec 26, 2023
Comment on lines
+240
to
+250
| line, col = determine_linecol( | ||
| data, | ||
| error.path, | ||
| # Handle the case where a validation error indicates that additional | ||
| # keys are not permitted. | ||
| key_or_value="extra keys not allowed" in error.error_message, | ||
| ) | ||
| except KeyError: | ||
| # Handle the case where a validation error occurs because a required | ||
| # key is missing. | ||
| line, col = determine_linecol(data, error.path[:-1], True) |
There was a problem hiding this comment.
I don't want this to be coupled with dvcfile. We should just move up the path if we don't find the keys.
skshetry
reviewed
Dec 26, 2023
| @@ -13,7 +13,7 @@ repos: | |||
| - id: check-executables-have-shebangs | |||
| - id: check-json | |||
| - id: check-merge-conflict | |||
| exclude: "tests/func/utils/test_strict_yaml.py" | |||
| exclude: "tests/func/test_dvcfile_validation.py" | |||
There was a problem hiding this comment.
Let's not move the tests please.
There was a problem hiding this comment.
Let's not move the tests please.
I apologize for any confusion caused by my previous commit. My intention was to keep the strictyaml module unaware of any details related to dvcfile, which is why I moved all functional tests concerning dvcfile validation to the tests/func/test_dvcfile_validation.py file. If you're confident that leaving the old name is the best approach, please let me know.
There was a problem hiding this comment.
In DVC, dvc.yaml validation is the only consumer of this. Any other tests could be considered as unit tests and may belong in tests/unit. The tests in test_strictyaml are graphical rather than acutal validation, so test_dvcfile_validation is not a good name either.
Anyway, let's not mix different things in the same PR.
skshetry
reviewed
Dec 26, 2023
Comment on lines
+148
to
+150
| self.emit_pretty_exception(exc, verbose=_is_verbose()) | ||
| if not _is_verbose(): | ||
| return |
There was a problem hiding this comment.
I don't trust the handling in strictyaml enough, that this should still fallback to other messages.
There was a problem hiding this comment.
I don't trust the handling in
strictyamlenough, that this should still fallback to other messages.
Thanks for raising your concerns about the code. I understand your point that we should consider including a fallback option for situations where the strictyaml module might not work correctly. To address this issue, I've created a separate ticket (#10201) so that we can discuss and implement a solution without muddying the current pull request with unrelated changes. Let's continue the conversation there.
skshetry
reviewed
Dec 26, 2023
Comment on lines
+200
to
+204
| return _normalize_linecol( | ||
| obj.lc.key(glob_pattern[-1]) | ||
| if key_or_value | ||
| else obj.lc.value(glob_pattern[-1]) | ||
| ) |
There was a problem hiding this comment.
I don't think there is anything wrong with returning both key and value linecols all the time. We can show a range of line numbers.
skshetry
reviewed
Dec 26, 2023
Comment on lines
+208
to
+210
| raise TypeError( | ||
| f"Expected commented seq or map, got {type(obj)} at path {glob_pattern!r}" | ||
| ) |
There was a problem hiding this comment.
Note there may be cases where we don't have paths. So this should just return linecol from data in that case.
Fixes iterative#10109 Signed-off-by: hqdncw <hqdncw@gmail.com>
|
Closing as stale. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODO:
Related #10109