Long strings dumped to dvc.lock contain extra space upon load

[janpawlowskiof]

Bug Report

repro: long strings dumped to dvc.lock contain extra space upon load

Edit:

In PR, I replaced the float("inf") with sys.maxsize as suggested in #9397 (comment)

Description

The actual bug seems to be originating from ruamel.yaml, but we should mitigate it here.

How it manifests:

I have this stage indvc.yaml

  faulty_stage:
    cmd: >-
      echo imruneverytime > faulty.txt
    params:
      - fault_parameter_name
    outs:
      - faulty.txt

and this in params.yaml

fault_parameter_name: |
  This is a prompt.
  This is a prompt.
  This is a prompt.
  This is a prompt.
  This is a prompt.

Despite not changing anything, dvc sees the step as changed, but then realizes the step was cached and loads it from cache.

# Running for the first time, expected

bash-5.2$ dvc repro -s faulty_stage
Running stage 'faulty_stage':                                         
> echo imruneverytime > faulty.txt
Updating lock file 'dvc.lock'                                                                                                                                              
Use `dvc push` to send your updates to remote storage.

# Nothing changes here, yet it tries to run, but finally does not, because of the previous run being cached

bash-5.2$ dvc repro -s faulty_stage
Stage 'faulty_stage' is cached - skipping run, checking out outputs   
Updating lock file 'dvc.lock'                                                                                                                                              
Use `dvc push` to send your updates to remote storage.
bash-5.2$ 

Why it occurs:

When long lines are dumped to dvc.lock the line gets wrapped.
The issue happens (only sometimes I think), when this wrap occurs after "\n" character.
When the yaml is then loaded it contains and additional space.

You can see this happen directly in the ruamel.yaml

This can be worked around by adding this line, which makes it so that string is dumped in one line.

yaml.width = float("inf")

You can see this fixes the bug.

How it can be solved.

This can be probably solved be adding this line

yaml.width = float("inf")

here

dvc/dvc/utils/serialize/_yaml.py

Line 48 in 7d14acb

yaml = YAML()

This solves my issue, and while I didn't do any real testing of this, I don't think this should cause any issues elsewhere.

Output of dvc doctor:

DVC version: 3.58.0 (pip)
-------------------------
Platform: Python 3.11.9 on macOS-14.3-arm64-arm-64bit
Subprojects:
        dvc_data = 3.16.7
        dvc_objects = 5.1.0
        dvc_render = 1.0.2
        dvc_task = 0.40.2
        scmrepo = 3.3.9
Supports:
        http (aiohttp = 3.11.11, aiohttp-retry = 2.9.1),
        https (aiohttp = 3.11.11, aiohttp-retry = 2.9.1)
Config:
        Global: /Users/jpawlowski/Library/Application Support/dvc
        System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/c7d0d0e3856141270a471060952e5a84

This will hopefully be a one line fix, and I can add a PR for it today.

Best regards!

[skshetry]

Setting the width might change the existing dvc.lock files. Although setting width may seem to fix it, the issue is really with ruamel-yaml not round-tripping properly.

See https://sourceforge.net/p/ruamel-yaml/tickets/526/.

[janpawlowskiof]

Correct, although since currently there seems to be no work being done ruamel-yaml's side, I sill would consider simply changing YAML's settings to avoid this issue.
Especially since I don't see a way to work around it.
Everytime I do dvc commit I need to confirm that I want to commit since params.yaml changed, when in reality they didn't. This adds a lot of confusion.

As for exisiting dvc.lock:

I'm pretty sure this change will not affect loading existing dvc.lock files.

The first time you recreate dvc.lock with this change, it will create a different dvc.lock, however when loaded, both old and new files should yield identical objects.
So I think it should not cause any troubles?
Let me know if you have doubts, so I can test for them.

[janpawlowskiof]

Hi!
I'm writing, so this does not go stale.
I have been using this branch for a while now with no issues.
So you think the PR can be merged, or does the upstream label mean that this is not te be merged and simply wait for a fix from ruamel-yaml?

[skshetry]

Hey @janpawlowskiof, sorry for the delay in responding. We had a few P0 issues that took up my time.

I'll try to review your PR this week.

My preference here is to fix this upstream if possible, since setting width to inf modifies YAML files, and we aim to keep unnecessary diff minimal. I see this as an upstream bug, as ruamel.yaml's round-trip behavior is broken, given that it was working correctly in earlier versions.

But if there's no easy way to fix this, I'll merge your PR.

[skshetry]

With diff, I mean that if you have something like the following (from above example):

fault_parameter_name: This is a prompt. This is a prompt. This is a prompt. This is
  a prompt. This is a prompt. This is a prompt.

will get changed into following:

fault_parameter_name: This is a prompt. This is a prompt. This is a prompt. This is a prompt. This is a prompt. This is a prompt.

[janpawlowskiof]

That is correct, and I see that creating unexpected diffs in your lockfiles is less than optimal, because it might be slightly confusing.
But other than some confusion, in reality I'm not sure if that is a problem, since the diff will be only made on lockfiles that are being updated anyway, and as far as python code is concerned, the loaded objects will not be changed.

[janpawlowskiof]

Hello again,
I forgot to add my conclusion last time, that since ruamel.yaml incorrectly dumps the strings, any fix to that library upstream, will alter the dvc.lock files anyway.

[skshetry]

This has been fixed with ruamel.yaml>=0.18.11.

https://sourceforge.net/p/ruamel-yaml/tickets/539/