dvc add dir doesn't use new files/md5 folder in cache

[BastiQ]

Bug Report

Description

With the new DVC 3 release, files should be cached to .dvc/cache/files/md5 if I understand the release note correctly. But adding a folder results in the files still being stored to .dvc/cache/<random-2digit-hash>.

Reproduce

I run dvc add <folder> and it creates 2e, 3a, 12... folders in the .dvc/cache folder instead of the new .dvc/cache/files/md5 location.

Expected

Instead, shouldn't folder contents also be added to .dvc/cache/files/md5 now? Otherwise, I run into issues with my remote when I want to retrieve the URL using dvc.api.get_url(). It returns s3://<bucket>/files/md5/<random-2digit-hash>/<random-30digit-hash> and when I load that object I run into KeyError: 'md5' since it doesn't exist locally or on S3.

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 3.1.0 (pip)
------------------------
Platform: Python 3.9.15 on macOS-13.4-x86_64-i386-64bit
Subprojects:
        dvc_data = 2.0.2
        dvc_objects = 0.23.0
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.0.2
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.6.0, boto3 = 1.26.76)
Config:
        Global: /Users/basti/Library/Application Support/dvc
        System: /Library/Application Support/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/6f17c3274f593b534e9639e1f1df1d8b

[pmrowla]

@BastiQ can you try the linked PR and verify that it fixes the issue for you?

pip install 'dvc @ git+https://github.com/iterative/dvc.git@refs/pull/9646/head'

[BastiQ]

Hey @pmrowla, thanks for looking into is so fast! I did pip install your checkpoint, but the problem seems to persist.

[pmrowla]

@BastiQ Ah sorry, I forgot to note that you will either need to revert your .dvc files back to their last 2.x created state, or just remove them entirely first (before running dvc add with the bugfix). (This is not normally necessary, they only need to be reverted/removed if you ran dvc add from DVC 3.0 or 3.1 which contained this bug)

[BastiQ]

@pmrowla works now! Thanks