azcopy list - show only top level directories

[vmazalov]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.3.4

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy list

What problem was encountered?

It shows list of all files recursively, but we want to show only list of directories at the top level.

How can we reproduce the problem in the simplest way?

Have you found a mitigation/solution?

[gtiao]

I'd also like to have this feature. Our directories have hundreds of thousands of files in subdirectories, so it makes azcopy ls uninformative.

[dannyman]

👍🏻 it is weird that I can only list recursively. I just want a quick list at the top level.

[dinith95]

yes, I would like to have this feature. I have a set of subdirectories with a large number of files ( thousands maybe millions ) so the plane azcopy list command becomes very time consuming and difficult to deal with.

[joeshull]

+1 here

[delgadom]

bash workaround...

# find top level contents
azcopy ls "https://{store}.blob.core.windows.net/{dir}?SAS" | cut -d/ -f 1 | awk '!a[$0]++'

# find directories with depth N
N=3
azcopy ls "https://{store}.blob.core.windows.net/{dir}?SAS" | cut -d/ -f 1-${N} | awk '!a[$0]++'

[vasanth-bk]

+1 here

[KingLTS]

Same here, I sync many thousands of files with azcopy. The workaround to cut the top-level directories works, but is very slow.

I recommend an optional parameter like "--depth" to determine the required level. But a parameter "--top-level-only" would also work for me.

[KrisJanssen]

Yes, this is sorely needed...

[marc-hb]

sftp:// is the best workaround if you can afford it.
https://learn.microsoft.com/en-us/azure/data-factory/connector-sftp

I recommend the lftp client, it has been working great for us. lftp has a mirror command that has been working much, much better than azcopy sync.

You can of course use any other sftp:// client at the same time, they're not mutually exclusive. And keep using azcopy too if you can find a reason for it (we could not)

Here's a list of other azcopy bugs that lftp + sftp:// does NOT have:

• azcopy copy --recursive --follow-symlink=false ignores symbolic links _silently_, not even a warning #1973, Broken symlinks barely reported (when using --follow-symlink=true) #1974, azcopy remove https://server/area/file_that_does_not_exist does not fail #1975, Do we need to write logs to ~/.azcopy directory? #221, azcopy sync does not follow symlinks #718, recursively removing a tree of empty directories fails #1294, azcopy sync ignoring AZCOPY_BUFFER_GB #1215, Add ability to read SAS tokens from a .netrc-like file #1990, ...

[matthai]

+1

[gabrielwebster]

+1

[marc-hb]

Please delete "+1" comments and use the "thumbs up" button at the top. Unlike "+1" comments, the latter is usable by both machines and humans.

[sizhky]

This is a work around using python's azure-blob-storage sdk

from azure.storage.blob import BlobServiceClient
from azure.storage.blob import BlobPrefix

# Replace these with your actual connection string and container name
connection_string = "your_connection_string"
container_name = "your_container_name"

# Create a BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)

# Get a reference to the container
container_client = blob_service_client.get_container_client(container_name)

# List only the top-level directories (virtual directories)
directories = set()

# Use walk_blobs with delimiter="/" to list virtual directories only
for blob in container_client.walk_blobs(name_starts_with='folder/where/you/want/to/ls/', delimiter='/'):
    if isinstance(blob, BlobPrefix):
        directories.add(blob.name[:-1])  # Remove trailing '/' from the name

# Print the top-level directories
for directory in directories:
    print(directory)

[seanmcc-msft]

Unfortunately, we are not going to add this functionality. We are focused on documentation, testing, and quality, and this feature would represent a large increase in scope for the product.

[KrisJanssen]

‘Completed’

[marc-hb]

Closed reasons are a relatively new feature:
https://github.blog/changelog/2022-05-19-the-new-github-issues-may-19th-update/

That's probably why they have been barely used:
https://github.com/Azure/azure-storage-azcopy/issues?q=is%3Aclosed+-reason%3Acompleted+

#1 was in 2018