Not able to move files to respective folders in ADLS blob storage

[mavijayk]

Dear experts,

I'm facing an error in python code while executing below script, please find below my requirement and do let me know what's wrong in my code.

Requirement:

1. Connect to ADLS blob storage
2. Search for multiple patterns in specific source path for json files ONLY
3. move those files to respective destination folders based on pattern match

python script:

import re
import json
import os
from azure.storage.filedatalake import (
    DataLakeServiceClient,
    FileSystemClient
)
from azure.storage.blob import BlobServiceClient
import datetime
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
from io import StringIO

timestamp = datetime.datetime.now().strftime("%Y-%m-%d:%H:%M:%S")

def generate_log(message):
    #timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    log_file_name = f'C:\\Users\\Vijay\\Scripts\\script.log'

    with open(log_file_name, 'a') as log_file:
        log_file.write(f'{message}\n')
        
def authenticate_storage_account():
    try:
        global service_client
        service_client = DataLakeServiceClient(account_url="https://myhome.dfs.core.windows.net".format(account_name), credential=account_key)
        print("Connected to ADLS Gen2 Blob Storage: ")
        generate_log(f"{timestamp}:- Connected to ADLS Gen2 Blob Storage")
    except Exception as e:
        print("Error While Connectiong ADLS Gen2 Blob Storage: ", e)
        generate_log(f"{timestamp}:- Error While Connectiong ADLS Gen2 Blob Storage: {e}")

patterns_and_destinations = [
    {"pattern": r"responseBody.*new-application", "destination": "//home//Test1"},
    {"pattern": r"requestBody.*new-application", "destination": "//home//Test2"}
]

def move_files_to_destination(source_directory_path, destination_directory_path, content_pattern):
    try:
        print("Initiated search pattern and moving event process...")
        generate_log(f"{timestamp}:- Initiated search pattern and moving event process...")
        try:
            source_directory_client = service_client.get_directory_client(directory=source_directory_path,file_system='raw')
            print("Searching Source directory...")
            generate_log(f"{timestamp}:- Searching Source directory...")
        except Exception as e:
            print("Error: Searching Source directory...")
            generate_log(f"{timestamp}:- Error: Searching Source directory...",e)
            try:
                destination_directory_client = service_client.get_directory_client(directory=destination_directory_path,file_system='raw')
                print("Destination directory...")
                generate_log(f"{timestamp}:- Destination directory...")
            except Exception as e:
                print("Error: Destination Source directory...")
                generate_log(f"{timestamp}:- Error: Destination Searching Source directory...",e)
        
        file_list = source_directory_client.list_files()
        
        for file in file_list:
            if file.is_directory:
                continue
            file_client = service_client.get_file_client(file.path)
            download = file_client.download_file()
            content = download.readall()
            
            if file.name.endswith(".json") and re.search(content_pattern, content.decode('utf-8')):
                destination_file_client = destination_directory_client.create_file(file.name)
                print(f"Moving file {file.name} to {destination_directory_path}")
                generate_log(f"{timestamp}:- Moving file {file.name} to {destination_directory_path}")
                destination_file_client.upload_data(StringIO(content.decode('utf-8')), overwrite=True)
                file_client.delete_file(file.name)
    except Exception as e:
        print("Error occurred while moving file: ", e)
        generate_log(f"{timestamp}:- Error occurred while moving file: {e}")


account_name = "homeadls"
account_key = "*********"

source_directory_path = "//home//POC"
destination_directory_path="//home//Error"
#authenticate function calling
authenticate_storage_account()

for pattern_and_destination in patterns_and_destinations:
    move_files_to_destination(source_directory_path, pattern_and_destination["destination"], pattern_and_destination["pattern"])

Error:
DataLakeDirectoryClient' object has no attribute 'list_files'

I tried with different options from below failing with same error like above with "respective change I have done"

file_list = source_directory_client.list_directories_and_files()
file_list = source_directory_client.get_subdirectory_client('').list_directories_and_files()
file_list = source_directory_client.get_files_and_directories()

and if I give like this script executing successfully but files are not moving to destination folders

file_list = source_directory_client.get_directory_properties()

Please suggest what is wrong in my code?

[mavijayk]

@jdesrosiers any help please?

[gregsdennis]

@mavijayk I don't think you're in the right place. This organization manages JSON Schema, and your question doesn't appear to be related.

[jdesrosiers]

I'm going to close this as off-topic for this space, but good luck finding an answer to your question. I'm not sure if this is more of a python question or an Azure DataLake question, but it's not something we have expertise in here. This space is for JSON Schema questions. You might have more luck asking your question in another space dedicated to python or Azure.

[mavijayk]

Yah! Thanks for the response, this issue resolved.