vsiaz working on VM, but not on AKS HTTPS endpoint scoring file

[dimattap]

I am building an app which takes latitude and longitude as inputs. I need to fetch the nearest pixel data of my input from a (large) tif file located in Azure blob storage, pass that data to ML model and results a prediction.
On my Azure VM, I tried the below code:

import gdal
gdal.SetConfigOption('AZURE_STORAGE_ACCOUNT', accountname)
gdal.SetConfigOption('AZURE_SAS',sast[1:])
a = gdal.Open('/vsiaz/container/folder/file.tif')
a.ReadAsArray(0,0,1,1)

This is working fine on the VM and working as expected.
When I try to use the same code in the scoring file in the AKS HTTPS endpoint deployment, I'm getting error:
"ERROR 4: /vsiaz/container/folder/file.tif": No such file or directory

Tried the same by using AZURE_STORAGE_CONNECTION_STRING instead of SAS key, still same response.
Example tif can be accessed from here.

Kindly help me as this is a critical component of my project.

[dimattap]

Hello GDAL team,
I tried changing the Azure storage container access to public.
Used the below code on a VM, but that is not working

gdal.SetConfigOption('AZURE_STORAGE_ACCOUNT', accountname)
gdal.SetConfigOption('AZURE_NO_SIGN_REQUEST', 'YES')
a = gdal.Open('/vsiaz/container/folder/file.tif')
type(a)

type of variable a is returning as NoneType
Am I missing something?
Kindly help

[rouault]

I've never used AKS, but it might perhaps be a permission issue on your AKS setup and/or blob storage ? https://stackoverflow.com/questions/56349424/accessing-azure-storage-blob-from-an-aks-cluster might perhaps help

[dimattap]

Many thanks for your response.
Later while trying out all other options, I made the container public and tried to access the files using vsicurl.
That was working, but when parallelized with 100 workers, Im getting issues similar to here and here.
I was trying the ThreadPoolExcecutor from concurrent.futures in python for parallisation.

[rouault]

but when parallelized with 100 workers, Im getting issues similar to here and here.

you'll have to be much more specific and show the exact errors you get and a snippet of your code

[dimattap]

but when parallelized with 100 workers, Im getting issues similar to here and here.

you'll have to be much more specific and show the exact errors you get and a snippet of your code

sample code snippet:

# example of iters1 is [(100,100),(0,0)]
def get_filename(filename,iters1):
    gdalfile = gdal.Open('/vsicurl/'+filename)
    ndval = gdalfile.GetRasterBand(1).GetNoDataValue()
    val = np.array([gdalfile.ReadAsArray( xoff=int(x[0]), yoff=int(x[1]), xsize=1, ysize =1)[0,0] for x in iters1])
    return np.where(val==ndval,np.nan,val)

#example element of filenames is like https://storageexample.blob.core.windows.net/container/folder/file.tif
# length of filenames is 112

with ThreadPoolExecutor(max_workers=100) as executor:
            futures = [
                executor.submit(get_filename, x,iters1)
                for x in filenames
            ]
df = pd.DataFrame([x.result() for x in futures])

Errors im observing from both VM and AKS:
At each iteration, Im getting errors from different files.

ERROR 1: TIFFFetchDirectory:/vsicurl/https://storageexample.blob.core.windows.net/container/folder/CRFVOL_M_sl6_250m_ll.tif: Can not read TIFF directory count
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 2117828952
ERROR 1: TIFFFetchStripThing:IO error during reading of 'StripByteCounts'
ERROR 4: `/vsicurl/https://storageexample.blob.core.windows.net/container/folder/BLDFIE_M_sl2_250m_ll.tif' not recognized as a supported file format.
ERROR 1: TIFFFetchStripThing:IO error during reading of 'StripOffsets'
ERROR 1: TIFFFetchStripThing:IO error during reading of 'StripByteCounts'

Getting status code 206 in Azure blob storage logs.

Regards,
Dinesh

[rouault]

Which GDAL version ?

[dimattap]

Which GDAL version ?

Not able to access the VM currently. As far as I remember, it was 2.3.0.

[rouault]

As far as I remember, it was 2.3.0.

oh, that's pretty old and no longer supported. You should consider upgrading to get likely fixes for the concurrency issues you hit

[dimattap]

As far as I remember, it was 2.3.0.

oh, that's pretty old and no longer supported. You should consider upgrading to get likely fixes for the concurrency issues you hit

I was just able to access the VM.
GDAL==3.0.2
Apologies for the confusion in previous reply