Permission denied when generating parquet data for lab runs

[steinnymir]

Beamtime directories generated for lab data seem to have restricted permissions to few users.
This is absoultely correct, but limits the work that other users (me and Zain for example) can do.

Any suggestions how to solve this?

• Add permissions for all interested users
• Add a dummy test beamtime where we can dump test files?

[steinnymir]

@zainsohail04 suggested using local directories for test purpouses.
The parquet dir was hardcoded but I added an optional argument to read_data to overwrite this.

now the parquet folder can be arbitrarily selected, and data readout works as intended

[zain-sohail]

Thanks. I think I will restructure the code because this functionality is also in express dataframe creator class and can be generalized for both.

[kutnyakhov]

We are currently trying to write parquet in the "raw" folder, but they should go into "processed" folder
/asap3/fs-flash-o/gpfs/hextof/2022/data/11013726/processed
For this folder, everyone should have write permission.

[kutnyakhov]

One of our last gray days at FLASH, where we were using fs-flash-o/hextof for data storage all parquet files were written into folder
/asap3/fs-flash-o/gpfs/hextof/2021/data/11013350/processed/parquet/per_file

So it was not even trying to write into "raw" folder - could it be that this was changed and now code trying to write into current ("raw") folder?

[zain-sohail]

When using readData, there is a optional argument for parquet_path. The default is it adds a new parquet dir to the the file path.