idr0036-gustafsdottir-cellpainting S-BIAD855 #640
Comments
|
Started conversion of the full dataset with omero-cli-zarr on pilot-zarr1-dev. |
|
Looks good: Each plate is about 10 GB -> 200 GB in total: |
will-moore
moved this from test convert
to create new Fileset to replace original Fileset
in NGFF conversion
will-moore
moved this from create new Fileset to replace original Fileset
to upload data to s3
in NGFF conversion
|
@sbesson OK for me to create a bucket at https://uk1s3.embassy.ebi.ac.uk/idr0036/ and upload 200 GB there? |
|
Created bucket and set policy... https://github.com/IDR/deployment/blob/master/docs/object-store.md#policy Started upload... |
will-moore
moved this from upload data to s3
to create new Fileset to replace original Fileset
in NGFF conversion
|
Now available (and valid) at e.g. https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0036/zarr/4403.zarr&well=all |
|
As discussed earlier today, converted a sample plate from this study using All the TIFFs from the
|
|
Looking at @sbesson's
A1 image from Compared with from `bioformats2raw: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0036/zarr/20585.zarr/A/1/0/ And IDR:
|
|
Thanks, echoing what I mentioned post-conversion |
|
A few additional notes on the channel order:
From my side there are two options: 1- either decide the current channel order in IDR is the ground truth. This means regenerating the NGFF accordingly probably via some symlinking Deferring to you and @francesw on what the right approach, it's additional work either way so we should consider the option that brings the maximal value |
|
Since we have all plates already exported to NGFF with |
|
For testing definitely, but the same set of decisions will apply to #638 where we the plates have not been converted yet so I wanted to make sure we have this discussion prior to doing so |
|
I noticed that the data exported with Probably will need to be re-exported with that fix (once tested etc) - I think that's going to be easier than fixing the dimension separator in place. |
will-moore
moved this from create new Fileset to replace original Fileset
to convert all data to NGFF
in NGFF conversion
|
Discussed in meeting today: if |
|
Updated to the latest release of |
|
Running in a screen |
|
Found a bug in |
|
Started to remove older data on s3 bucket... |
|
Uploaded Plate:4403.zarr to uk1s3 to replace deleted plate. NB: Since we only have 20 Plates, we can simply rename before zipping... We might-as-well also add the |
|
Export with Then zipped.... |
|
Started upload... |
will-moore
moved this from convert all data to NGFF
to Zip and upload to BioStudies
in NGFF conversion
will-moore
moved this from Zip and upload to BioStudies
to BioStudies Submission
in NGFF conversion
|
Delete everything...
|
francesw
changed the title
francesw
moved this from BioStudies Submission
to Data on Embassy s3
in NGFF conversion
will-moore
moved this from Data on Embassy s3
to create new Filesets in idr-next
in NGFF conversion
|
All 20 Plates are at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD855.html UPDATE: after 14 hours running, only 8 Filesets have been completed... |
Since these Plates were created with E.g. For the first Fileset above, check for paths etc... with |
|
http://localhost:1080/webclient/?show=image-2002281 to check for images... |
|
Oh dear - trying to view that image today: Due to mounting of s3 bucket. Created issue at #671 |
|
Since we cancelled on last Fileset above, rerunning that one on idr0125-pilot... |
|
Let's start from scratch again on idr-testing... using idr0036.csv from above,,, |
|
Done |
|
Checking for re-ran that one on idr0125-pilot as wmoore user... |
|
Fixed: |
will-moore
moved this from check_pixels in progress
to pixels validated
in NGFF conversion
idr0036-gustafsdottir-cellpainting
Sample plate conversion failed with:
Already handled by: IDR/bioformats#29 .
@will-moore successfully exported it using omero-cli-zarr.
The text was updated successfully, but these errors were encountered: