Key Error : APGEN1

[Ryurin]

Hello,

I used Clinica to convert ADNI to BIDS and I have a pandas error : "KeyError : APGEN1"

This error concern adni_to_bids.convert_clinical_data with participants_df = bids.create_participants_df() and more precisely value_to_read = file_to_read[participant_field_db[i]]

It's when participants.tsv are created and no tsv are generated in my case. I just download all ADNI data between ADNI 1 to ADNI 4 (on July 2024) et just run clinica without options (for all datas).

Do you have an idea why I have this error ? I suppose that one key isn't here in clinical data for ADNI but if we have a larger problem, I would like to submit it.

Thank you for your help !

[AliceJoubert]

Hi, thanks for reporting this issue !

Could you tell me first what OS you use, as well as the version of python and of clinica you are currently running ? Clinica does not support ADNI4 yet but I want to make sure there is no other issue first.

You suppose well I think but we still need to fix it ^^ !

Best,

[Ryurin]

Of course ! I use CentOS Linux release 7.6.1810 with python 3.9 and clinica version 0.7.5

ADNI love to change its information between 1, 2 and 3 so I imagine that 4 have some changes too ! If Clinica doesn't support ADNI4, maybe neuroimaging can't be converted ? Or clinica can use ADNI's CSV like ADNI1,2,3 ?

[AliceJoubert]

Thanks for precisions ! First I would use a new python environment with the latest release of clinica since we are now using python 3.10 and clinica 0.8.4 (the instructions for installation are here).

If you still have issues with converting data then it is probably due to changes in csv as you said. It will mainly break when trying to write metadata files (participants, scans, sessions tsv files) but the BIDS architecture will be there. Depending on what you want to do it could suffice while we fix this which is not planned yet.

Let me know how it goes :-)

[GabrieleLozupone]

Solution for ADNI to BIDS Conversion Issue

Hi, I encountered the same error (KeyError: APGEN1) when trying to convert ADNI data to BIDS using Clinica, specifically while generating the participants.tsv file. After some investigation, it seems that the error arises because of changes in the GENOTYPE column in the clinical data file (APOERES_16Sep2024.csv).

Here’s the solution that worked for me:

1. The GENOTYPE column in the current version of the data is structured differently from previous versions (it now contains values that need to be split into two columns: APGEN1 and APGEN2). To fix this, I modified the CSV file as follows:

import pandas as pd

# Path to the APOE results file
path = "your-path/APOERES_16Sep2024.csv"

# Read the CSV file
df = pd.read_csv(path)

# Get the position (index) of the 'GENOTYPE' column
genotype_index = df.columns.get_loc('GENOTYPE')

# Split the 'GENOTYPE' column into 'APGEN1' and 'APGEN2'
apgen = df['GENOTYPE'].str.split('/', expand=True)
apgen.columns = ['APGEN1', 'APGEN2']

# Drop the original 'GENOTYPE' column
df = df.drop(columns=['GENOTYPE'])

# Insert 'APGEN1' and 'APGEN2' at the position of the old 'GENOTYPE' column
for i, col in enumerate(apgen.columns):
    df.insert(genotype_index + i, col, apgen[col])

# Save the modified DataFrame and replace the old file
df.to_csv(path, index=False)

2. Additionally, I had to rename the DXSUM_{date}.csv file to DXSUM_PDXCONV_{date}.csv to resolve another issue with missing files during conversion.

With these changes, I was able to successfully convert the without errors.

I hope this helps others facing the same issue.

[AliceJoubert]

Thank you very much @GabrieleLozupone for your contribution ! I would advise to follow your suggestion till we change the source code to fix this issue (soon, hopefully 🤞).

[Ryurin]

Thank you for your help ! It worked perfectly for .tsv !

I have one more error after that :

INFO:Creating scans files...
Traceback (most recent call last):
File "/homes_unix/nlambert/.local.hpc/bin/clinica", line 8, in
sys.exit(main())
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/clinica/cmdline.py", line 92, in main
cli()
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids_cli.py", line 74, in cli
adni_to_bids.convert_clinical_data(
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids.py", line 154, in convert_clinical_data
adni_utils.create_adni_scans_files(conversion_path, bids_subjs_paths)
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 982, in create_adni_scans_files
converted_mod = find_conversion_mod(file_name)
File "/homes_unix/nlambert/.local.hpc/lib/python3.9/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 1023, in find_conversion_mod
raise ValueError(f"Conversion modality could not be found for file {file_name}")
ValueError: Conversion modality could not be found for file sub-ADNI031S4194_ses-M000_FLAIR_real.nii

This file seems to not be very important but it's a problem. I checked and I have the .NII image with the good name and format so I don't understand why I have this error, maybe I can rerun ?

[AliceJoubert]

Glad to hear the previous issue is resolved !

When you mean "the good name" do you mean "sub-ADNI031S4194_ses-M000_FLAIR_real.nii" ?

[Ryurin]

Yes !

[AliceJoubert]

Are you still using your environment with python 3.9, clinica 0.7.5 ? ADNI-to-BIDS also require dcm2nix, do you know which version are you using ?

[Ryurin]

Yes, I ask to update because I can't make it myself but it's not done. I have dcm2niix==1.0.20220715. I have scan files for few subjects but one failed and stop for the rest of subjects so I think it's maybe not the problem ?

I try to rerun because internal connections between data and servers (not in the same place) can fail.

[AliceJoubert]

Sometimes dcm2niix produces images with the suffix "_real". They are not BIDS compliant so clinica removes them after they have been written by dcm2niix. I tried the same set-up as you (same version of python, clinica, dcm2niix) but did not see anything wrong in this behavior so that is not the issue.

Running again could indeed work, tell me if that changed anything. You can also try downloading the image data again. If this persists : do you know if this issue arises for only one subject or more ?

Since the converter is going to remove these images anyway, you can remove the subject/session from your original data or use a subject list (see clinica documentation) without it.

[Ryurin]

Rerun change nothing. I think I will have the same problem with others subjects (because I checked repertories and I have others _real suffix, for FLAIR). What is the difference between images with and without "_real" suffix ? Maybe I can rename it ?

[shahzadali21]

I followed your instruction, the issue of APGEN1 is resolved. However, now I am having the error as below:

2024-10-18 19:55:12,883:DEBUG:PET_TAU conversion done.
2024-10-18 19:55:12,903:INFO:Creating modality agnostic files...
2024-10-18 19:55:12,924:INFO:Creating participants.tsv...
Traceback (most recent call last):
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/bin/clinica", line 8, in
sys.exit(main())
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/cmdline.py", line 113, in main
cli()
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, kwargs)
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids_cli.py", line 64, in cli
convert(
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids.py", line 47, in convert
adni_to_bids.convert_clinical_data(
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids.py", line 134, in convert_clinical_data
participants_df = create_participants_df(
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/iotools/bids_utils.py", line 400, in create_participants_df
file_to_read = load_clinical_csv(
File "/Users/shahzadali/miniconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 1115, in load_clinical_csv
raise IOError(
**OSError: Expecting to find exactly one file in folder /Volumes/CrucialX6/ADNI_Clinica/Clinical_Data matching pattern APOERES(\d{1,2}[A-Za-z]{3}\d{4})?.csv. 2 files were found instead :
[- P- o- s- i- x- P- a- t- h- (- '- /- V- o- l- u- m- e- s- /- C- r- u- c- i- a- l- X- 6- /- A- D- N- I- _- C- l- i- n- i- c- a- /- C- l- i- n- i- c- a- l- _- D- a- t- a- /- A- P- O- E- R- E- S- _- 1- 6- A- u- g- 2- 0- 2- 4- .- c- s- v- '- )- ,- - P- o- s- i- x- P- a- t- h- (- '- /- V- o- l- u- m- e- s- /- C- r- u- c- i- a- l- X- 6- /- A- D- N- I- _- C- l- i- n- i- c- a- /- C- l- i- n- i- c- a- l- _- D- a- t- a- /- .- _- A- P- O- E- R- E- S- _- 1- 6- A- u- g- 2- 0- 2- 4- .- c- s- v- '- )- ]

I have double checked the directory but there is only one file in the CLinicalData Directory. However, When I searched on my machine with the name "APOERES". There are two files retreived but with the name "participant.csv" as indicated in the screenshot (I guess one for ADNI, and one for ABIDE and OASIS).