Integration into PyCharm

[jschmieder14]

I use PyCharm as my Python IDE and am having trouble accessing the databases in the IDE. I was able to successfully access the MIMIC data in BigQuery and have been able to access my personal projects in PyCharm. Despite this, I am not able to access the MIMIC data through PyCharm.

[briangow]

@jschmieder14 , thank you for posting this to the public repository. Could you add additional detail around what you are trying to do? In particular, please provide examples around what you have attempted when trying access the MIMIC data through PyCharm.

[alistairewj]

I can't say for certain since I've never used those tools but one important note is that your active project ID for BigQuery should not be physionet-data. Users aren't given any permission on the physionet-data project itself. If you use this as your active project you should expect forbidden errors. Instead, users are exclusively given read permission to the BigQuery datasets, which are hosted on the physionet-data project. I would try and configure it so that (1) your personal project ID is used (so compute / queries are billed against your personal project), while (2) the dataset is visible to you for querying (perhaps by starring physionet-data.mimiciv_hosp or some other similar mechanism available in the IDE).

[jschmieder14]

I assume that only the hosp and icu datasets are able to be copied in the way mentioned in the readme files on the repository? I was hoping to be able to use all six (derived, ecg, ed, hosp, icu, note) databases that are available through Google's service. There isn't a way to just change the active project without copying the databases over and it appears that keeping the same structure as is on the service is not possible.

[alistairewj]

All datasets will be visible for read access, so if you want you can copy data from them. For the other modules, you just need to be added to the project access list (e.g. request access to MIMIC-IV-ED to get access to mimiciv_ed).

If PyCharm won't let you have a different active project than the one that you are sourcing data from, probably your best bet is to copy all the data to your own BigQuery dataset. You'll have to pay for ongoing storage of it but I think that's fairly cheap: https://cloud.google.com/bigquery/pricing#storage

Logically MIMIC-IV takes around 70 GB / 65 GiB so you'd look at a cost of $1.50 per month to store it (and this drops to $0.75 after three months of not modifying the data).

Copying can be done with gsutil: bq cp physionet-data:mimiciv_3_1_hosp.admissions PERSONAL_BQ_PROJECT:mimiciv_hosp.admissions, etc. bq ls physionet-data:mimiciv_v3_1_hosp will list out all the tables. https://cloud.google.com/bigquery/docs/reference/bq-cli-reference#bq_cp

[jschmieder14]

With read access I should be able to perform any queries I have on BigQuery though, correct? At least that's what it seems that I have been able to do. If so, I'm still not sure why I am unable to connect the physionet-data project into my IDE.

An aside from this question: is there a place to download the derived database? I have not seen one thus far.

[alistairewj]

Yes correct but PyCharm could require job usage on the physionet-data project which you don't have (or some other similar low-level GCP permission).

No there isn't a place to download the derived data, the most straightforward way is to bq cp it to a google storage account and then download the data from the storage account.