Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On Study View handle larger number of resource files #5034

Merged
merged 10 commits into from
Nov 1, 2024
70 changes: 61 additions & 9 deletions src/pages/studyView/resources/FilesAndLinks.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,19 @@ class FilesLinksTableComponent extends LazyMobXTable<{

const RECORD_LIMIT = 500;

function getResourceDataOfEntireStudy(studyIds: string[]) {
// Only handle the first studyId for now. Can be expanded to make a call per
// studyId.
const studyId = studyIds[0];
const allResources = internalClient.getAllStudyResourceDataInStudyPatientSampleUsingGET(
{
studyId: studyId,
projection: 'DETAILED',
}
);
return allResources;
}

function getResourceDataOfPatients(studyClinicalData: {
[uniqueSampleKey: string]: ClinicalData[];
}) {
Expand Down Expand Up @@ -97,7 +110,7 @@ function buildItemsAndResources(resourceData: {

async function fetchFilesLinksData(
filters: StudyViewFilter,
sampleIdResourceData: { [sampleId: string]: ResourceData[] },
selectedSamples: Array<any>,
searchTerm: string | undefined,
sortAttributeId: string | undefined,
sortDirection: 'asc' | 'desc' | undefined,
Expand All @@ -112,13 +125,52 @@ async function fetchFilesLinksData(
0
);

const resourcesForPatients = await getResourceDataOfPatients(
studyClinicalDataResponse.data
const selectedStudyIds = [
...new Set(selectedSamples.map(item => item.studyId)),
];
const selectedSampleIds = selectedSamples.map(item => item.sampleId);
const selectedPatientIds = [
...new Set(selectedSamples.map(item => item.patientId)),
];

const resourcesForEntireStudy = await getResourceDataOfEntireStudy(
hweej marked this conversation as resolved.
Show resolved Hide resolved
selectedStudyIds
);
const resourcesForPatientsAndSamples: { [key: string]: ResourceData[] } = {
...sampleIdResourceData,
...resourcesForPatients,
};

const resourcesForPatientsAndSamples = resourcesForEntireStudy
hweej marked this conversation as resolved.
Show resolved Hide resolved
// Filter the resources to consist of only studyView selected samples
// Also keep patient level resources (e.g. Those don't have a sampleId)
.filter(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one more thing here: this is operation is Rsamples+Rpatients. could easily get to billion. easily optimized by putting selectedSamples in a map so you can use IN operator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have replaced this with Maps instead of arrays to remove the linear lookup. I don't think you can use the IN operator with Maps so we're opting for the .has('key') expression instead.

(resource: ResourceData) =>
selectedSampleIds.includes(resource.sampleId) ||
selectedPatientIds.includes(resource.patientId)
)
.reduce(
(
idMap: { [key: string]: ResourceData[] },
resource: ResourceData
) => {
const { resourceType } = resource.resourceDefinition;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole operation would be more clearly handled by lodash's keyBy function. we like to use these kind of named helpers because they make the operation more explicit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I couldn't make lodash keyBy function to work the way I wanted it to for this use case. In python this would be a case for collections.defaultDict. But I couldn't make this work where I could create new arrays and append to them.

LMK if you want me to work on this part some more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hweej sorry, i mean _.groupBy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have replaced these lines with a single lodash groupBy('patientId'). Thanks @alisman !

const { patientId, sampleId } = resource;

if (resourceType == 'PATIENT') {
if (idMap.hasOwnProperty(patientId)) {
idMap[patientId].push(resource);
} else {
idMap[patientId] = [resource];
}
} else if (resourceType == 'SAMPLE') {
if (idMap.hasOwnProperty(sampleId)) {
idMap[sampleId].push(resource);
} else {
idMap[sampleId] = [resource];
}
}

return idMap;
},
{}
);

// we create objects with the necessary properties for each resource
// calculate the total number of resources per patient.
Expand Down Expand Up @@ -197,7 +249,6 @@ export class FilesAndLinks extends React.Component<IFilesLinksTable, {}> {
await: () => [
this.props.store.selectedSamples,
this.props.store.resourceDefinitions,
this.props.store.sampleResourceData,
],
onError: () => {},
invoke: async () => {
Expand All @@ -207,12 +258,13 @@ export class FilesAndLinks extends React.Component<IFilesLinksTable, {}> {

const resources = await fetchFilesLinksData(
this.props.store.filters,
this.props.store.sampleResourceData.result!,
this.props.store.selectedSamples.result,
this.searchTerm,
'patientId',
'asc',
RECORD_LIMIT
);

return Promise.resolve(resources);
},
});
Expand Down
Loading