Adding ability to redefine cache timeout

[VitalyFedyunin]

Stack from ghstack (oldest at bottom):

• -> Adding ability to redefine cache timeout #571

Differential Revision: D37723437

[VitalyFedyunin]

@parmeet can you please review this code (@ejguan and @NivekT are on PTO)

[ejguan]

LGTM. Thank you.

I just realize one general question for Adapter. Since adapters would modify the DataPipe graph in-place, do we want to revert the modification for the DataPipe graph after one?epoch?

[VitalyFedyunin]

IMO DLv2 should apply adapter modifications once on the local copy of the graph it gets from the args.

[VitalyFedyunin]

@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[VitalyFedyunin]

@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[parmeet]

LGTM! Thanks @VitalyFedyunin for adding the ability to manually set time for caching files. Just to be sure, the users can add this adapter on top of datapipe we return for datasets, in other words, we do not have to add this adapter internally in the dataset implementation right?

[VitalyFedyunin]

LGTM! Thanks @VitalyFedyunin for adding the ability to manually set time for caching files. Just to be sure, the users can add this adapter on top of datapipe we return for datasets, in other words, we do not have to add this adapter internally in the dataset implementation right?

Correct