What to do with Processed Manifest Files?

[DanielWright]

Idempotence is all well and good, but the processing time will scale linearly unless we move or remove or otherwise mark the manifest files as processed. Ideas?

[peterberkenbosch]

I think removing them after successful process is the best way. When not successful, i.e. on error it will stay put to investigate.

[DanielWright]

I suppose S3 versioning protects us from malfunctions; still, I would feel a little more comfortable, at least at first, if we only moved/marked them for manual auditing.

[peterberkenbosch]

we could add a processed folder config and move the s3 object there after the run