Add archive_files to scrape args

[alexobaseki]

DATA-4855

• Set is_archive_files argument

[jessemortenson]

LGTM

[jessemortenson]

You know my first read was just looking at the code as is, and taking a moment to think I'm realizing you took a different way to implement this than I was imagining. I was imagining that the configuration flag would be in the os-realtime lambda function config. So that if you wanted files to start archiving, you'd change the Lambda config in AWS admin console.

This way should work fine, and it has the advantage of allowing a dev to run an ad hoc scrape with this flag set to test things (without changing behavior of other data flowing through). One disadvantage I can think of is that we'd have to redeploy DAGs (or I guess in the short run update about a dozen task definitions in the OS task defs repo?) if we wanted to switch the behavior for all realtime scraper runs.

Were there other advantages/disadvantages you were thinking of with this approach?

[alexobaseki]

You know my first read was just looking at the code as is, and taking a moment to think I'm realizing you took a different way to implement this than I was imagining. I was imagining that the configuration flag would be in the os-realtime lambda function config. So that if you wanted files to start archiving, you'd change the Lambda config in AWS admin console.

This way should work fine, and it has the advantage of allowing a dev to run an ad hoc scrape with this flag set to test things (without changing behavior of other data flowing through). One disadvantage I can think of is that we'd have to redeploy DAGs (or I guess in the short run update about a dozen task definitions in the OS task defs repo?) if we wanted to switch the behavior for all realtime scraper runs.

Were there other advantages/disadvantages you were thinking of with this approach?

Thanks for additional insight. I approached the problem this based on the context(or lack of it). I have on our the system works. I did think of it having that advantage for easy local testing but I think I like your approach better. I am assuming that the lambda function config will be reflected in context args of process_import_function(event, context): such that
if context.file_archiving_enabled: #ARCHIVE_FILE?

[elseagle]

LGTM, thank you! A couple of follow-up actions are needed here (just in case you might not be aware):

1. We need to merge and release this first (all the way to pypi)
2. Then update openstates-scraper with the latest release of openstates-core via poetry
3. Optionally: update openstates-realtime with the latest release of openstates-core via poetry, but this is not really important this time because the change in openstates-core is not directly used in the openstates-realtime project i.e we are not calling openstates-core library or any of the changes used directly just communication via SQS and Lambda and that's handled in the openstates realtime PR

[alexobaseki]

LGTM, thank you! A couple of follow-up actions are needed here (just in case you might not be aware):

1. We need to merge and release this first (all the way to pypi)
2. Then update openstates-scraper with the latest release of openstates-core via poetry
3. Optionally: update openstates-realtime with the latest release of openstates-core via poetry, but this is not really important this time because the change in openstates-core is not directly used in the openstates-realtime project i.e we are not calling openstates-core library or any of the changes used directly just communication via SQS and Lambda and that's handled in the openstates realtime PR

Thank you Sogo, I probably need to a lecture on how all of this work on a call.