[ETL-683] Increase memory of raw lambda

[philerooski]

Changes:

• Increased memory from 1024 to 1769 (1 vCPU).
• Put compressed_data in a context manager so that we don't have to manually close
• Removed return statement at end of main in case we ever want to use a batch size larger than 1.
• Add raw lambda tests to CI/CD

Details about benchmarking exploration are here.

[thomasyu888]

Great benchmarking! I'll leave this to @rxu17 for a review.

👀 the nesting of context managers (unfortunate, but makes sense)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[rxu17]

LGTM! Just a few minor comments

[rxu17]

Nit: Depending on if this gets merged first or mine (may be a slight merge conflict), but i think you can link the tests like:

- name: Test scripts with pytest (lambda, etc.)
        run: |
          pipenv run python -m pytest \
            tests/test_s3_event_config_lambda.py \
            tests/test_s3_to_glue_lambda.py \
            tests/test_lambda_dispatch.py \
            tests/test_consume_logs.py \
            tests/test_lambda_raw.py -v

[philerooski]

Eh, it's better for the commands to be discrete and explicit, imo. I'm not sure if the behavior is the same running the tests one at a time (separate pipenv run...) vs running all the tests at one time (a single pipenv run...). But if there are advantages (other than brevity) I'm open to running all the tests at one time.

[rxu17]

Nit: Something i was thinking of but do we want to make this a variable instead? So the develop version of the lambda can declare a smaller memory size than the prod version since it won't use as much? Or were you running into this for both?

[philerooski]

I don't think there's much benefit. This is not an expensive function to run, so no harm in keeping the configurations the same.