Disk cache the manifests, fixes #392

[peterbe]

The gist of this is that when we download the .csv.gz files we break it up.

1. First clear out any old downloaded .csv.gz files that are older than 48 hours.
2. For each .csv.gz file in the manifest.json, instead of downloading it and yielding each CSV row, we download it to $CSV_DOWNLOAD_DIRECTORY (*)
3. After the file has been downloaded, instead of reading it from a network URL, we read it from the local file.

(*) In docker-compose this becomes ./csv-download-directory but outside Docker it becomes $TMPDIR/csv-download-directory. This way it's kept between stopping and starting Docker.

[peterbe]

Oops. I have to delete this.

[leplatrem]

nit: use makedirs(..., exists_ok=True) ?

[leplatrem]

move to DOWNLOAD_MAX_AGE constant ?

[leplatrem]

nit: add explicit st_size > 0

[peterbe]

Out-of-curiousity; why?

Isn't it explicit already that the test is for the st_size to be anything greater than 0?

[leplatrem]

Matter of taste maybe :)

[leplatrem]

Here maybe you should try/except and delete the partially downloaded file when an error happens, or maybe you don't want to check the md5sum when resuming

[peterbe]

I was totally not sure how to do this in the aiosync world. Especially not how to test it. But I did write this, locally:

import time
import os
import asyncio
import aiohttp
import aiofiles

CHUNK_SIZE = 1024 * 256  # 256 KB

async def download_csv(loop, url):
    file_path = '/tmp/' + os.path.basename(url).split('?')[0]
    os.path.isfile(file_path) and os.remove(file_path)

    async with aiohttp.ClientSession(loop=loop) as session:
        try:
            async with aiofiles.open(file_path, 'wb') as destination:
                async with session.get(url) as resp:
                    print(resp.status)
                    while True:
                        print(resp.status, end='', flush=True)
                        time.sleep(0.05)
                        chunk = await resp.content.read(CHUNK_SIZE)
                        if not chunk:
                            break
                        await destination.write(chunk)
        except aiohttp.client_exceptions.ClientPayloadError:
            print('\n')
            mb = os.stat(file_path).st_size / 1024 / 1024
            print(f'WROTE {mb:.1f}MB')
            os.remove(file_path)
            raise
    print('\nall Done!')
def run():

    loop = asyncio.get_event_loop()
    url ='http://10.0.0.80:8080/file.csv'
    loop.run_until_complete(download_csv(loop, url))

if __name__ == '__main__':
    run()

That's using plain aiohttp.ClientSession(loop=loop) though. Not await s3_client.get_object(...)

When I run that in one terminal, then quickly switch to the other terminal where I run the HTTP server on :8080 and kill it, the except block successfully cleans up the half-downloaded file.

r?

[leplatrem]

r+ :)