Flattener #235
Flattener #235
Conversation
|
@jpmckinney, this is what I have so far. Some additional things I needed to do this time:
|
views:
- Guard against path traversal by malicious input
- Set export_format query string parameter to full suffix to avoid extra view logic
- Abbreviate export_format to format in query string, and export_formats to formats in template
template:
- Sort years in consistent order (as before)
- Add condition for all full files
- Add {% empty %} if files not yet available
util:
- Ensure the template always receives expected keys
- Don't process unknown suffixes
- Fix "jsonl" vs "json" typo for export_format
- Fix "csv.gz" -> "csv.tar.gz" typo
worker:
- Do not compress Excel file (an Excel file is already a ZIP file)
- Take advantage of Export().directory returning a pathlib.Path (temp file had ".jsonl.jsonl" suffix)
- Do "var = condition" not "if condition: var = True; else: var = False"
- Merge flatten_and_package_file() into callback()
perf:
- Use tempfile module to clean up files (temp files are written to the SSD, which is faster)
- Use scandir() (faster) instead of listdir()
|
I made some edits and fixes. Have you tested it manually? After we deploy, to process existing data, we can run The other option is to create a Task (in the database) for every Job, and then change the Job status accordingly, but this would be a lot more effort. I think the first option will work – we just won't have any job management features. |
|
Also, we need to translate "Files not yet available." |
Yes, I've tested it so I can test it again tomorrow before merging. I tried using a Temp directory instead of creating and removing the directory and file and I had some issues, so I want to test this before merging. |
No description provided.