Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1714 migrate old data #1742

Merged
merged 10 commits into from
Jun 4, 2024
Merged

Conversation

Skydodle
Copy link
Member

@Skydodle Skydodle commented May 31, 2024

Fixes #1714

Note

I only migrated and enabled 2022 data in this PR so far, would like reviewers to play around within the 2022 calendar dates to see if there are any issues with this implementation. Once approved I will continue to add the rest of older years.

Changes

  • Created Python script cleanHfDataset with functions as following:
    • Download csv from Edwin's hf
    • Clean abnormal strings & reformat to parquet
    • Create according year's repo on 311's hf then upload the parquet to it
    • Clean up local directory.
  • Note: Script needs to be run manually and locally with a year input in the terminal, ie. python3 cleanHfDataset.py 2022
  • Added dataset registration for 2022
  • Enabled 2022 for the day picker calendar
  • Updated tooltip text

Pre-Merge Checklist

  • Up to date with main branch
  • Branch name follows guidelines
  • All PR Status checks are successful
  • Peer reviewed and approved

Copy link
Member

@ryanfchase ryanfchase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, approved

@Skydodle Skydodle merged commit b5d1380 into hackforla:main Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate external Huggingface data to 311 Data Huggingface repo
2 participants