Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove icu_testdata #4960

Merged
merged 2 commits into from
May 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 0 additions & 64 deletions .github/workflows/artifacts-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -572,70 +572,6 @@ jobs:
git checkout empty
gsutil -m cp -r benchmarks/binsize/* gs://${{ env.GCP_MAIN_BUCKET_ID }}/gha/benchmarks/binsize

# Data size benchmark: track size of provider/datagen/tests/data/testdata.postcard (total data size).
bench-datasize:
# This is too expensive to run on every push, so only run it on main.
# When running on a PR, comment this out and set the BASELINE variable below to the baseline commit.
if: github.ref == 'refs/heads/main' && github.repository == 'unicode-org/icu4x'
concurrency:
# Allow one run at a time to include the previous run's results
group: bench-datasize
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

# GCP Boilerplate for jobs in main repository
- id: gcp-auth
name: "Authenticate to Google Cloud with Workload Identity Provider"
uses: google-github-actions/auth@v1
with:
workload_identity_provider: "projects/66042061814/locations/global/workloadIdentityPools/icu4x-gha-pool1/providers/icu4x-gha-provider1"
service_account: "icu4x-main@dev-infra-273822.iam.gserviceaccount.com"
- name: "Set up Google Cloud SDK"
uses: google-github-actions/setup-gcloud@v1

- name: Install rustfmt
run: |
rustup component add rustfmt

- name: Setup output data directory
run: |
mkdir -p benchmarks/datasize

- name: Generate testdata
run: |
cargo run --bin make-testdata-legacy --manifest-path tools/testdata-scripts/Cargo.toml

- name: Measure size of selected data package provider/testdata/data/testdata.postcard
run: |
cargo run --package icu_benchmark_binsize -- provider/testdata/data/testdata.postcard file | tee benchmarks/datasize/output.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought: It might be nice to run this benchmark with a postcard file generated from plain datagen, but this benchmark is not very useful because no one should be including the whole postcard file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have per-locale-per-key stats somehow, to catch things like the dictionary snafu


- name: Download previous benchmark data
run: |
mkdir -p benchmarks
gsutil -m cp -rn gs://${{ env.GCP_MAIN_BUCKET_ID }}/gha/benchmarks/datasize/* benchmarks/datasize

- name: Store benchmark result & create dashboard
# Use gregtatum special feature to process ndjson-formatted benchmark data
uses: gregtatum/github-action-benchmark@d3f06f738e9612988d575db23fae5ca0008d3d12
with:
tool: 'ndjson'
output-file-path: benchmarks/datasize/output.txt
benchmark-data-dir-path: ./benchmarks/datasize
# Tentative setting, optimized value to be determined
alert-threshold: '100%'
fail-on-alert: false
# comment-on-alert: true
github-token: ${{ secrets.GITHUB_TOKEN }}
gh-pages-branch: empty
# alert-comment-cc-users: '@gnrunge,@sffc,@zbraniecki,@echeran'

- name: Upload new benchmark data
if: success() || failure()
run: |
git checkout empty
gsutil -m cp -r benchmarks/datasize/* gs://${{ env.GCP_MAIN_BUCKET_ID }}/gha/benchmarks/datasize

gh-pages:
name: "Deploy to GitHub Pages"
needs: [rust-docs, cpp-docs, ts-docs, dart-docs, wasm-demo, bench-perf, bench-memory, bench-datasize] # bench-binsize
Expand Down
31 changes: 1 addition & 30 deletions .github/workflows/build-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -196,35 +196,6 @@ jobs:
- name: Run `cargo make ci-job-testdata`
run: cargo make ci-job-testdata


# ci-job-testdata-legacy
testdata-legacy:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

# Cargo-make boilerplate
- name: Install cargo-make
uses: taiki-e/install-action@cargo-make

# Toolchain boilerplate
- name: Potentially override rust version with nightly
run: cargo make set-ci-toolchain
- name: Show the selected Rust toolchain
run: rustup show

# Job-specific dependencies
- name: Install cargo-binstall
uses: taiki-e/install-action@cargo-binstall
- name: Install cargo-all-features
run: cargo-binstall -y cargo-all-features

# Actual job
- name: Run `cargo make ci-job-testdata-legacy`
run: cargo make ci-job-testdata-legacy


# ci-job-full-datagen
full-datagen:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -561,7 +532,7 @@ jobs:

# Notify on slack
notify-slack:
needs: [msrv, test, testdata, testdata-legacy, test-docs, full-datagen, test-c, test-js, test-dart, nostd, diplomat, gn, fmt, tidy, clippy, doc]
needs: [msrv, test, testdata, test-docs, full-datagen, test-c, test-js, test-dart, nostd, diplomat, gn, fmt, tidy, clippy, doc]
if: ${{ always() && contains(needs.*.result, 'failure') && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref_name == 'main')) }}
runs-on: ubuntu-latest
steps:
Expand Down
2 changes: 0 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 0 additions & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,6 @@ members = [
# Note: Workspaces in subdirectories, such as tutorials/crates, are
# implicitly excluded from the main workspace.
exclude = [
# Testdata will still be published in the 1.x stream, but is deprecated
# and we don't use it anymore. As we don't want to keep the actual data
# in the repo it doesn't build without running `cargo make testdata-legacy`
# first.
"provider/testdata",
# Tutorials are tested outside the workspace to simulate external users
"tutorials",
]
Expand Down
8 changes: 0 additions & 8 deletions Makefile.toml
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,6 @@ dependencies = [
"testdata-check",
]

[tasks.ci-job-testdata-legacy]
description = "Run all tests for the CI 'testdata' job"
category = "CI"
dependencies = [
"testdata-legacy",
"testdata-legacy-test",
]

[tasks.ci-job-test-docs]
description = "Run all tests for the CI 'test-docs' job"
category = "CI"
Expand Down
1 change: 0 additions & 1 deletion provider/testdata/.gitignore

This file was deleted.

Loading
Loading