Release v2.4.0

[ChenYi015]

Purpose of this PR

Release v2.4.0.

Features

• fix(chart): add revisionHistoryLimit option (#2625 by @t3mi)
• Suspend/Resume feature on SparkApplication (#2387 by @everpeace)
• Set ControllerReference on driver pod and non-Controller OwnerReference on executor pod at submission time (#2670 by @everpeace)
• added SparkApp name validator to accept valid DNS-1035 format (#2711 by @aryankumar04)
• No propagate Kueue labels to driver and executor pods (#2714 by @everpeace)
• feat: add support for customizing connect service (#2709 by @ChenYi015)

Bug Fixes

• Grant get/patch permissions for SparkConnect CRD to hook ClusterRole (#2605 by @ChenYi015)
• Grant create events permissions to Controller (#2616 by @Rockawear)
• fix(connect): propagate executor memory setting to spark options (#2656 by @mrjoe7)
• [fix] add miss cli params. Add leader election params in chart (#2657 by @aagumin)
• fix: webhook encoder configuration (#2664 by @pvbouwel)
• bugfix: hook.upgradeCrd use cases (#2663 by @pvbouwel)
• Correct entrypoint.sh for Openshift (#2645 by @Reamer)
• Fix driver host configuration to handle IPv6 addresses (#2703 by @tiagotxm)
• Add permissions for sparkconnects/finalizers (#2727 by @ChenYi015)
• Remove redundant name and namespace context in logs (#2723 by @ChenYi015)
• Logging info when resources associated with SparkApplication still exist (#2725 by @ChenYi015)

Unit Tests

• Add kube-scheduler podgroup unit tests (#2689 by @shadowinlife)
• Generate HTML coverage report after running unit tests (#2691 by @shadowinlife)
• Avoid 409 (Conflict) error at e2e tests with retry. (#2695 by @shadowinlife)
• Add unit test for sparkapplication_validator (#2692 by @shadowinlife)
• Add unit test for scheduledsparkapplication_validator (#2694 by @shadowinlife)
• Refactor the unit tests for web_ui.go (#2696 by @shadowinlife)

Refactor

• unify SparkApplication defaulting logic (#2671 by @zhzhuang-zju)
• Replace strconv.Atoi with strconv.ParseInt (#2699 by @ChenYi015)
• refactor: use ptr.To to replace util.XxxPtr (#2693 by @ChenYi015)

Dependencies

• Bump actions/checkout from 4 to 5 (#2623 by @dependabot[bot])
• Bump actions/download-artifact from 4 to 5 (#2624 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.18.4 to 3.18.5 (#2627 by @dependabot[bot])
• Bump github.com/go-viper/mapstructure/v2 from 2.3.0 to 2.4.0 (#2635 by @dependabot[bot])
• Bump actions/setup-go from 5 to 6 (#2651 by @dependabot[bot])
• Bump actions/stale from 9 to 10 (#2649 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.32.0 to 0.33.1 (#2650 by @dependabot[bot])
• Bump github.com/golang/glog from 1.2.4 to 1.2.5 (#2620 by @dependabot[bot])
• Bump github.com/onsi/ginkgo/v2 from 2.23.3 to 2.26.0 (#2660 by @dependabot[bot])
• Bump sigs.k8s.io/scheduler-plugins from 0.31.8 to 0.32.7 (#2675 by @dependabot[bot])
• Bump github/codeql-action from 3 to 4 (#2672 by @dependabot[bot])
• Bump github.com/stretchr/testify from 1.10.0 to 1.11.1 (#2673 by @dependabot[bot])
• Bump golang.org/x/time from 0.9.0 to 0.14.0 (#2686 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.22.0 to 1.23.2 (#2682 by @dependabot[bot])
• Bump github.com/spf13/viper from 1.20.1 to 1.21.0 (#2683 by @dependabot[bot])
• Bump github/codeql-action from 3.29.2 to 4.31.0 (#2684 by @dependabot[bot])
• Bump github.com/containerd/containerd from 1.7.27 to 1.7.29 (#2712 by @dependabot[bot])
• Bump sigs.k8s.io/yaml from 1.5.0 to 1.6.0 (#2704 by @dependabot[bot])
• Bump github.com/spf13/cobra from 1.9.1 to 1.10.1 (#2705 by @dependabot[bot])
• Bump github/codeql-action from 4.31.0 to 4.31.2 (#2707 by @dependabot[bot])
• Bump actions/upload-artifact from 4 to 5 (#2681 by @dependabot[bot])
• Bump actions/download-artifact from 5 to 6 (#2680 by @dependabot[bot])
• Bump actions/checkout from 4.2.2 to 5.0.0 (#2685 by @dependabot[bot])
• Bump ossf/scorecard-action from 2.4.2 to 2.4.3 (#2687 by @dependabot[bot])
• Bump github.com/onsi/ginkgo/v2 from 2.26.0 to 2.27.2 (#2719 by @dependabot[bot])
• Bump golang.org/x/mod from 0.27.0 to 0.29.0 (#2720 by @dependabot[bot])
• Bump helm/chart-testing-action from 2.7.0 to 2.8.0 (#2722 by @dependabot[bot])

Misc

• Add changelog for v2.3.0 (#2614 by @ChenYi015)
• Update README (#2634 by @ChenYi015)
• Add Spark Operator Roadmap (#2622 by @ChenYi015)
• CI: Add Scorecard workflow (#2654 by @AdamKorcz)
• Update GitHub action workflow permissions (#2678 by @ChenYi015)

Full Changelog

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from chenyi015. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment