[SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates

[yaooqinn]

What changes were proposed in this pull request?

This pull request introduces functionalities that enable 'Document and Feature Preview on the master branch via Live GitHub Pages Updates'.

Why are the changes needed?

• Instead of limited 72-hour voting phases, it provides the developer community with more opportunities to preview and verify the documentation contents.
• Instead of waiting for the final announcement of an official spark feature release, users can now preview some of the ongoing documented features, increasing the willingness to upgrade, sensing breaking changes in advance, and reducing the burden during the final upgrades.

Does this PR introduce any user-facing change?

No

How was this patch tested?

An example has been established by this repo - https://github.com/yaooqinn/spark-gh-pages

• Broken logo links are fixed at [SPARK-49494][DOCS] Use spark logos from spark-website in docs #47966

Was this patch authored or co-authored using generative AI tooling?

no

[HyukjinKwon]

While it's great to have the documentation for the latest development branch, I wonder if it's better to have the actual full nightly build and/or frequent release instead via setting up the release automation process.

My concern is that the Google result might show this page first, and people might get confused of, e.g., the new features in development.

[HyukjinKwon]

Actually one of my colleagues is working on setting up the release automation, cc @bozhang2820 WDYT?

[yaooqinn]

Thank you for the feedback @HyukjinKwon.

I wonder if it's better to have the actual full nightly build

This action can be scheduled w/ cron or via push, I tended to submit this PR using the push-and-cancelation trigger for a more up-to-date doc view.

We can benefit from this for a post-review on docs-related PRs

frequent release

While I'm also a supporter of the 'Release Early, release often' principle, after being the RM for 3.5.2, I think this could cost much on the infrastructures, such as 'pypi', 'ASF svn', 'mvn central', 'docker registry', e.t.c.

My concern is that the Google result might show this page first

This site will be marked as 'in-progress' or something else to reduce the ambiguity. Also, some directives of reStructuredText or markdown can be used in the doc build phase to enhance the warnings within a page in follow-up PRs.

[dongjoon-hyun]

Thank you, @yaooqinn . I guess we need a vote for this because GitHub Pages is a new channel from Apache Spark perspective.

If we use the official nightly channel, we may avoid the vote. However, IIUC, you want to use GitHub Pages specifically here, right, @yaooqinn ?

• https://nightlies.apache.org/spark/

[yaooqinn]

Thank you @dongjoon-hyun, I will raise a discussion thread in dev soon

[dongjoon-hyun]

Thank you!

[HyukjinKwon]

+1 for going for a vote. I think that'd be easier to decide.

[yaooqinn]

FYI, I have sent a mail to the dev https://lists.apache.org/thread/xojcdlw77pht9bs4mt4087ynq6k9sbqq

Stale

[dongjoon-hyun]

I saw the vote thread a few minutes ago. I voted. Thank you! :)

https://lists.apache.org/thread/lw6rqkvvth44ykncpjy52qjctmbdyrzq

[dongjoon-hyun]

+1, LGTM. I believe we can merge this after the vote passes.

[yaooqinn]

Thank you @dongjoon-hyun and @HyukjinKwon

[yaooqinn]

Since the vote passed, I will merge this to the master. There will be some followups to make this feature more user-friendly and less ambiguous.

Thank you @dongjoon-hyun @HyukjinKwon

[HyukjinKwon]

@yaooqinn let's revert this. I think it's better be tested in your fork first, and merge it in single Pr.

[HyukjinKwon]

I reverted it for now.

[dongjoon-hyun]

+1 for the reverting. Thank you.

[HyukjinKwon]

Seems like 58d73fe isn't reflected directly. @yaooqinn mind taking a look please?

[yaooqinn]

The GitHub action in my fork has passed - https://github.com/yaooqinn/spark/actions/runs/10917659739/job/30301422921.

It seems that the ASF infra setting causes the action here to fail. Let me create a jira ticket to consult the infra team.

[yaooqinn]

FYI, https://issues.apache.org/jira/browse/INFRA-26133

[HyukjinKwon]

Thanks for following it up!

[dongjoon-hyun]

Unfortunately, the CI is still broken. Do you happen to know how to remove it completely, @yaooqinn and @HyukjinKwon ?

• https://github.com/apache/spark/commits/master/
  

[yaooqinn]

Hi @dongjoon-hyun, this action was not set up by the action yaml we added. I can't find it's setting and don't know why it keeps running.

I'm asking for help from the ASF Infra team in that ticket

[dongjoon-hyun]

Thank you so much!