Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-45195] Update examples with docker official image #477

Closed
wants to merge 1 commit into from

Conversation

zhengruifeng
Copy link
Contributor

@zhengruifeng zhengruifeng commented Sep 18, 2023

1, add docker run commands for PySpark and SparkR;
2, switch to docker official image for SQL, Scala and Java;

refer to https://hub.docker.com/_/spark

also manually checked all the commands, e,g,:

ruifeng.zheng@xxxxx:~$ docker run -it --rm spark:python3 /opt/spark/bin/pyspark
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/09/18 06:02:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.0
      /_/

Using Python version 3.8.10 (default, May 26 2023 14:05:08)
Spark context Web UI available at http://4861f70118ab:4040
Spark context available as 'sc' (master = local[*], app id = local-1695016951087).
SparkSession available as 'spark'.
>>> spark.range(0, 10).show()
+---+
| id|
+---+
|  0|
|  1|
|  2|
|  3|
|  4|
|  5|
|  6|
|  7|
|  8|
|  9|
+---+

update
@zhengruifeng zhengruifeng changed the title Update Example with docker official image [SPARK-45195] Update example with docker official image Sep 18, 2023
@zhengruifeng
Copy link
Contributor Author

@zhengruifeng zhengruifeng changed the title [SPARK-45195] Update example with docker official image [SPARK-45195] Update examples with docker official image Sep 18, 2023
@srowen srowen closed this in 6b10f7f Sep 18, 2023
@zhengruifeng zhengruifeng deleted the offical_image branch September 18, 2023 12:32
Copy link
Contributor

@allisonwang-db allisonwang-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update!

Comment on lines -91 to +97
<div style="font-size: 16px;">Installing with 'pip'
<div style="font-size: 16px;">Install with 'pip' or try offical image
</div>
<div class="code">
<p>$ pip install pyspark</p>
<p>$ pyspark</p>
<p>$ </p>
<p>$ docker run -it --rm spark:python3 /opt/spark/bin/pyspark</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhengruifeng I feel this "try official image" is a bit unclear. Maybe we can reformat to this?

Install with pip
<code block>
Install with docker
<code block>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sense, will send a follow up pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants