Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to run a job with main file point to s3 bucket #2301

Open
nownikhil opened this issue Oct 31, 2024 · 1 comment
Open

Unable to run a job with main file point to s3 bucket #2301

nownikhil opened this issue Oct 31, 2024 · 1 comment

Comments

@nownikhil
Copy link

nownikhil commented Oct 31, 2024

What question do you want to ask?

Error

│ Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found                          │
│   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)                                                                              │
│   at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)                                                                            │
│   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)                                                                              │
│   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)                                                                                     │
│   at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)                                                                             │
│   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)                                                                                     │
│   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)                                                                                            │
│   at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831)                                                                                  │
│   at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:727)                                                                                           │
│   at org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264)                                                                      │
│   at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:103)                                                         │
│   ... 18 more                                                                                                                                            │
│ Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found                                                      │
│   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)                                                                        │
│   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)                                                                              │

Additional context

This is happening because spark-operator image doesn't have hadoop-aws jar. Is there a recommended way to pull jars from S3?

No response

Have the same question?

Give it a 👍 We prioritize the question with most 👍

@ujjawal-khare
Copy link

Facing the same issue, the only solution that I found was to bake the hadoop jar inside the operator but not sure if this is the right way as we would be having dynamic jars coming in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants