Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EMR Serverless supported? #25

Open
set92 opened this issue Feb 21, 2025 · 0 comments
Open

EMR Serverless supported? #25

set92 opened this issue Feb 21, 2025 · 0 comments
Assignees

Comments

@set92
Copy link

set92 commented Feb 21, 2025

Describe the bug
I have done the installation on a jupyter notebook with an EMR Serverless 7.7.0. But when I try to access the URL from the Spark UI, it shows a white screen with only This request is not valid. Please generate a new URL..

Environment
spark version: 3.5
platform: EMR

To Reproduce
Steps to reproduce the behavior:

  1. Create EMR Serverless application.
  2. Open a Notebook, and run
    %%configure -f
    {
    "conf": {
    "spark.jars.packages" : "io.dataflint:spark_2.12:0.2.9",
    "spark.plugins": "io.dataflint.spark.SparkDataflintPlugin",
    "spark.sql.maxMetadataStringLength": "1000",
    "spark.dataflint.iceberg.autoCatalogDiscovery": "true",
    }
    }
  3. After running spark on another cell, you will get the link to access the Spark UI (https://spark-live-ui.emr-serverless.amazonaws.com/), which in my case takes me to https://j-00fqdv663u738h0a.dashboard.emr-serverless.us-east-1.amazonaws.com/.
  4. Try to access https://j-00fqdv663u738h0a.dashboard.emr-serverless.us-east-1.amazonaws.com/dataflint will return the error mentioned.

Expected behavior
Be able to access the interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants