Skip to content

Should we use this project current? #221

Open
@sola1tmy

Description

@sola1tmy

We are consider integration some spark related functions in our project.
But seems the progress for support spark 3.4 in this project still have a lot of work.

#195

we consider

  • java: it has a poor grammer to do hard work, but maybe the best way now, because it can communicate with both kotlin and spark
  • scala: seems hard communicate with kotlin since our main project is write in java and kt. Maybe there will have problems hard to solve.
  • kotlin: That's what I most like. But unfortunely Spark 3.4+ / -Connect support #195 remains, we can still use old version for a quite long time, but migration is Inevitable maybe after 1 or 2 years.

So should we use this project now? when will the next release come?And will the api keep same next release?

Activity

Jolanrensen

Jolanrensen commented on Jul 23, 2024

@Jolanrensen
Collaborator

Hi!
Thanks for your interest :) I'd say, if you can use the latest version of Spark (3.3) supported by the Kotlin API in your project, you could at least try if it works for you. The plan for 3.4+ is to keep the migration as small as possible.

I'm not yet sure whether the Kotlin Spark API should move to spark-connect only or also generic spark, but if you keep your project to the concepts mentioned here you should be good to go for any direction.
The main thing to take away is to try to stick to your own defined data classes for Datasets (so try to avoid Pair, Triple, and data classes from other libraries). This will make the eventual transition easier as for 3.4, encodable Kotlin data classes will have to be annotation with @Sparkify.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @sola1tmy@Jolanrensen

        Issue actions

          Should we use this project current? · Issue #221 · Kotlin/kotlin-spark-api