Add convert asColumn operation as compiler plugin friendly variant oа replace with

[koperagen]

I think it's more or less common to perform column-wide operations in async / parallel context, so having such compiler plugin friendly operation is useful
Another use case, although not as handy, is creating a ColumnGroup like
df.convert { col }.asColumn { dataFrameOf("a" to listOf(123), "b" to listOf(321).asColumnGroup() }

[zaleslaw]

Parallel streams should be avoided in single-threaded libraries because they can introduce race conditions, synchronization issues, and unnecessary overhead.

However, in this case, the use of parallelStream is localized and safe, as it only transforms column values without affecting global state or column names.

But! Even though this use of parallelStream is logically safe, it can unexpectedly increase CPU load, especially on weaker machines. Parallel streams use the shared ForkJoinPool, which may cause performance issues if the system has limited resources or is already running other parallel tasks. This can lead to slowdowns or contention for threads, impacting the overall responsiveness of the application.

When running in Kotlin Notebooks, parallel streams can compete with notebook execution and UI rendering for limited CPU resources. This may cause lags or freezes, especially in constrained environments like containers or shared servers. Therefore, parallelism in notebooks should be used carefully to avoid degrading the interactive experience.

[koperagen]

Usually i wouldn't recommend using parallel streams for any trivial operations, but there can be a situation when it's needed. For example, one time i used a library that performs IO, parses file into kind of AST. So

df.add("data") { 
  Library.parse(file)
}

In my case single threaded execution took literally minutes of real time because CPU was loaded 5%. Opting to use parallel made it 20 times faster. #723. But in any case, we should minimize number of operations that plugin can't understand. replace with is one of them, convert asColumn would be an alternative should users need it

[Jolanrensen]

How is asColumn different from public fun <T, C> Convert<T, C>.to(columnConverter: DataFrame<T>.(DataColumn<C>) -> AnyBaseCol): DataFrame<T> we already have? Just an extra type R right?

Then I'd either add R to convert.to { } or deprecate to {} in favor of the better named asColumn {}. wdyt?

[Jolanrensen]

Actually, why don't we just add the type R to ReplaceClause.with if you want compiler plugin support for it?

[koperagen]

Then I'd either add R to convert.to { } or deprecate to {} in favor of the better named asColumn {}. wdyt?

The major difference is ignoring name changes. So, let's say df.replace { col }.with { produceRandomColumnWithRandomName(it) } is a valid code - but impossible for the plugin to interpret. So convert asColumn renames all new columns to original name

[koperagen]

@Jolanrensen I think because convert { }.to { } works differently, let's postpone decision whether it should be removed or not

[Jolanrensen]

@koperagen ooh right I see. Actually I don't think many people would mind if we restricted that names cannot change for the entire convert operation. For that they'd need replace. That way we could make convert {}.to {} and convert {}.asColumn {} the same and replace {}.with {} different.