Is possbile to Mark a Column optional

[hantsy]

For example, in our projects, we have a reader to read CSV, but as time goes by, the column name is planned to change to a new column name.

@ColumnName("Old Name")
val oldName: String?

@ColumnName("New Name")
val newName: String

We need some time to migrate the new column names, so in the migration period, we should cover old and new version together, but Old Name should be optional.

Currently, if we upload the new CSV(use 'New Name' instead), it will lack the Old Name.

[koperagen]

Hi! Closest thing in DataFrame terms to optional column would be a column filled with nulls. df.convertTo operations automatically creates such columns when actual dataframe misses some, for example:

@DataSchema
interface MySchema {
    @ColumnName("Old Name")
    val oldName: String?

    @ColumnName("New Name")
    val newName: String
}

val res = dataFrameOf("New Name")("1").convertTo<MySchema>()
res

[hantsy]

In our new format, we do not contain the old column at all. There is a parsing error, if it is missing in the CSV file.

Here is our use case.

1. The old format contains the old column.
2. The new format use a new Col to replace the old column.(these files are not controlled by us, we can not add old column with null here.)

In the migration period (for example, two weeks), we want to the DataSchema cover these two formats. And in the future, we will update DataSchema to new format.