Sort and finalize throws when staged columns are of different but promotable types (dynamic schema)

[vasil-pashov]

Describe the bug

If one column is of type int64 and the same column is in another segment but of type int32 sort and finalize will throw. However the stream descriptor will be of the largest type because merge descriptors uses valid_common_type.

Steps/Code to Reproduce

import pandas as pd
import numpy as np
import arcticdb as adb
from arcticdb.version_store.library import StagedDataFinalizeMethod
ac = adb.Arctic("lmdb://test")
opts = adb.LibraryOptions(dynamic_schema=True)
lib = ac.get_library("test", create_if_missing=True, library_options=opts)
symbol = "test"
lib.write("sym", pd.DataFrame({"a": np.array([1], dtype="int32")}, index=pd.DatetimeIndex([pd.Timestamp(2024, 1, 1)])), staged=True)
lib.write("sym", pd.DataFrame({"a": np.array([1], dtype="int64")}, index=pd.DatetimeIndex([pd.Timestamp(2024, 1, 2)])), staged=True)
lib.sort_and_finalize_staged_data("sym", mode=StagedDataFinalizeMethod.WRITE)

Throws:

InternalException: E_ASSERTION_FAILURE Type mismatch in set_scalar, expected 4

Note the exception is thrown after all descriptors were resolved, during the merge phase. The exception is thrown when setting a value in a column.

finalize_staged_data does not allow having staged of different (but compatible types). The difference is due to this

Expected Results

Two options:

• Promote the "smaller" type and succeed
• Act as finalize_staged_segments and require exactly the same types in all staged segments

OS, Python Version and ArcticDB Version

Python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
OS: Windows-10-10.0.22631-SP0
ArcticDB: 4.5.0

Backend storage used

No response

Additional Context

No response