Blocking tree can not be saved in cloud environment

[sonalgoyal]

the blocking tree gets written using local file system apis which do not work in cloud env like databricks. If we do trainMatch in one go, the file probably goes into the local machine form where it will get picked up. Otherwise it will not work if we do train and match separately.

[sonalgoyal]

can you please check if there is a way to write the serialized blockingtree to parquet using spark.write? We may(?) need to coalesce to 1 to make sure only one file is written. Then we can read like standard spark files without worrying about the location being local. @navinrathore

[navinrathore]

From the typical datasources used in Spark, The solution mentioned https://stackoverflow.com/questions/35200988/writing-custom-java-objects-to-parquet looks promising. but it will need additional dependency jar (org.apache.parquet (parquet-avro.jar))For this or other format, we need schema of the object. It can get the same from class object of (Tree). But I am getting runtime issue to get the same, thoughPlease take a glance on the approach, we can discuss it then. or let me know if you have any suggestion
Stack OverflowStack Overflow
Writing custom java objects to Parquet
I have some custom java objects (which internally are composed of other custom objects). I wish to write these to HDFS in parquet format. Even after a lot of searching, most suggestions seem to be
2:48
Seems the getScehme() api here does not work for generic type. Our type is Tree
2:50
Can we have this schema defined separately?

[navinrathore]

Can’t we serialise the bytes?
Serealise object to bytearray and write that
Then read and reconstruct

We can only read binary_file. reading not supported

[sonalgoyal]

I don’t have a ready answer here but I think one of the following approaches should work.
A. Write object as bytes
B. Write as json
C. Build spark dataset with canopy class and read and write through that

in all cases, we should invoke the spark apis for reading and writing. So essentially serialise as byte/json/custom obj and build a dataset and use spark.read or spark.write

[navinrathore]

Option A (object in bytes) has worked. PR #120