[FEATURE] Ragged Encoding. NOT WORKING YET. Only for Intermediate presentations

[Paulo-Mansano]

New classes: ColumnEncoderRagged, TransformDummySeparatedTest, TransformRaggedTest.
Modified: TfUtils, ColumnEncoder, EncoderFactory, TransformCustomTest

What changes were proposed?

• Added ColumnEncoderRagged in order to implement ragged array encoding.
• Added tests: TransformDummySeparatedTest, TransformRaggedTest.
• Modified TfUtils, ColumnEncoder, EncoderFactory just to add RAGGED as a encoding method.

How was this tested?

• Local tests not passed, still looking to make everything work.

Related Issues

• Still looking for the solutions and running tests to understand where are the problems.

[Baunsgaard]

Hi @Paulo-Mansano ,

Thanks for opening the PR!
It seems we slightly miscommunicated, since the intention is to 'modify' the existing transformations to construct ragged arrays as their metadata, not to create new transformations.

In specific we need to look at :

systemds/src/main/java/org/apache/sysds/runtime/transform/encode/Encoder.java

Line 70 in 692dcc3

void initMetaData(FrameBlock meta);

Where the metadata is allocated, (we want to modify it to allocate the ragged array)

and

systemds/src/main/java/org/apache/sysds/runtime/transform/encode/Encoder.java

Line 63 in 692dcc3

FrameBlock getMetaData(FrameBlock out);

Where the metadata frame is populated with information. (we want it to be in the form of a ragged array)

I hope this helps clarify things.

[codecov]

Codecov Report

❌ Patch coverage is 42.10526% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.94%. Comparing base (692dcc3) to head (137cf20).
⚠️ Report is 35 commits behind head on main.

Files with missing lines	Patch %	Lines
.../runtime/transform/encode/ColumnEncoderRagged.java	35.71%	48 Missing and 6 partials ⚠️
...sysds/runtime/transform/encode/EncoderFactory.java	87.50%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2282      +/-   ##
============================================
- Coverage     72.95%   72.94%   -0.01%     
- Complexity    46086    46103      +17     
============================================
  Files          1479     1480       +1     
  Lines        172654   172744      +90     
  Branches      33796    33817      +21     
============================================
+ Hits         125955   126007      +52     
- Misses        37201    37233      +32     
- Partials       9498     9504       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.