Encoder refactor V2 #2370

dantreiman · 2022-08-09T23:24:16Z

This PR contains all of Connor's commits from #2269 (:encoder_refactor) merged atop the master branch, plus additional refactoring and test fixes in :daniel/encoder_refactor

master -----------------+------+>
                        |      |
daniel/encoder_refactor |      +*---->
                        |       | 
encoder_refactor        +------->

+ = branch
* = merge commit

A few notes:

We have a glue function load_config_with_kwargs which loads a config from a dictionary, and collects any key-value pairs not captured by the config in an extra kwargs dictionary. I removed this from instantiation of input features, now all input feature and encoder params are captured by the config.
load_config_with_kwargs is still in use for combiners to pass in output_size.

…oders

ludwig/schema/features/base.py

…iel/encoder_refactor

for more information, see https://pre-commit.ci

ludwig/features/bag_feature.py

ludwig/features/feature_utils.py

ludwig/features/bag_feature.py

ludwig/features/sequence_feature.py

ludwig/trainers/trainer_lightgbm.py

…an/ludwig into daniel/encoder_refactor

for more information, see https://pre-commit.ci

…k params and threshold back to output feature.

…ead of storing it in the decoder.

…if they are present on output features.

connor-mccorm added 30 commits July 22, 2022 15:08

Added base files and some initial code

d288dea

More files created, fleshing out binary feature and corresponding enc…

50b2a00

…oders

Added more schema infra

2ae5523

Registered all feature encoders

060ba1b

Separated feature utils infra

008d805

Added all preprocessing classes

6726ed1

Filled out rest of schema configs

04f694c

Fixed preproc dataclass

4f4fa39

Fixed small errors blocking import

36ad46b

Tests should be passing

1b6fb5d

Deleted unnecesssary files and removed commented out code

6132b50

fixed flake8

aacb394

Fixed most tests

2124ee5

fixed pattern validation

934f179

Fixed missing val strategies and solved custom encoder update issue

03fbdf0

Removed preprocessing from features due to schema SSOT

fe12df0

fix flake 8

0d4135a

Started encoder schema work

72634be

Parallel CNN Encoder

d351899

StackedCNN Encoder

086aaac

Added image encoders

65ac68f

Finished sequence encoders

4d295ec

Partway through text encoders

bcaeade

Added text encoders

9caac37

Bag Encoders

5732418

Binary and Date Encoders

e30f5e6

category, date, h3, and set encoders

577907c

Wired up encoder schemas

7df04d4

Switched input feature encoder schema definitions

7b3c9c0

Fixed handful of issues

3463ef1

connor-mccorm reviewed Aug 16, 2022

View reviewed changes

ludwig/schema/features/base.py Show resolved Hide resolved

connor-mccorm and others added 3 commits August 16, 2022 17:08

Merge branch 'master' of https://github.com/ludwig-ai/ludwig into dan…

6494150

…iel/encoder_refactor

[pre-commit.ci] auto fixes from pre-commit.com hooks

628be76

for more information, see https://pre-commit.ci

Adds test for construction of output features with dependencies.

61a06f0