Feat(#23) Replace filterIncompleteRecords boolean with Imputation Enum for Enhanced Data Handling #72
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the 'handler' package under 'de.eudx.data' to handle the cleaning and processing of incomplete records within a dataset, in correspondence with issue Samyssmile#23. Changes Made: - Added 'handler' package under 'de.edux.data' - Introduced 'EIncompleteRecordsHandlerStratetgy' enum with the constans: DO_NOT_HANDLE, DROP_RECORDS, FILL_RECORDS_WITH_AVERAGE - Introduced 'IIncompleteRecordsHandler' interface - Implemented 'DropIncompleteRecordsHandler' class - Implemented 'AverageFillIncompleteRecordsHandler' class - Implemented 'DoNotHandleIncompleteRecords' class because of DataProcessorTest.testLoadDataWithoutNormalizationAndShuffling() - Started to write the tests for the above mentioned classes and their methods - Revired the existing codebase to be compatible with the new enum
…m for Enhanced Data Handling Implemented the ImputationStrategy enum with constants of average and mode and the corresponding classes.
Samyssmile
reviewed
Oct 31, 2023
|
|
||
| @Override | ||
| public double[][] getTestFeatures( int[] inputColumns) { | ||
| return getInputs(testData, inputColumns); | ||
| public void drop_incomplete_records() { |
There was a problem hiding this comment.
In Java we dont use snake case. Use camelCase
There was a problem hiding this comment.
Use snake case only for constants private final static String MY_CONSTANT
There was a problem hiding this comment.
I have writed a few lines in python before, and it seems I forgot how to name variables in Java 😅
| public double[][] getTestFeatures( int[] inputColumns) { | ||
| return getInputs(testData, inputColumns); | ||
| public void drop_incomplete_records() { | ||
| dataset = dataset.stream().filter((record) -> { |
There was a problem hiding this comment.
do not mix fuctional way and classic way. I suggest straight function way here
There was a problem hiding this comment.
What do you mean ?
| @@ -1,5 +1,17 @@ | |||
| package de.edux.functions.imputation; | |||
|
|
|||
| public enum ImputationStrategy { | |||
| DUMMY, MEAN, AVERAGE, MODE; | |||
| //TODO: DUMMY, MEAN | |||
|
|
||
| @Test | ||
| void performImputationWithNumericalValuesTest() { | ||
| String[] numerical_features_with_missing_values = {"1", "","2", "3", "", "4"}; |
| skipHead(); | ||
| } | ||
|
|
||
| List<String> uniqueClasses = new ArrayList<>(); |
There was a problem hiding this comment.
Use List<String> uniqueClasses = dataset.stream().map(row -> row[targetColumn]).distinct().toList();
Average imputation throwed RuntimeException beacuse the condition in the isDigit() method was returning false for blank values. Fixed with an or operator.
Samyssmile
approved these changes
Oct 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.