fear(#23): Replace filterIncompleteRecords boolean with Imputation Enum for Enhanced Data Handling #61
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the 'handler' package under 'de.eudx.data' to handle the cleaning and processing of incomplete records within a dataset, in correspondence with issue Samyssmile#23. Changes Made: - Added 'handler' package under 'de.edux.data' - Introduced 'EIncompleteRecordsHandlerStratetgy' enum with the constans: DO_NOT_HANDLE, DROP_RECORDS, FILL_RECORDS_WITH_AVERAGE - Introduced 'IIncompleteRecordsHandler' interface - Implemented 'DropIncompleteRecordsHandler' class - Implemented 'AverageFillIncompleteRecordsHandler' class - Implemented 'DoNotHandleIncompleteRecords' class because of DataProcessorTest.testLoadDataWithoutNormalizationAndShuffling() - Started to write the tests for the above mentioned classes and their methods - Revired the existing codebase to be compatible with the new enum
Samyssmile
requested changes
Oct 19, 2023
|
|
||
| return dataset.stream().filter(this::containsOnlyCompletedFeatures).toList(); | ||
| if (cleanedDataset.size() < dataset.size() * 0.5) { |
| return dataset.stream().filter(this::containsOnlyCompletedFeatures).toList(); | ||
| if (cleanedDataset.size() < dataset.size() * 0.5) { | ||
| throw new RuntimeException( | ||
| "More than 50% of the records will be dropped with this IncompleteRecordsHandlerStrategy. " |
| @@ -26,41 +26,51 @@ void initializeList() { | |||
| void dropRecordsWithIncompleteCategoricalFeature() { | |||
There was a problem hiding this comment.
looks like drop test in AverageFill Test class. Do Drop test in Drop Test Class and AverageFill Tests in AverageFill Test Classes...
| } | ||
|
|
||
| @Test | ||
| void testDropThreeIncompleteResults() { | ||
| void testDropTwoIncompleteResult() { |
There was a problem hiding this comment.
pls rename the tests with this pattern "should*What Test should do" e.g. shouldDropTwoIncompleResults
| @@ -0,0 +1,17 @@ | |||
| package de.edux.data.handler; | |||
|
|
|||
| public enum EIncompleteRecordsHandlerStrategy { | |||
There was a problem hiding this comment.
In Java World wie never prefix enums with 'E'. As in isssue#23 described you need name it "Imputation" here.
Imputation .DROP_RECORDS....
|
|
||
| import java.util.List; | ||
|
|
||
| public interface IIncompleteRecordsHandler { |
| public List<T> loadDataSetFromCSV(File csvFile, char csvSeparator, boolean normalize, boolean shuffle, boolean filterIncompleteRecords) { | ||
| List<String[]> x = csvDataReader.readFile(csvFile, csvSeparator); | ||
| List<T> unmodifiableDataset = csvDataReader.readFile(csvFile, csvSeparator) | ||
| public List<T> loadDataSetFromCSV(File csvFile, char csvSeparator, boolean normalize, boolean shuffle, EIncompleteRecordsHandlerStrategy incompleteRecordHandlerStrategy) { |
There was a problem hiding this comment.
We need one more method here
public List<T> loadDataSetFromCSV(File csvFile, char csvSeparator, boolean normalize, boolean shuffle){ this.loadDataSetFromCSV(...., Imputation.Do Nothing)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the 'handler' package under 'de.eudx.data' to handle the cleaning and processing
of incomplete records within a dataset, in correspondence with issue #23.
Changes Made:
DO_NOT_HANDLE, DROP_RECORDS, FILL_RECORDS_WITH_AVERAGE
of DataProcessorTest.testLoadDataWithoutNormalizationAndShuffling()