Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace filterIncompleteRecords boolean with Imputation Enum for Enhanced Data Handling #23

Closed
Samyssmile opened this issue Sep 27, 2023 · 1 comment
Closed

Replace filterIncompleteRecords boolean with Imputation Enum for Enhanced Data Handling #23

Samyssmile opened this issue Sep 27, 2023 · 1 comment
Assignees
@acsolle66
Labels
enhancement New feature or request hacktoberfest hacktoberfest 2023 New Feature

Comments

@Samyssmile
Copy link
Owner

Samyssmile commented Sep 27, 2023 •
edited
Loading

There is an open discussion for this ticket. #62
In our data processing logic, we currently use a boolean named filterIncompleteRecords to decide whether to filter out incomplete records or not. To provide more flexibility and future extensibility in handling missing data, we propose replacing this boolean with an enum named Imputation.

Let see on this example tiger dataset.

Tiger size
Syberian Tiger Big ok
Indian Tiger Small ok
Amur Tiger Missed Value in dataset

With filterIncompleteRecords = true, our Data Processing Class will auto remove Amur Tiger from the dataset, because the size value is missed.

The new Imputation enum would have various strategies such as MEDIAN, MODE, FILTEROUT, and AVERAGE. For the scope of this ticket, we'll focus on implementing FILTEROUT and AVERAGE, MODE.

Advantages of this new Feature

A user can auto fill missed values.

Hint

How filterIncompleteRecords works you can see in the Seaborn Pinguin k-nearest neighbors algorithm (knn) example.
https://github.com/Samyssmile/edux/blob/main/example/src/main/java/de/example/knn/KnnSeabornExample.java

https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00516-9
@Samyssmile Samyssmile added enhancement New feature or request New Feature labels Sep 27, 2023
@acsolle66 acsolle66 self-assigned this Oct 13, 2023
@acsolle66
Copy link
Collaborator

I will take this ticket, gather my thoughts on how I would solve this, and present my idea.

acsolle66 added a commit to acsolle66/edux that referenced this issue Oct 18, 2023
@acsolle66
feat(23): add package for handling incomplete records
85eebf8 
This commit introduces the 'handler' package under 'de.eudx.data' to handle the cleaning and processing
of incomplete records within a dataset, in correspondence with issue Samyssmile#23.

Changes Made:
- Added 'handler' package under 'de.edux.data'
- Introduced 'EIncompleteRecordsHandlerStratetgy' enum with the constans:
  DO_NOT_HANDLE, DROP_RECORDS, FILL_RECORDS_WITH_AVERAGE
- Introduced 'IIncompleteRecordsHandler' interface
- Implemented 'DropIncompleteRecordsHandler' class
- Implemented 'AverageFillIncompleteRecordsHandler' class
- Implemented 'DoNotHandleIncompleteRecords' class because
  of DataProcessorTest.testLoadDataWithoutNormalizationAndShuffling()
- Started to write the tests for the above mentioned classes and their methods
- Revired the existing codebase to be compatible with the new enum
acsolle66 added a commit to acsolle66/edux that referenced this issue Oct 19, 2023
@acsolle66
feat(Samyssmile#23): add exception handling and tests for incompleteR…
af01c5a 
…ecordsHandlers
@acsolle66 acsolle66 mentioned this issue Oct 19, 2023
Closed
Samyssmile pushed a commit to acsolle66/edux that referenced this issue Oct 20, 2023
feat(Samyssmile#23): Prepare
8c969fd
@Samyssmile Samyssmile added the hacktoberfest hacktoberfest 2023 label Oct 20, 2023
acsolle66 added a commit to acsolle66/edux that referenced this issue Oct 30, 2023
@acsolle66
feat(23): add package for handling incomplete records
f1ddf4b 
This commit introduces the 'handler' package under 'de.eudx.data' to handle the cleaning and processing
of incomplete records within a dataset, in correspondence with issue Samyssmile#23.

Changes Made:
- Added 'handler' package under 'de.edux.data'
- Introduced 'EIncompleteRecordsHandlerStratetgy' enum with the constans:
  DO_NOT_HANDLE, DROP_RECORDS, FILL_RECORDS_WITH_AVERAGE
- Introduced 'IIncompleteRecordsHandler' interface
- Implemented 'DropIncompleteRecordsHandler' class
- Implemented 'AverageFillIncompleteRecordsHandler' class
- Implemented 'DoNotHandleIncompleteRecords' class because
  of DataProcessorTest.testLoadDataWithoutNormalizationAndShuffling()
- Started to write the tests for the above mentioned classes and their methods
- Revired the existing codebase to be compatible with the new enum
acsolle66 added a commit to acsolle66/edux that referenced this issue Oct 30, 2023
@acsolle66
feat(Samyssmile#23): add exception handling and tests for incompleteR…
c498e8f 
…ecordsHandlers
acsolle66 pushed a commit to acsolle66/edux that referenced this issue Oct 30, 2023
@acsolle66
feat(Samyssmile#23): Prepare
6f86149
Samyssmile added a commit that referenced this issue Oct 31, 2023
@Samyssmile
Merge pull request #72 from acsolle66/feat#23
e6dd0c3 
Feat(#23) Replace filterIncompleteRecords boolean with Imputation Enum for Enhanced Data Handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@acsolle66 acsolle66

Labels
enhancement New feature or request hacktoberfest hacktoberfest 2023 New Feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Samyssmile @acsolle66