Added 3 python files in reference to the Task Maintioned in TODO for … #30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Ensemble_strategy, GTPModel and XLNetTransformer.
NirantK
requested changes
Apr 19, 2024
| return predicted_labels | ||
|
|
||
| def main(): | ||
| data_path = "/kaggle/input/imdb-dataset-of-50k-movie-reviews/IMDB Dataset.csv" |
There was a problem hiding this comment.
Can you please avoid hard coding the paths?
Comment on lines
+16
to
+22
| def clean(text): | ||
| for token in ["<br/>", "<br>", "<br>"]: | ||
| text = re.sub(token, " ", text) | ||
|
|
||
| text = re.sub("[\s+\.\!\/_,$%^*()\(\)<>+\"\[\]\-\?;:\'{}`]+|[+——！，。？、~@#￥%……&*（）]+", " ", text) | ||
|
|
||
| return text.lower() |
There was a problem hiding this comment.
Given that this clean is being re-used everywhere, do you want to refactor this out into a separate file?
Comment on lines
+25
to
+29
| def load_imdb_dataset(data_path, nrows=100): | ||
| df = pd.read_csv(data_path, nrows=nrows) | ||
| texts = df['review'].apply(clean) | ||
| labels = df['sentiment'] | ||
| return texts, labels |
There was a problem hiding this comment.
Perhaps a good idea to not tie to this to new data loaders and use/upgrade the existing ones?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GPTModel.py: Added a new Python file GPTModel.py which implements a text classification model based on the GPT architecture. This file contains the necessary code to preprocess text data, train the GPT model, and make predictions on new text inputs.
ensemble_method.py: Introduced a new Python file ensemble_method.py that implements an ensemble learning method. This method combines predictions from multiple base models to improve overall performance and robustness.
XLNetTransformer.py: Implemented a transformer-based feature transformer in XLNetTransformer.py. This file contains a class that preprocesses text data using the XLNet tokenizer, encoding it into input IDs suitable for the XLNet model.
These additions address the TODO tasks specified in the issues tab, enhancing the project's functionality and providing more options for text classification and model ensemble techniques.