Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add code for sentiment analysis #417

Open
wants to merge 32 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
e1ec479
WIP
jaidevd Feb 23, 2021
4f5b50d
WIP: MLHandler support for Huggingface transformers
jaidevd Feb 24, 2021
52e72cb
WIP
jaidevd Feb 24, 2021
4b01634
WIP
jaidevd Feb 25, 2021
1f85677
Ensure label IDs are long ints
jaidevd Feb 26, 2021
6af50ea
FIX: Enforce correct order of features before prediction in MLHandler
jaidevd Mar 1, 2021
a3fa51f
WIP
jaidevd Mar 1, 2021
2322e8e
Merge branch 'jd-mlhandler-featnames-order' of github.com:gramener/gr…
jaidevd Mar 1, 2021
e6be0d5
WIP: MLHandler Refactoring
jaidevd Mar 2, 2021
2a30092
Merge branch 'master' of github.com:gramener/gramex into jd-transformers
jaidevd Mar 2, 2021
8317fe1
ENH: Transfomers - model persistence
jaidevd Mar 2, 2021
2d5b213
WIP
jaidevd Mar 3, 2021
168c7e1
Merge branch 'master' of github.com:gramener/gramex into jd-transformers
jaidevd Mar 3, 2021
3775939
WIP
jaidevd Mar 5, 2021
d446803
ENH: Modify gramex.install.safe_rmtree to remove files outside $GRAME…
jaidevd Mar 9, 2021
80b6c27
Add code for sentiment analysis and remove print statements
MSanKeys963 Jun 21, 2021
13116a0
Remove print statement
MSanKeys963 Jun 21, 2021
7f40258
Solve merge conflicts
MSanKeys963 Jul 6, 2021
dac5359
Add space
MSanKeys963 Jul 6, 2021
e9c6bb2
Add new install.py
MSanKeys963 Jul 7, 2021
07807c7
Remove space
MSanKeys963 Jul 9, 2021
4ee107d
Remove merge conflicts test_mlhandler.py
MSanKeys963 Jul 10, 2021
20ce370
Merge branch 'master' into jd-transformers
MSanKeys963 Jul 10, 2021
e4f59e0
Remove unnecessary space & lines
MSanKeys963 Jul 10, 2021
2f00a8e
Restore formhandler.py to original version & Minor Change to mlhandle…
MSanKeys963 Jul 13, 2021
e84a064
Remove unused function
MSanKeys963 Jul 13, 2021
dd3ec9c
Rename dl_utils.py to dl.py
MSanKeys963 Jul 20, 2021
d6a7132
Remove dl_utils.py
MSanKeys963 Jul 20, 2021
2d2cfb1
Fix imports
MSanKeys963 Jul 20, 2021
326b93d
Add changes to support new gramex.yaml config.
MSanKeys963 Aug 5, 2021
db0b20c
Remove spaces
MSanKeys963 Aug 5, 2021
7c59db3
Add changes to run PyTorch and Hugging Face only when requested and f…
MSanKeys963 Aug 9, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions gramex/dl_utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
from torch.utils.data import Dataset
MSanKeys963 marked this conversation as resolved.
Show resolved Hide resolved
import torch


class SentimentDataset(Dataset):
def __init__(self, encodings, labels):
self.encodings = encodings
self.labels = labels

def __getitem__(self, idx):
item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
item['labels'] = torch.tensor(self.labels[idx]).to(torch.int64)
return item

def __len__(self):
return len(self.labels)
1 change: 1 addition & 0 deletions gramex/handlers/formhandler.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ def setup(cls, **kwargs):
cls.headers = conf_kwargs.pop('headers', {})
# Top level formats: key is special. Don't treat it as data
cls.formats = conf_kwargs.pop('formats', {})
cls.task = conf_kwargs.pop('task', {})
MSanKeys963 marked this conversation as resolved.
Show resolved Hide resolved
default_config = conf_kwargs.pop('default', None)
# Remove other known special keys from dataset configuration
cls.clear_special_keys(conf_kwargs)
Expand Down
Loading