OverflowError: Invalid Nan value when encoding double #11757
Comments
|
This looks like a duplicate of #10217. It looks like the issue is that you are using the enumerated ID of your labels as the value of Out of curiosity, how did you come up with the current code for assigning labels? |
polm
added
training
|
Brilliant, thank you. I was using the code in this Medium article building-a-text-classifier-with-spacy. I was under the assumption that the label had to be unique. Maybe the documentation needs to be clearer? |
|
It's true that the label has to be unique - duplicate labels will be treated as the same thing - but the label value does not have to be unique, it just needs to be 0 or 1. This is actually called out in the API docs for textcat, which were updated to be clearer as part of #9041. Is there any other place you were checking where we could make this more explicit? I will look at adding a check for this during training. |
|
Ah I did not read the documentation properly. I think it seemed a bit overwhelming (which I think is a little my fault) which is why I went to other tutorials. I think the error message is also not very clear. I think there should be some sort of page on the spaCy section on TextCategorizer or Training Pipelines & Models with common issues and examples of how to fix them. |
|
Thanks for the suggestions, and the note that the documentation seems overwhelming. The error message here is definitely unhelpful, and I've written a PR (#11763) to help with that part. We do have an FAQ label in Discussions, and a top-level FAQ, but I'll see if there's some way we can make these more accessible from the main docs. |
|
Thank you very much 😃 |
|
This issue has been automatically closed because it was answered and there was no follow-up discussion. |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
How to reproduce the behaviour
An extract of my python code:
... def train(self) -> None: """ Train the model on the given dataset for the given number of epochs. """ if self.config.rebuild_data: self._prepare_data() if self.config.rebuild_config: os.system( "poetry run python -m spacy init fill-config .\\data\\intents\\base_config.cfg .\\data\\intents\\config.cfg" ) os.system( "poetry run python -m spacy train .\\data\\intents\\config.cfg --output .\\models\\intents" ) def _make_spacy_docs( self, data: list[tuple[str, str]], labels_to_idx: dict[str, int] ) -> list: """ Helper function to create take a list of texts and labels and create a list of spaCy docs. returns: A list of spaCy docs. """ docs = [] for doc, label in tqdm(self.nlp.pipe(data, as_tuples=True), total=len(data)): doc.cats[label] = labels_to_idx[label] docs.append(doc) return docs def _prepare_data(self) -> None: """ Helper function to prepare and save the data for training and validation. """ dataset = data.IntentClassifierDataset( Path(self.config.data_path), shuffle=True ) train_data, test_data = dataset.split(self.config.train_percentage) labels_to_idx = {label: idx for idx, label in enumerate(dataset.intents)} train_docs = self._make_spacy_docs(train_data.values.tolist(), labels_to_idx) test_docs = self._make_spacy_docs(test_data.values.tolist(), labels_to_idx) train_bin = DocBin(docs=train_docs) test_bin = DocBin(docs=test_docs) train_bin.to_disk(self.config.train_data_save_path) test_bin.to_disk(self.config.valid_data_save_path) ...After creating the data and running the spaCy train loop I get an overflow error:
Your Environment
The text was updated successfully, but these errors were encountered: