TypeError: ufunc 'isnan' not supported for the input types...

[mitchdetailed]

Hoping for some clarification .

using socketcan .log files , I receive the following errors after installing on my new laptop, (mint 20.04, same as desktop)

File "Main.py", line 85, in
id_dictionary, j1979_dictionary = pre_processor.generate_arb_id_dictionary(a_timer,
File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 87, in generate_arb_id_dictionary
self.import_csv(a_timer, self.data_filename)
File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 43, in import_csv
self.data = read_csv(filename,
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read
data = parser.read(nrows)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read
ret = self._engine.read(nrows)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 2231, in read
index, names = self._make_index(data, alldata, names)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1677, in _make_index
index = self._agg_index(index)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1770, in _agg_index
arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1871, in _infer_types
mask = algorithms.isin(values, list(na_values))
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py", line 443, in isin
if np.isnan(values).any():
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

odd thing is, it doesn't happen on my desktop.. (Same OS)

[brent-stone]

That's weird .isnan() of all things is giving the error. Did you already look at where the "strict" flag is set in your environment?

…

On Tue, Oct 6, 2020, 5:53 PM Mitch ***@***.***> wrote: Hoping for some clarification . using socketcan .log files , I receive the following errors after installing on my new laptop, (mint 20.04, same as desktop) File "Main.py", line 85, in id_dictionary, j1979_dictionary = pre_processor.generate_arb_id_dictionary(a_timer, File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 87, in generate_arb_id_dictionary self.import_csv(a_timer, self.data_filename) File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 43, in import_csv self.data = read_csv(filename, File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv return _read(filepath_or_buffer, kwds) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read data = parser.read(nrows) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read ret = self._engine.read(nrows) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 2231, in read index, names = self._make_index(data, alldata, names) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1677, in _make_index index = self._agg_index(index) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1770, in _agg_index arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1871, in _infer_types mask = algorithms.isin(values, list(na_values)) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py", line 443, in isin if np.isnan(values).any(): TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' odd thing is, it doesn't happen on my desktop.. (Same OS) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEG4NYDSMPOEBKVSIOSMNXDSJOGUDANCNFSM4SGS6CZQ> .

[JoshuaArking]

Hoping for some clarification .

using socketcan .log files , I receive the following errors after installing on my new laptop, (mint 20.04, same as desktop)

File "Main.py", line 85, in
id_dictionary, j1979_dictionary = pre_processor.generate_arb_id_dictionary(a_timer,
File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 87, in generate_arb_id_dictionary
self.import_csv(a_timer, self.data_filename)
File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 43, in import_csv
self.data = read_csv(filename,
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read
data = parser.read(nrows)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read
ret = self._engine.read(nrows)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 2231, in read
index, names = self._make_index(data, alldata, names)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1677, in _make_index
index = self._agg_index(index)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1770, in _agg_index
arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1871, in _infer_types
mask = algorithms.isin(values, list(na_values))
File "/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py", line 443, in isin
if np.isnan(values).any():
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

odd thing is, it doesn't happen on my desktop.. (Same OS)

After some digging, it appears to be an issue with pandas 1.1.3 specifically, and the issue is already being worked on in the pandas-dev repository. Here's a link to the issue tracker on the pandas-dev repo: https://github.com/pandas-dev/pandas/issues/37094 Currently the bug is slated to be fixed in the 1.1.4 release which is due to come out tomorrow on October 30th. It appears that it is related to the index_col argument for read_csv() when > 1000000 lines, so a quick fix would be to set index separately from read_csv() instead of including it as an argument or to process fewer than 1000000 lines per file. Here's an example from the issue tracker:

I can confirm I have the same problem, which arises as soon as I pass the threshold of 1M rows.

I only need to specify index_col to get the bug, though. Specifying dtypes is not needed.

Pandas 1.1.3

And as a temporary workaround, I am reading without index_col, and then setting my index. E.g.

df = pd.read_csv(filepath, nrows=1000001)
df.set_index(0) # for example if I wanted the first column

In short, upgrade to pandas 1.1.4 or downgrade to pandas 1.1.2 and this should be fixed. Since the issue is not with this repo I don't think any changes are needed besides changing requirements.txt line 2 from
pandas
to
pandas>=1.1.4
or
pandas<=1.1.2
I am not familiar with requirements.txt syntax, so if there's a way to specifically exclude only version 1.1.3, that would be ideal.

[brent-stone]

Thanks for following up on this. There's three lines in this project for bandaid fixing another pandas indexing bug. I spent about two weeks trying to figure out what I was doing wrong. Only problem is it would took sharing a private data file to demonstrate how to reproduce the error so I didn't create an issue with Pandas :(

…

On Thu, Oct 29, 2020, 10:50 AM Joshua Arking ***@***.***> wrote: Hoping for some clarification . using socketcan .log files , I receive the following errors after installing on my new laptop, (mint 20.04, same as desktop) File "Main.py", line 85, in id_dictionary, j1979_dictionary = pre_processor.generate_arb_id_dictionary(a_timer, File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 87, in generate_arb_id_dictionary self.import_csv(a_timer, self.data_filename) File "/home/mitch/Canbus/CAN_Reverse_Engineering/Pipeline/PreProcessor.py", line 43, in import_csv self.data = read_csv(filename, File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv return _read(filepath_or_buffer, kwds) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read data = parser.read(nrows) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read ret = self._engine.read(nrows) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 2231, in read index, names = self._make_index(data, alldata, names) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1677, in _make_index index = self._agg_index(index) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1770, in _agg_index arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1871, in _infer_types mask = algorithms.isin(values, list(na_values)) File "/home/mitch/.local/lib/python3.8/site-packages/pandas/core/algorithms.py", line 443, in isin if np.isnan(values).any(): TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' odd thing is, it doesn't happen on my desktop.. (Same OS) After some digging, it appears to be an issue with pandas 1.1.3 specifically, and the issue is already being worked on in the pandas-dev repository. Here's a link to the issue tracker on the pandas-dev repo: pandas-dev/pandas#37094 <http://url> Currently the bug is slated to be fixed in the 1.1.4 release which is due to come out tomorrow on October 30th. It appears that it is related to the index_col argument for read_csv() when > 1000000 lines, so a quick fix would be to set index separately from read_csv() instead of including it as an argument or to process fewer than 1000000 lines per file. Here's an example from the issue tracker: I can confirm I have the same problem, which arises as soon as I pass the threshold of 1M rows. I only need to specify index_col to get the bug, though. Specifying dtypes is not needed. Pandas 1.1.3 And as a temporary workaround, I am reading without index_col, and then setting my index. E.g. df = pd.read_csv(filepath, nrows=1000001)df.set_index(0) # for example if I wanted the first column In short, upgrade to pandas 1.1.4 or downgrade to pandas 1.1.2 and this should be fixed. Since the issue is not with this repo I don't think any changes are needed besides changing requirements.txt line 2 from pandas to pandas>=1.1.4 or pandas<=1.1.2 I am not familiar with requirements.txt syntax, so if there's a way to specifically exclude only version 1.1.3, that would be ideal. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEG4NYC5CIUAC4VHHYTEOY3SNF6MZANCNFSM4SGS6CZQ> .