Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OverflowError: Python int too large to convert to C long #989

Closed
dazza-codes opened this issue Dec 5, 2020 · 3 comments
Closed

OverflowError: Python int too large to convert to C long #989

dazza-codes opened this issue Dec 5, 2020 · 3 comments
Assignees

Comments

@dazza-codes
Copy link

Obscure error when writing GeoJSONSeq to AWS S3:

    csv_gdf.to_file(s3uri, driver="GeoJSONSeq")
  File "/opt/conda/envs/project/lib/python3.7/site-packages/geopandas/geodataframe.py", line 746, in to_file
    _to_file(self, filename, driver, schema, index, **kwargs)
  File "/opt/conda/envs/project/lib/python3.7/site-packages/geopandas/io/file.py", line 257, in _to_file
    colxn.writerecords(df.iterfeatures())
  File "/opt/conda/envs/project/lib/python3.7/site-packages/fiona/collection.py", line 361, in writerecords
    self.session.writerecs(records, self)
  File "fiona/ogrext.pyx", line 1291, in fiona.ogrext.WritingSession.writerecs
  File "fiona/ogrext.pyx", line 390, in fiona.ogrext.OGRFeatureBuilder.build
OverflowError: Python int too large to convert to C long

some gdf info

>>> csv_gdf.info()
<class 'geopandas.geodataframe.GeoDataFrame'>
RangeIndex: 830 entries, 0 to 829
Data columns (total 13 columns):
 #   Column         Non-Null Count  Dtype   
---  ------         --------------  -----   
 0   locationId     830 non-null    int64   
 1   region         830 non-null    object  
 2   countryName    830 non-null    object  
 4   cityName       830 non-null    object  
 5   streetAddress  830 non-null    object  
 6   latitude       830 non-null    float64 
 7   longitude      830 non-null    float64 
 9   geometry       830 non-null    geometry
 10  uuid           830 non-null    object  
 11  s2_cell_id     830 non-null    uint64  
dtypes: float64(2), geometry(1), int64(1), object(8), uint64(1)
memory usage: 84.4+ KB

versions

>>> fiona.__version__
'1.8.18'
>>> fiona.__gdal_version__
'2.4.4'
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

$ python --version
Python 3.7.8

$ pip --version
pip 20.3.1 from /opt/conda/envs/project/lib/python3.7/site-packages/pip (python 3.7)

Installed from pip package/wheel (not from conda).

@rbuffat
Copy link
Contributor

rbuffat commented Dec 5, 2020

Could you check if the maximum integer you want to write is bigger than 2 ** 63 - 1? This is the maximum integer size gdal supports. (OGR_F_SetFieldInteger64 takes a long long as input value)

@dazza-codes
Copy link
Author

I might have isolated this to some google s2 CellId values

  • https://s2geometry.io/devguide/s2cell_hierarchy.html
  • converting from an int to a string for those values avoids the fiona crash, not sure what other options are available for JSON serializable values; it's curious that the s2 cell id as a uint64 seemed to be JSON serializable in vanilla JSON but not for fiona/gdal (IIRC)

@sgillies
Copy link
Member

Converting these cell ids to strings is the only solution. OGR simply can't handle them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants