-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GDAL errors are also "printed" to stdout in addition to raised as Python exception #91
Comments
I noticed that the GDAL python SWIG bindings have this handler defined, which I suppose is basically what we want as well (hide errors logs because those are already handled as python exception):
(copying the snippet here, because doesn't allow to link to the line: https://github.com/OSGeo/gdal/blob/master/swig/python/extensions/ogr_wrap.cpp) There is also a builtin |
This also affects warnings. Ideally, you'd be able to supress warnings using the stdlib, but since those are simply logged from gdal there is no way to do so. The annoying one I'm facing:
|
@Oreilles did you figure out a way to solve that? I write the same file with fiona and it doesn't complain. But pyogrio does. Would love to write at the speed of pyogrio. I assume this negatively impacts reading the files afterwards as well? Are the values NULL? Or are they truncated floats? Would be good to know what happens. @brendan-ward any ideas? |
I could not fix the bug, however, I found it was possible to redirect the GDAL output elsewhere so as to not pollute my program output. Here is the code to just throw GDAL's output away (Change from pyogrio import set_gdal_config_options
set_gdal_config_options({"CPL_LOG": "/dev/null"})
All values were already correctly written for me, the warning was unexpected which is why I decided to suppress it. Of course this comes with the downside that if a valid warning were to be emitted I would miss it. |
@codeananda which version of pyogrio are you using? Because my understanding is that the warnings issue should also be solved by #242 (and I know in our CI we use |
We had this specific warning in the geopandas tests as well, and added a filter for this (so filtering specifically this warning with python I don't really understand why GDAL raises this warning, but our conclusion was also that it is probably harmless (you could open an issue about it upstream in osgeo/gdal) |
Wow thanks for the super speedy responses! I'm using pyogrio 0.7.2. Good to know that the values are actually written correctly. @jorisvandenbossche maybe I'm missing something but it seems like #242 is related to suppressing warnings for tests... but are these warnings actually suppressed in the code? |
A small example:
In the above case, we detect that GDAL errors, get the error message and include that in the Python exception we raise. But, you can also see that the original GDAL error is still printed as well (the first line before the traceback,
ERROR 1: ...
).This is not that a big deal, but I think ideally we would only have the exception (for example, if you catch the exception to handle it in some way, you still get the printed message). And, when using fiona, this doesn't happen (
df.to_file("test_fiona_mixed.shp", driver="ESRI Shapefile")
is the equivalent of the above pyogrio write, and this only gives a python exception).I have been wondering why this is (we do use
CPLErrorReset
after getting the error type and message). Comparing to Fiona, the error handling itself is very similar (since copied from Fiona), but in Fiona there is in addition also a error handler registered (CPLPushErrorHandler
):https://github.com/Toblerity/Fiona/blob/9c8fe736d4381526905c6976c399a33d10e3ecbf/fiona/_env.pyx#L137-L155
https://github.com/Toblerity/Fiona/blob/9c8fe736d4381526905c6976c399a33d10e3ecbf/fiona/_env.pyx#L387-L395
The text was updated successfully, but these errors were encountered: