-
Notifications
You must be signed in to change notification settings - Fork 362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running info on vector memory inputs return UNIX timestamps instead of ISO datetimes #4241
Comments
Hm, yes since what is returned is a text record I agree there is no good reason it should not be formatted as you expect here. If -C was used then you would get a numerical record and it would have UNIX time of course, but here it is text. I will try to have a look as to why it is not doing the formatting. |
I would like to add another section to the DEVELOPER RESOURCES sidebar on debugging, for how to debug pyGMT via Xcode on macOS. Been a month since last time and cannot find all the step that @seisman helped with. Since I am a complete python rookie there is not much mental memory on even the most basics, so I need that recipe, which also had some issues with setting shared-dir and possibly library paths for all to work. Maybe @seisman could resurrect his instructions and I can turn it into a section in the developer RST? |
And @joa-quim has given similar instructions for setting up GMT.jl so I can debug cases he finds directly in Xcode as well? |
Issue is at #3778. I should probably start to learn C too at some point 😆 |
The steps to debug PyGMT via Xcode was discussed in #3829. Here is a summary:
|
Please, please join the Force. Debugging from any of the environments (Matlab, Julia, Python) should all the same procedure.
|
@weiji14 Perhaps we should merge GenericMappingTools/pygmt#619 first (mark the tests as |
Ofc, on Unix everything is more complicated 😈 |
http://www.nooooooooooooooo.com/
Yeah, I'm on Linux which doesn't seem to have Xcode 🤷
Yep, give me a few minutes to finish up that PR. |
Yes, it would be great if a snipped of Python code demonstrating the problem is submitted since that is not something I will know how to do. Same with Julia of course and @joa-quim usually posts a Julia command or mex command. We just want to lower the threshold for all of us to do these things. Even C-novices should be able to step in the the C library and at least detect where a crash happens - often that is enough of a clue for the C-team to figure things out. |
Good question for Linux. I guess that |
On Linux, I have done lots of debugging using ddd, and even on macOS before Apple switched to clang and it became a bit harder (and I have not paid too much attention). But I dont know if you can connect to the process in ddd the same way we do in Xcode or Visual Studio. |
ddd is just a graphical front-end to gdb which I assume can connect to other processes. |
Ok, GenericMappingTools/pygmt#619 has been merged, and I'm starting to have a go at debugging using gdbgui (installed via import pygmt
import pandas as pd
table = pd.DataFrame(
data={
"z": [10, 13, 12, 15, 14],
"time": pd.date_range(start="2020-01-01", periods=5),
}
)
output = pygmt.info(table=table, V="d")
print(output)
# <vector memory>: N = 5 <10/15> <1577836800/1578182400> |
Hi @seisman, since I have done these before and have a pygmt directory:
is there a simpler step now to just pull new pygmt and then proceed to the make install step? |
Just run the following commands to update your local pygmt:
you even don't need to install pygmt again. then open a Python console, and type: import pygmt
pygmt.show_versions() |
As for GMT/MEX there are some quirks we need to figure out and probably improve (this is for macOS - for Windows I think it is simpler because @joa-quim sticks the mex builds in with the main GMT - and these are also in the official installer). Below. xbuild is the top of my build directory with Xcode:
I can think of a few options for this mess:
Maybe gmtmex should be a supplement of GMT, @seisman and @joa-quim ? |
Must be a missing step:
|
Hmm, can you try |
Thanks that works. I forgot I have miniconda so presumably there is a difference in the install vs full conda. Yet, this used to run a month ago... |
Can now recreate your case and have it stop in the C in Xcode again. But pushing my bedtime... |
No hurry, can do it tomorrow 😉 |
OK, so quick to figure out why this happens: Some modules read an entire dataset into memory (e.g., gmtconvert) while others read record-by-record (gmtinfo). The dataset-readers are carefully scanning the first record and would figure out what data type is in each column (in the case you did not use -f). Here, there is no such scanning and there is no -f1T passed to gmt info to tell it to expect absolute time in col 1. If I add f="1T" then I get ': N = 5 <10/15> <2020-01-01T00:00:00/2020-01-05T00:00:00>\n' but also some memory WARNINGS from the lib since I compile with a MEM_DEBUG setting. That some modules requires -f and others can figure out is an inconsistency we should work on to fix. |
You mean the master version? Installing
Note, the |
I think we install julia via brew and port so more like |
@joa-quim I cannot do julia> push!(Libdl.DL_LOAD_PATH, "$GMT_LIBRARY_PATH") Can I use a shell variable in this command? It works with copy/paste of the actual path of course. |
Ah, you are talking about In Julia you can run shell commands using the syntax |
OK, so for Julia I can set things like Yes, I was coming at this from the C side, so we want to use GMT master of course. As or GMT.jl master or not, will need your input on that, but presumably you dont want bug reports for older GMT.jl? So OK to use GMT#master for this? |
PyGMT is trying its best to avoid system calls, but |
So not out of the woods with Julia. I made sure I rebuilt the xcode libgmt. I have the new Xcode 12.0, released a few days ago. I follow the recipe of attaching to julia etc. Running one of Joaquim's examples (using GMT, then a few calcs, then a call to coast) takes me to the stop point I have set in the API in GMT_Call_Module. But there is sits, and I cannot examine any variables nor can I click step to get into the function. So unable to debug. I have done this before with an earlier Xcode so the recipe should work, but maybe 12.0 has a glitch. Yet, the PyGMT debug seemed to work. I do not know what the issue might be though. Update: Works with PyGMT. |
Funny, since sometime now I notice that in VS when I try to access the call stack it hangs for a while (and processors are working hard doing no-shit), but then after that while things resume to work fine. This happens only if I try to access the stack, otherwise all works fine. |
Yes, so gets to break point but the little console window in Xcode that shows the stack variables just has a spinning thingy indicating it is working hard. I will let it sit but this is not just a few seconds - I don't think it will finish. So clicking step in wont work because it is "busy". Notice all the things listed under the Thread in the side-bar. I dont remember all that before. |
So I cannot get out of this spin cycle and thus cannot debug Julia anymore. Not cool. |
Will try Xcode 11.7. Only 7.5 Gb zip to download and expand, etc.. |
Xcode 11.7 works so will make a note in the debug notes. |
Good, but damn it. |
I think the initial issue here got buried in debugging discussion. Hi @weiji14, please confirm that this is still a problem.. |
Yes, still a problem on GMT master. Had to silence the error messages two days ago at GenericMappingTools/pygmt#668 as it was getting annoying. |
This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions. |
Fixes problem with pygmt.info not being able to handle datetime64 inputs. I.e. `ValueError: could not convert string to float: '2021-01-01T12:34:56'` However, users will still need to use`pygmt.info(..., f="0T")` until upstream issue at GenericMappingTools/gmt#4241 is resolved. Also added two extra unit tests using numpy datetime64 types.
Fixes problem with pygmt.info not being able to handle datetime64 inputs. I.e. `ValueError: could not convert string to float: '2021-01-01T12:34:56'` However, users will still need to use`pygmt.info(..., coltypes="0T")` until upstream issue at GenericMappingTools/gmt#4241 is resolved. Also added two extra unit tests using numpy datetime64 types. * Use common alias coltypes (f) in tests and make a note on the workaround
@weiji14 the output appears as you wanted it - is this because of workarounds implemented in PyGMT? Would you be able to provide an example that still demonstrates the bug for debugging purposes?
|
Hmm, you might be right! I see that the these two units tests at https://github.com/GenericMappingTools/pygmt/blob/v0.3.1/pygmt/tests/test_info.py#L67-L107 are passing on GMT master (i.e. 6.2.0) now at the GMT Latest tests (https://github.com/GenericMappingTools/pygmt/runs/2104362367?check_suite_focus=true#step:14:324). Not sure when this got fixed 😅 Just to double check, could you try this example out (passing in only one datetime column from a numpy array). import pandas as pd
import pygmt
table = pd.date_range(start="2020-01-01", periods=5).to_numpy()
output = pygmt.info(table=table, per_column=True, V="d")
print(output)
# ['2020-01-01T00:00:00' '2020-01-05T00:00:00'] |
Output looks good, I think:
|
Nice, I'll close this issue then and wait for GMT 6.2.0 to be released. Thanks everyone! |
Fixed by #4849 |
…appingTools#960) Fixes problem with pygmt.info not being able to handle datetime64 inputs. I.e. `ValueError: could not convert string to float: '2021-01-01T12:34:56'` However, users will still need to use`pygmt.info(..., coltypes="0T")` until upstream issue at GenericMappingTools/gmt#4241 is resolved. Also added two extra unit tests using numpy datetime64 types. * Use common alias coltypes (f) in tests and make a note on the workaround
Description of the problem
UNIX timestamps (e.g. 1577836800) are returned instead of ISO datetimes (e.g. 2020-01-01T00:00:00) when running
info
on virtual tables. Originally noticed at GenericMappingTools/pygmt#619 (comment), check also #3396 where datetime inputs were originally accepted intoGMT_Put_Vector
.Full script that generated the error
Sorry that I can't write C code, but given an input like so:
and then running
info
from PyGMT from the branch at GenericMappingTools/pygmt#619 (each column is passed into C API usingGMT_Put_Vector
)Full error message
Actual outcome
Expected outcome
System information
gmt --version
): 6.1.1The text was updated successfully, but these errors were encountered: