-
Notifications
You must be signed in to change notification settings - Fork 128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce resampling filter and nvectors. #763
Conversation
the code had been forcing a spherical earth model to match the existing grtcirc.cc calculations.
commit 40429dd Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Dec 6 16:37:53 2021 -0700 fix dotnet time conversion (#784) commit 54b0136 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Dec 6 07:13:44 2021 -0700 Kill CarteSurTable format. (#783) Apparently it hasn't ever been used. Our sample has coordinates in UTM, but we have been interpretting them as degrees/100000. "Le pic du midi" referred to in the sample is The Pic du Midi de Bigorre. commit 1ed5895 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Dec 5 15:30:05 2021 -0700 harden gbfgetcstr, gbfgetpstr. (#782) Hangs were detected with afl and the gdb reader and gbfgetcstr. One such hang reads a fuzzed 32 bit url count with FREAD_i32, and then proceeds to fetch that many strings with FREAD_CSTR_AS_QSTR. This results in fetches from beyond EOF that are undetected. gbfgetcstr_old is modified to throw a fatal error if reading is attempted beyond EOF. A similar failure was not detected with gbfgetpstr, but it is also modified to throw a fatal error if reading is attempted beyond EOF. commit 85502cb Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Dec 3 07:16:02 2021 -0700 hopefully the real fix for garmin_xt tool issues. (#781) Fix LGTM detected "Comparison between i of type uint8_t and Count of wider type int." This latest find by LGTM is one in a long series of issue detections by various tools. However, close analysis reveals the previous solutions to these deteced issues didn't find or fix the real root issues: commit 233f3c8, 10/22/2013, Pad internal buffer to appease -fsanitize=address. While the fix appeased the sanitizer, it didn't fix the root problem. The root problem was a bug in the original translation of format_garmin_xt_decrypt_trk_blk when converting to zero-based array indexing. commit dcf0dd8, 7/8/2015, A bunch of busy work to satisfy hyperactive warnings in newer GCC builds. Again, the fix didn't fix the root problem. The root problem was a bug in the original translation of format_garmin_xt_proc_strk. This bug resulted in a mismatch between our test output and that of the original project. In our output the timestamp of the last two points in a track was identical. This fix results in our output matching the reference file of the original project (with allowances for precision). commit 727df52 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 17:17:40 2021 -0700 retire cet (#777) * teach mkshort to split utf8 on grapheme boundaries. and retire some cet functions. * add the grapheme test. * wean gbfile from cet. * restore cet.cc, cet.h in prepare for deprecation. * retire cet.cc, cet.h * drop attempted support of \r line endings in gbfgetutf16str. * add test of surrogate pairs for gbfgetutf16str. * fix test commit 2371d11 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 13:11:33 2021 -0700 add lgtm configuration file (#780) commit 1682055 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 12:40:38 2021 -0700 move Qt6 CI builds to 6.2.2. (#779) commit d377ef6 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 06:26:08 2021 -0700 convert arcdist, polygon, and xcsv to use gpsbabel::TextStream instead of gbfgetstr (#778) * read style files with a textstream. * use gpsbabel::textstream instead of gbfile. commit f2da18d Merge: 41cdf06 2341445 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Tue Nov 30 15:46:26 2021 -0700 Merge pull request #776 from tsteven4/nits3 miscellaneous minor fixes III commit 41cdf06 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Nov 29 07:29:41 2021 -0700 move Qt 5.12 CI builds to 5.12.12 (#774) commit 2341445 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 28 07:44:28 2021 -0700 remove support for non standard entity replacements. This was disabled in d47e364 August 29, 2005. commit dd56f42 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sat Nov 27 23:37:59 2021 -0700 fix lgtm detected ill-defined for loop. commit 59a325d Merge: 56c86e2 0769479 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sat Nov 27 17:45:30 2021 -0700 Merge pull request #772 from tsteven4/lgtm miscellaneous minor fixes commit 0769479 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 26 14:42:42 2021 -0700 sequester temporary style files in TMPDIR commit 3f38cef Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Thu Nov 25 10:09:04 2021 -0700 prefer rte_waypt_ct() over waypoint_list.count(). this provides some insulation from the underlying WaypointList container. commit bdd7865 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Thu Nov 25 08:37:57 2021 -0800 sort SOURCES and HEADERS in build. this gives a reasonable initial project view in Xcode. this gives a resaonable order to compilation. commit dd735b6 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 11:11:15 2021 -0800 minor correction to resample doc. commit f7d3d60 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 10:30:58 2021 -0800 further minimize xcsv_get_char_from_constant_table commit f483fc3 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 09:51:14 2021 -0800 use list initialization to construct xcsv char map table. commit d62f479 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 08:52:22 2021 -0800 more nits. commit b2957b5 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Tue Nov 23 16:09:14 2021 -0800 limit variable scope, found by LGTM. Poor global variable name 'ops'. Prefer longer, descriptive names for globals (eg. kMyGlobalConstant, not foo). commit 56c86e2 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Tue Nov 23 09:40:30 2021 -0700 fix parsing of nmea hms fields. (#770) commit fa20976 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 21 14:04:55 2021 -0700 Introduce resampling filter and nvectors. (#763) * Introduce resampling filter and nvectors. * use ellipsoid model with nvectors. the code had been forcing a spherical earth model to match the existing grtcirc.cc calculations. commit 1a422a6 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 21 14:04:03 2021 -0700 improve string handling, fix 9 year old FIXME. (#769) commit e09f7fe Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 21 13:34:23 2021 -0700 support generation of xcode project files with qmake. (#760) commit ff76e25 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sat Nov 20 08:05:01 2021 -0700 update fit locations test with improved sample (#767) * update fit test with improved sample. * reference file mode correction. commit 67ad13d Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 19 06:48:53 2021 -0700 Pick up fit locations test case (#766) * Add support for Locations.fit files * patch by tsteven4 to address review comments * Add a test case for Locations.fit files from Garmin devices Co-authored-by: Fabrizio Gennari <fabrizio.ge@tiscali.it> commit 368ad39 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Thu Nov 18 16:40:49 2021 -0700 add test for fit Locations(Lctns) files. (#765) * add test for fit Locations(Lctns) files. * fix new bug with change in Qt6 behavior for fit. commit 871efe3 Author: fabzzap <fabrizio.ge@tiscali.it> Date: Thu Nov 18 22:07:59 2021 +0100 Add support for Locations.fit files (#762) * Add support for Locations.fit files * patch by tsteven4 to address review comments commit 2504d97 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 17 08:52:08 2021 -0700 Eliminiate redundant route_head data member (#764) eliminate redundant route_head data member rte_waypt_ct, replacing it with a member function rte_waypt_ct() that gets the count from the waypoint_list container. commit 0beebdd Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Nov 8 05:22:30 2021 -0700 kill xmap, xmap2006. (#758) * kill xmap, xmap2006. * dprecate obsolete readme. commit 6593293 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 5 15:13:05 2021 -0600 kill xmapwpt. (#757) note is was wounded previously, see #754. commit 7b6235c Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 5 13:01:00 2021 -0600 add fedora 35 to CI. (#756) commit c3bc9c8 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 5 11:17:45 2021 -0600 serial port names (#755) * try qserialportinfo again. * update docker images with qt serial port. * enhance windows device text to match device manager on windows.
…aggeo, mapsend, nm4, pcx, pocketfms, skyforce, stmsdf, stmwpp, tiger, vpl, vidaone, vitosmt, vitovtt, wfff, and yahoo. (#785) * Remove compegps, cst, g7to2in, gopal, gpsutil, jtr, maggeo, mapsend, nm4, pcx, pocketfms, skyforce, stmsdf, stmwpp, tiger, vpl, wfff, and yahoo. * Missing reference files from above. * Change classic-3 test to read gzipped NMEA instead of deprecated formats. * Drop gpsutil from tomtom test as it tested nothing unique. * Remove gpsman test. It's a unicsv test and this tests nothing of value. * Eliminate use of gpsutil in gpx test; it covered no unique paths. * Update other GPSBabel.pro :-( * Update Cmaeklists enough for successful Mac compile * Cleanup internal doc references to newly retired formats. * Scrub (and commit) the remaining doc changes for what we just deleted. * Squashed commit of the following: commit 40429dd Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Dec 6 16:37:53 2021 -0700 fix dotnet time conversion (#784) commit 54b0136 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Dec 6 07:13:44 2021 -0700 Kill CarteSurTable format. (#783) Apparently it hasn't ever been used. Our sample has coordinates in UTM, but we have been interpretting them as degrees/100000. "Le pic du midi" referred to in the sample is The Pic du Midi de Bigorre. commit 1ed5895 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Dec 5 15:30:05 2021 -0700 harden gbfgetcstr, gbfgetpstr. (#782) Hangs were detected with afl and the gdb reader and gbfgetcstr. One such hang reads a fuzzed 32 bit url count with FREAD_i32, and then proceeds to fetch that many strings with FREAD_CSTR_AS_QSTR. This results in fetches from beyond EOF that are undetected. gbfgetcstr_old is modified to throw a fatal error if reading is attempted beyond EOF. A similar failure was not detected with gbfgetpstr, but it is also modified to throw a fatal error if reading is attempted beyond EOF. commit 85502cb Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Dec 3 07:16:02 2021 -0700 hopefully the real fix for garmin_xt tool issues. (#781) Fix LGTM detected "Comparison between i of type uint8_t and Count of wider type int." This latest find by LGTM is one in a long series of issue detections by various tools. However, close analysis reveals the previous solutions to these deteced issues didn't find or fix the real root issues: commit 233f3c8, 10/22/2013, Pad internal buffer to appease -fsanitize=address. While the fix appeased the sanitizer, it didn't fix the root problem. The root problem was a bug in the original translation of format_garmin_xt_decrypt_trk_blk when converting to zero-based array indexing. commit dcf0dd8, 7/8/2015, A bunch of busy work to satisfy hyperactive warnings in newer GCC builds. Again, the fix didn't fix the root problem. The root problem was a bug in the original translation of format_garmin_xt_proc_strk. This bug resulted in a mismatch between our test output and that of the original project. In our output the timestamp of the last two points in a track was identical. This fix results in our output matching the reference file of the original project (with allowances for precision). commit 727df52 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 17:17:40 2021 -0700 retire cet (#777) * teach mkshort to split utf8 on grapheme boundaries. and retire some cet functions. * add the grapheme test. * wean gbfile from cet. * restore cet.cc, cet.h in prepare for deprecation. * retire cet.cc, cet.h * drop attempted support of \r line endings in gbfgetutf16str. * add test of surrogate pairs for gbfgetutf16str. * fix test commit 2371d11 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 13:11:33 2021 -0700 add lgtm configuration file (#780) commit 1682055 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 12:40:38 2021 -0700 move Qt6 CI builds to 6.2.2. (#779) commit d377ef6 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Dec 1 06:26:08 2021 -0700 convert arcdist, polygon, and xcsv to use gpsbabel::TextStream instead of gbfgetstr (#778) * read style files with a textstream. * use gpsbabel::textstream instead of gbfile. commit f2da18d Merge: 41cdf06 2341445 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Tue Nov 30 15:46:26 2021 -0700 Merge pull request #776 from tsteven4/nits3 miscellaneous minor fixes III commit 41cdf06 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Nov 29 07:29:41 2021 -0700 move Qt 5.12 CI builds to 5.12.12 (#774) commit 2341445 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 28 07:44:28 2021 -0700 remove support for non standard entity replacements. This was disabled in d47e364 August 29, 2005. commit dd56f42 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sat Nov 27 23:37:59 2021 -0700 fix lgtm detected ill-defined for loop. commit 59a325d Merge: 56c86e2 0769479 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sat Nov 27 17:45:30 2021 -0700 Merge pull request #772 from tsteven4/lgtm miscellaneous minor fixes commit 0769479 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 26 14:42:42 2021 -0700 sequester temporary style files in TMPDIR commit 3f38cef Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Thu Nov 25 10:09:04 2021 -0700 prefer rte_waypt_ct() over waypoint_list.count(). this provides some insulation from the underlying WaypointList container. commit bdd7865 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Thu Nov 25 08:37:57 2021 -0800 sort SOURCES and HEADERS in build. this gives a reasonable initial project view in Xcode. this gives a resaonable order to compilation. commit dd735b6 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 11:11:15 2021 -0800 minor correction to resample doc. commit f7d3d60 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 10:30:58 2021 -0800 further minimize xcsv_get_char_from_constant_table commit f483fc3 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 09:51:14 2021 -0800 use list initialization to construct xcsv char map table. commit d62f479 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 24 08:52:22 2021 -0800 more nits. commit b2957b5 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Tue Nov 23 16:09:14 2021 -0800 limit variable scope, found by LGTM. Poor global variable name 'ops'. Prefer longer, descriptive names for globals (eg. kMyGlobalConstant, not foo). commit 56c86e2 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Tue Nov 23 09:40:30 2021 -0700 fix parsing of nmea hms fields. (#770) commit fa20976 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 21 14:04:55 2021 -0700 Introduce resampling filter and nvectors. (#763) * Introduce resampling filter and nvectors. * use ellipsoid model with nvectors. the code had been forcing a spherical earth model to match the existing grtcirc.cc calculations. commit 1a422a6 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 21 14:04:03 2021 -0700 improve string handling, fix 9 year old FIXME. (#769) commit e09f7fe Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sun Nov 21 13:34:23 2021 -0700 support generation of xcode project files with qmake. (#760) commit ff76e25 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Sat Nov 20 08:05:01 2021 -0700 update fit locations test with improved sample (#767) * update fit test with improved sample. * reference file mode correction. commit 67ad13d Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 19 06:48:53 2021 -0700 Pick up fit locations test case (#766) * Add support for Locations.fit files * patch by tsteven4 to address review comments * Add a test case for Locations.fit files from Garmin devices Co-authored-by: Fabrizio Gennari <fabrizio.ge@tiscali.it> commit 368ad39 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Thu Nov 18 16:40:49 2021 -0700 add test for fit Locations(Lctns) files. (#765) * add test for fit Locations(Lctns) files. * fix new bug with change in Qt6 behavior for fit. commit 871efe3 Author: fabzzap <fabrizio.ge@tiscali.it> Date: Thu Nov 18 22:07:59 2021 +0100 Add support for Locations.fit files (#762) * Add support for Locations.fit files * patch by tsteven4 to address review comments commit 2504d97 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Wed Nov 17 08:52:08 2021 -0700 Eliminiate redundant route_head data member (#764) eliminate redundant route_head data member rte_waypt_ct, replacing it with a member function rte_waypt_ct() that gets the count from the waypoint_list container. commit 0beebdd Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Mon Nov 8 05:22:30 2021 -0700 kill xmap, xmap2006. (#758) * kill xmap, xmap2006. * dprecate obsolete readme. commit 6593293 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 5 15:13:05 2021 -0600 kill xmapwpt. (#757) note is was wounded previously, see #754. commit 7b6235c Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 5 13:01:00 2021 -0600 add fedora 35 to CI. (#756) commit c3bc9c8 Author: tsteven4 <13596209+tsteven4@users.noreply.github.com> Date: Fri Nov 5 11:17:45 2021 -0600 serial port names (#755) * try qserialportinfo again. * update docker images with qt serial port. * enhance windows device text to match device manager on windows. * gbfile test can't depend on PCX any longer.\n Might be a hole in coverage. * Remove vitosmt and vitovtt. * Remove jogmap vidaone yahoo * Remove gpsman. * Remove more crumbs of g7towin. * Run automated cleanup on the list of files in commit message to clean up last pieces. * keep gpsutil.h in GPSBabel.pro Co-authored-by: Robert Lipe <robertlipe@gmail.com>
@tsteven4 i.e.: My GPX file has 200 points and I'd like to have 956 points in the end. I used a factor of 5 to be closer to 956 but it would be great if I could match exactly a specific number. |
you can change the sample rate by rational factor by using both interpolate and decimate, e.g. $ ./gpsbabel -t -i random,points=200 -f dummy -x validate,debug -x resample,interpolate=24,decimate=5,average=24 -x validate,debug Processing waypts Processing routes Processing tracks Processing waypts Processing routes Processing tracks |
Thanks, not very intuitive at first (how do you end up with 956 with those parameters?) I'll try that. |
agreed. I started with finding a rational factor that would change the sample rate by 956/200 = 239/50. But while this changed the sample rate as expected, it yields 952 points, so I tried 240/50, which gave me 956 points. I reduced 240/50 = 24/5. But we can find a general solution. If you start with n points and you want to end up with m points covering the same span of time then (n-1)*Ts = (m-1)*Ts' where Ts is the original sample rate and Ts' is the modified sample rate. So in your case n=200 and m = 956. Ts/Ts' = (m-1)/(n-1) = 955/199. 199 is prime so we can't reduce it further. Processing waypts Processing routes Processing tracks Processing waypts Processing routes Processing tracks |
Actually, my goal is to use timeless GPX files, would it be possible in this case to specify any |
Resample doesn't use timestamps at all. Normally this kind of signal
processing is applied to data that is sampled at a constant rate, but the
filter doesn't actual use any time information from the points. My last
answer, (m-1)/(n-1), will result in the number of points you asked for.
…On Thu, Dec 23, 2021, 8:15 PM FJBDev ***@***.***> wrote:
Actually, my goal is to use timeless GPX files, would it be possible in
this case to specify any m numbers of endpoints ? or do you need time
data for resampling ?
—
Reply to this email directly, view it on GitHub
<#763 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADHXMMKDNNOZBJJUUS6F2H3USPQUBANCNFSM5IEVUT3A>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
This implements classic digital signal processing resampling techniques. It can be used to change a track sample rate by a rational factor, e.g. twice, half, two thirds. It uses nvectors to average position, so it works across the antimeridian and at the poles. The track points are assumed to be sampled at a constant rate. The track points do not need to have timestamps. It can also be used to smooth a track without changing the sample rate.
It can be used to create similar output to the existing interpolate filter, but for points that are very far apart the nvector averaging is somewhat distorted from great circle calculations. Also, the propagation of unknown altitude values is worse than the existing interpolate filter.
The version of nvectors/vector3d included here incorporates the improvements suggested in #761 and a few more.
The ability to decimate data was desired some time ago by someone working with very large data sets. This is an enhanced version of code he tested. As I recall he was experiencing run times with the simplify filter in excess of a day. In such a case it may be desirable to either use the resample filter to decimate the data and then the simplify filter, or use the resample filter alone to decimate and smooth the data.