Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large csv #3010

Merged
merged 46 commits into from
Aug 25, 2015
Merged

Large csv #3010

merged 46 commits into from
Aug 25, 2015

Conversation

artemp
Copy link
Member

@artemp artemp commented Jul 31, 2015

This is work-in-progress to update CSV plugin to support parsing large input and utilise spatial index for efficient bounding box queries.

  • removed boost::tokenizer usage
  • boost::spirit based `line' grammar/parser (supporting various quoting options)
  • re-factoring

Next steps

  • implement spatial_index similar to geojson.input and topojson.input (@flippmoke - if you have bandwidth next week, would be great if you could push on this)
    • support both in-memory and on-disk modes

artemp added 30 commits June 4, 2015 17:21
@artemp
csv plugin : add file_length standalone helper, make file_length loca…
e0e46eb 
…l variable
@artemp
fix spelling
f208717
@artemp
Merge branch 'master' into large_csv
ce33e2d
@artemp
Merge branch 'master' into large_csv
82a8735
@artemp
Merge branch 'master' into large_csv
6bb95d2
@artemp
refactor - detect_separator free func
35ff68a
@artemp
add csv_grammar
908f03b
@artemp
add 'csv' dir
f4ec974
@artemp
fix typedef's
11e5827
@artemp
use csv_grammar to parse csv lines
989af2e
@artemp
Merge branch 'master' into large_csv
f1feedc
@artemp
format
cb832c0
@artemp
handle escape characters
e5f1379
@artemp
better not to rely on implicit conversion to bool but use pointer com…
793a2f9 
…parison with nullptr
@artemp
csv_grammar - handle various quotting options + disable csv_utils::fi…
40b963f 
…x_json_quoting(csv_line)
@artemp
remove dead code
1a95f17
@artemp
remove unused grammar + remove stderr
b228da7
@artemp
Merge branch 'master' into large_csv
d893718
@artemp
make 'closing' quote an optional
6923613
@artemp
add `autodetect_newline' as a separate function
9c7186e
@artemp
Merge branch 'master' into large_csv
fec6d11
@artemp
Merge branch 'master' into large_csv
ad43a98
@artemp
Merge branch 'master' into large_csv
73e0ac6
@artemp
Merge branch 'master' into large_csv
8e40711
@artemp
Merge branch 'master' into large_csv
7158c19
@artemp
Merge branch 'svg-parser-errors' into large_csv
f9bd21e
@artemp
update test data
df8c5ce
@artemp
Merge branch 'master' into large_csv
f7d1cf8
@artemp
update test data
28a7bdf
@artemp
update 'test/data' submodule to track 'large_csv' branch
2b25f02
artemp added 16 commits August 17, 2015 15:27
@artemp
work-in-progress
318a821
@artemp
Merge branch 'master' into large_csv
153c93d
@artemp
temp workaround boost 1.59 geometry
d4bc329
@artemp
formatting
710ec05
@artemp
CSV plug-in - refactor and bring some sanity, sigh ..
6c3d9bb
@artemp
remove debug stderr + update data
d7e2f63
@artemp
cleanup
93fcc0a
@artemp
remove unused includes
4943cb4
@artemp
CSV - implement spatial index access to features on disk + preserve s…
4babec8 
…upport for inline data (work-in-progress)
@artemp
CSV - remove unused params and member vars
5dead08
@artemp
keep on untangling spaghetti
2477d87 
* implement standalone ignore case equality to avoid copying
* fix various logic shortcommings
@artemp
simplify + factor out properties parsing logic
c9d1d51
@artemp
Merge branch 'master' into large_csv
622ab90
@artemp
add parse_line accepting iterator range and avoid string ctor
6c9257a
@artemp
CSV - optimise parsing by providing num_columns hint
8709fb6
@artemp
CSV - revive row_limit parameter + fix stderr
3753d50
artemp added a commit that referenced this pull request Aug 25, 2015
@artemp
Merge pull request #3010 from mapnik/large_csv
35c2636 
Large csv
@artemp artemp merged commit 35c2636 into master Aug 25, 2015
@artemp artemp added this to the Mapnik 3.0.4 milestone Aug 26, 2015
@springmeyer springmeyer mentioned this pull request Aug 26, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Mapnik 3.0.4
Development

Successfully merging this pull request may close these issues.
1 participant
@artemp