Skip to content
This repository has been archived by the owner on Nov 22, 2018. It is now read-only.

Commit

Permalink
Add a new back end file format.
Browse files Browse the repository at this point in the history 
To use the new format for newly created databases set the FileFormat field
in *Options passed to OpenFile to value 2 or use the driver named "ql2"
instead of "ql".

- Both the old and new driver will properly open and use, read and write the
  old (V1) or new file (V2) format of an existing database.

- V1 format has a record size limit of ~64 kB. V2 format record size limit
  is math.MaxInt32.

- V1 format uncommitted transaction size is limited by memory resources. V2
  format uncommitted transaction is limited by free disk space.

- A direct consequence of the previous is that small transactions perform
  better using V1 format and big transactions perform better using V2
  format.

benchmark                        old ns/op      new ns/op      delta
SelectOrderedFile1kBx1e4-4       1329006807     255517723      -80.77%
CrossJoinFile1e3NoX1e4-4         3107615747     619794528      -80.06%
CrossJoinFile1e4NoX1e3-4         3354802646     700357061      -79.12%
SelectOrderedFile1kBx1e3-4       123908159      26627421       -78.51%
CrossJoinFile1e2NoX1e3-4         27598600       6474143        -76.54%
CrossJoinFile1e3X1e4-4           335359531      78790606       -76.51%
SelectFile1kBx1e3-4              4562622        1092952        -76.05%
SelectBoolFileNoX1e3Perc5-4      1781983        517165         -70.98%
SelectBoolFileNoX1e3Perc50-4     1789031        519676         -70.95%
SelectBoolFileX1e4Perc5-4        1199502        359018         -70.07%
SelectFile1kBx1e4-4              40393899       12168884       -69.87%
SelectFile1kBx1e5-4              398938019      122302661      -69.34%
SelectBoolFileNoX1e4Perc50-4     17760045       5522018        -68.91%
SelectBoolFileX1e3Perc50-4       1050091        328726         -68.70%
SelectBoolFileNoX1e4Perc5-4      17723524       5808971        -67.22%
SelectBoolFileX1e4Perc50-4       10402097       3697792        -64.45%
InsertBoolFileX1e3-4             41825364       15918052       -61.94%
SelectOrderedFile1kBx1e2-4       9774335        4492106        -54.04%
CrossJoinFile1e3X1e2-4           1617783        785663         -51.44%
InsertBoolFileX1e1-4             4927433        9665456        +96.16%
InsertBoolFileNoX1e2-4           4638139        9045240        +95.02%
SelectBoolFileX1e3Perc5-4        61143          31704          -48.15%
InsertFile1kBn1e3t1e4-4          227403851      417962957      +83.80%
InsertBoolFileX1e2-4             5205319        9355281        +79.73%
SelectFile1kBx1e2-4              180977         101025         -44.18%
InsertFile1kBn1e0t1e2-4          493998605      882267286      +78.60%
CrossJoinFile1e2X1e3-4           1266888        711538         -43.84%
SelectBoolFileX1e2Perc5-4        9774           5497           -43.76%
CrossJoinFile1e3NoX1e2-4         11920217       6735563        -43.49%
CrossJoinFile1e2X1e1-4           32795          19315          -41.10%
InsertFile1kBn1e1t1e2-4          51884617       86750690       +67.20%
CrossJoinFile1e1X1e2-4           30959          18643          -39.78%
InsertBoolFileNoX1e1-4           4936511        8171014        +65.52%
CrossJoinFile1e4X1e3-4           138960718      88406050       -36.38%
InsertFile1kBn1e3t1e3-4          22921971       35507302       +54.91%
SelectBoolFileX1e1Perc5-4        5363           3477           -35.17%
SelectBoolFileX1e2Perc50-4       52708          34471          -34.60%
InsertFile1kBn1e1t1e3-4          513335347      775494042      +51.07%
InsertFile1kBn1e2t1e2-4          6725217        10065333       +49.67%
CrossJoinFile1e1NoX1e2-4         241349         162557         -32.65%
InsertFile1kBn1e2t1e4-4          834129466      1229457620     +47.39%
SelectBoolFileX1e1Perc50-4       10724          7279           -32.12%
InsertFile1kBn1e2t1e3-4          70257020       102477109      +45.86%
SelectBoolFileNoX1e1Perc5-4      9691           6647           -31.41%
CrossJoinFile1e2NoX1e1-4         178373         123161         -30.95%
SelectBoolFileNoX1e2Perc50-4     79529          55208          -30.58%
SelectBoolFileNoX1e1Perc50-4     9276           6542           -29.47%
InsertFile1kBn1e3t1e5-4          2581234930     3629377500     +40.61%
SelectBoolFileNoX1e2Perc5-4      82316          58945          -28.39%
InsertBoolFileNoX1e3-4           8478206        10126515       +19.44%

- V2 format uses substantially less memory.

benchmark                        old bytes     new bytes     delta
InsertFile1kBn1e3t1e3-4          4683629       214099        -95.43%
InsertFile1kBn1e3t1e4-4          42326873      2174786       -94.86%
InsertFile1kBn1e3t1e5-4          397886672     21182824      -94.68%
InsertFile1kBn1e2t1e3-4          4111234       221634        -94.61%
InsertFile1kBn1e2t1e4-4          40488844      2214520       -94.53%
InsertFile1kBn1e2t1e2-4          400895        22224         -94.46%
InsertFile1kBn1e1t1e3-4          5063280       411028        -91.88%
InsertFile1kBn1e1t1e2-4          498687        41186         -91.74%
InsertFile1kBn1e0t1e2-4          1571434       238228        -84.84%
SelectBoolFileX1e3Perc5-4        52729         10235         -80.59%
InsertBoolFileX1e1-4             33802         7251          -78.55%
SelectOrderedFile1kBx1e2-4       1486433       373721        -74.86%
SelectOrderedFile1kBx1e3-4       16538318      4318323       -73.89%
SelectBoolFileX1e2Perc5-4        7307          1944          -73.40%
SelectOrderedFile1kBx1e4-4       170936232     48409036      -71.68%
CrossJoinFile1e3X1e2-4           814207        255715        -68.59%
CrossJoinFile1e1X1e2-4           17059         5385          -68.43%
CrossJoinFile1e2X1e3-4           573394        191340        -66.63%
CrossJoinFile1e4X1e3-4           78447465      27922856      -64.41%
CrossJoinFile1e2X1e1-4           19173         6954          -63.73%
SelectFile1kBx1e3-4              3518328       1344859       -61.78%
SelectFile1kBx1e5-4              351685469     134437012     -61.77%
SelectFile1kBx1e4-4              35166002      13444622      -61.77%
SelectFile1kBx1e2-4              351753        134910        -61.65%
InsertBoolFileX1e3-4             2646597       1070045       -59.57%
InsertBoolFileX1e2-4             179300        86247         -51.90%
SelectBoolFileX1e3Perc50-4       220300        108229        -50.87%
InsertBoolFileNoX1e1-4           9348          4622          -50.56%
SelectBoolFileX1e2Perc50-4       22429         11300         -49.62%
SelectBoolFileX1e4Perc5-4        221089        113039        -48.87%
CrossJoinFile1e3X1e4-4           46730586      24064380      -48.50%
SelectBoolFileX1e1Perc5-4        2521          1360          -46.05%
SelectBoolFileX1e4Perc50-4       1929656       1055191       -45.32%
SelectBoolFileX1e1Perc50-4       4530          2600          -42.60%
SelectBoolFileNoX1e4Perc50-4     2403360       1521074       -36.71%
SelectBoolFileNoX1e4Perc5-4      2403349       1521074       -36.71%
SelectBoolFileNoX1e3Perc5-4      241282        153072        -36.56%
SelectBoolFileNoX1e3Perc50-4     241280        153072        -36.56%
CrossJoinFile1e2NoX1e3-4         2169344       1389074       -35.97%
CrossJoinFile1e3NoX1e4-4         241111128     154415488     -35.96%
CrossJoinFile1e4NoX1e3-4         278177376     178324552     -35.90%
CrossJoinFile1e3NoX1e2-4         2714091       1752226       -35.44%
SelectBoolFileNoX1e2Perc5-4      25076         16272         -35.11%
SelectBoolFileNoX1e2Perc50-4     25076         16272         -35.11%
CrossJoinFile1e1NoX1e2-4         57853         37680         -34.87%
CrossJoinFile1e2NoX1e1-4         52762         35472         -32.77%
SelectBoolFileNoX1e1Perc5-4      3472          2592          -25.35%
SelectBoolFileNoX1e1Perc50-4     3472          2592          -25.35%
InsertBoolFileNoX1e2-4           45394         39760         -12.41%
InsertBoolFileNoX1e3-4           423187        391102        -7.58%

	modified:   Makefile
	modified:   all_test.go
	modified:   doc.go
	modified:   driver.go
	modified:   driver/all_test.go
	modified:   driver/driver.go
	new file:   encode2.go
	modified:   etc.go
	modified:   file.go
	new file:   file2.go
	new file:   file2_test.go
	modified:   plan.go
	modified:   storage.go
	modified:   storage_test.go
	new file:   v2.sh
  • Loading branch information
Jan Mercl committed Nov 4, 2018
1 parent 8e18b65 commit 3a24a15
Show file tree
Hide file tree
Showing 15 changed files with 2,512 additions and 93 deletions.
8 changes: 6 additions & 2 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,16 @@ cover:
t=$(shell mktemp) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t

cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go test -run @ -bench BenchmarkInsertBoolFileNoX1e2 -cpuprofile cpu.out -benchmem -benchtime 4s
go tool pprof -lines *.test cpu.out

edit:
@ 1>/dev/null 2>/dev/null gvim -p Makefile *.l *.y *.go testdata.ql testdata.log

edit2:
touch log
@ 1>/dev/null 2>/dev/null gvim -p Makefile all_test.go log driver*.go encode2.go file*.go mem.go ql.go storage*.go testdata.ql testdata.log

editor: ql.y scanner.go parser.go coerce.go
gofmt -s -l -w *.go
go test -i
Expand All @@ -51,7 +55,7 @@ later:
@grep -n $(grep) MAYBE * || true

mem: clean
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
go test -run @ -bench BenchmarkInsertBoolFileNoX1e2 -memprofile mem.out -memprofilerate 1 -timeout 24h -benchmem -benchtime 4s
go tool pprof -lines -web -alloc_space *.test mem.out

nuke: clean
Expand Down
Loading

0 comments on commit 3a24a15

Please sign in to comment.