Convertion from bz1 to tsm doesn't work as described #5469

PierreF · 2016-01-28T09:51:05Z

When following instruction from https://github.com/influxdata/influxdb/blob/master/cmd/influx_tsm/README.md to convert database from 0.9.6.1 to 0.10.0-beta2 / bz1 to tsm, the new server still use bz1 storage for .bak file.

If the .bak file are removed it works, but by default .bak file are used.

Step to reproduce:

Grab both influxdb 0.9.6.1 & 0.10.0-beta2:

$ wget https://s3.amazonaws.com/influxdb/influxdb_0.9.6.1_x86_64.tar.gz
$ wget https://influxdb.s3.amazonaws.com/influxdb-v0.10.0-beta2_linux_amd64.tar.gz
$ tar xvfz influxdb_0.9.6.1_x86_64.tar.gz
$ tar xvfz influxdb-v0.10.0-beta2_linux_amd64.tar.gz

Start 0.9.6.1 with default on empty database:

$ rm -fr ~/.influxdb/*
$ ./influxdb_0.9.6.1_x86_64/usr/bin/influxd
2016/01/28 09:42:15 InfluxDB starting, version 0.9.6.1, branch 0.9.6, commit 6d3a8603cfdaf1a141779ed88b093dcc5c528e5e, built 2015-12-10T23:40:23+0000
2016/01/28 09:42:15 Go version go1.4.2, GOMAXPROCS set to 4
2016/01/28 09:42:15 no configuration provided, using default settings

Using CLI, create a database and insert data:

$ ./influxdb_0.9.6.1_x86_64/usr/bin/influx
Connected to http://localhost:8086 version 0.9.6.1
InfluxDB shell 0.9.6.1
> create database test
> use test
Using database test
> insert metric value=42
> exit

As said in https://github.com/influxdata/influxdb/blob/master/cmd/influx_tsm/README.md :

Stop influxdb
Restart influxdb 0.9
Wait until it respond to query
Stop server and run

$ ./usr/bin/influx_tsm -parallel ~/.influxdb/data/
b1 and bz1 shard conversion.                                                                                                                                                    [1/9025]
-----------------------------------
Data directory is:        /root/.influxdb/data/
Databases specified:      all
Database backups enabled: yes
Parallel mode enabled:    yes


2 shard(s) detected, 2 non-TSM shards detected.

Database        Retention       Path                                            Engine  Size
_internal       monitor         /root/.influxdb/data/_internal/monitor/1        bz1     131072
test            default         /root/.influxdb/data/test/default/2             bz1     32768

These shards will be converted. Proceed? y/N: y
Conversion starting....
Backing up 2 databases...
Database _internal backed up (341.687µs)
Database test backed up (136.496µs)
Conversion of /root/.influxdb/data/_internal/monitor/1 successful (4.079804ms)
Conversion of /root/.influxdb/data/test/default/2 successful (521.546µs)

Summary statistics
========================================
Databases converted:                 2
Shards converted:                    2
TSM files created:                   2
Points read:                         694
Points written:                      694
NaN filtered:                        0
Inf filtered:                        0
Points without fields filtered:      0
Disk usage pre-conversion (bytes):   163840
Disk usage post-conversion (bytes):  12506
Reduction factor:                    99%
Bytes per TSM point:                 18.02
Total conversion time:               5.209903ms

Start new verion of server

$ ./usr/bin/influxd 

2016/01/28 09:43:44 InfluxDB starting, version v0.10.0-beta2, branch 0.10.0, commit 58e0eed9cb166e0b004e6b41a4d64c39d24b40f4, built 2016-01-23T02:41:00.429242
2016/01/28 09:43:44 Go version go1.4.3, GOMAXPROCS set to 4
2016/01/28 09:43:44 no configuration provided, using default settings
[meta] 2016/01/28 09:43:44 Starting meta service
[meta] 2016/01/28 09:43:44 Listening on HTTP: 127.0.0.1:8091
[metastore] 2016/01/28 09:43:44 Using data dir: /root/.influxdb/meta
[metastore] 2016/01/28 09:43:44 Node at localhost:8088 [Follower]
[metastore] 2016/01/28 09:43:46 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[meta] 2016/01/28 09:43:46 127.0.0.1 - - [28/Jan/2016:09:43:46 +0000] GET /?index=0 HTTP/1.1 200 209 - Go 1.1 package http a08b1129-c5a3-11e5-8001-000000000000 992.248µs
[meta] 2016/01/28 09:43:46 127.0.0.1 - - [28/Jan/2016:09:43:46 +0000] POST /execute HTTP/1.1 200 28 - Go 1.1 package http a08b4f4e-c5a3-11e5-8003-000000000000 3.41668ms
[meta] 2016/01/28 09:43:46 127.0.0.1 - - [28/Jan/2016:09:43:46 +0000] GET /?index=18 HTTP/1.1 200 217 - Go 1.1 package http a08b4337-c5a3-11e5-8002-000000000000 4.939432ms
[store] 2016/01/28 09:43:46 Using data dir: /root/.influxdb/data
[tsm1wal] 2016/01/28 09:43:46 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/01/28 09:43:46 tsm1 WAL writing to /root/.influxdb/wal/_internal/monitor/1
[filestore]2016/01/28 09:43:46 /root/.influxdb/data/_internal/monitor/1/000000001-000000001.tsm (#0) opened in 73.712µs
[wal] 2016/01/28 09:43:46 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/28 09:43:46 WAL writing to /root/.influxdb/wal/_internal.bak/monitor/1
[tsm1wal] 2016/01/28 09:43:46 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/01/28 09:43:46 tsm1 WAL writing to /root/.influxdb/wal/test/default/2
[filestore]2016/01/28 09:43:46 /root/.influxdb/data/test/default/2/000000001-000000001.tsm (#0) opened in 44.93µs
[wal] 2016/01/28 09:43:46 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/28 09:43:46 WAL writing to /root/.influxdb/wal/test.bak/default/2
[handoff] 2016/01/28 09:43:46 Starting hinted handoff service

Check for data & insert one new value:

$ ./usr/bin/influx
Connected to http://localhost:8086 version v0.10.0-beta2
InfluxDB shell v0.10.0-beta2
> use test
Using database test
> select * from metric;
name: metric
------------
time                    value
1453974167488955239     42
> insert metric value=43

Backup is not possible:

$ ./usr/bin/influxd backup -database test /tmp/backup1
2016/01/28 09:45:10 backing up db=test since 0001-01-01 00:00:00 +0000 UTC
2016/01/28 09:45:10 backing up metastore to /tmp/backup1/meta.00
2016/01/28 09:45:10 backing up db=test rp=default shard=2 to /tmp/backup1/test.default.00002.00 since 0001-01-01 00:00:00 +0000 UTC
2016/01/28 09:45:10 backup complete
$ ls -l /tmp/backup1
total 4
-rw-r--r-- 1 root root 289 Jan 28 09:45 meta.00
-rw-r--r-- 1 root root   0 Jan 28 09:45 test.default.00002.00


On server console:
[snapshot] 2016/01/28 09:45:10 backup not implemented on bz1 engine

See that it still use .bak file (.bak file are openned, .bak are modified after convertion/new server start):

$ lsof -p `pgrep influxd`|grep ~/.influxdb/
influxd 457 root  mem-W  REG     252,2   131072 461886 /root/.influxdb/data/_internal.bak/monitor/1
influxd 457 root  mem-W  REG     252,2    65536 461887 /root/.influxdb/data/test.bak/default/2
influxd 457 root  mem-W  REG     252,2    65536 461643 /root/.influxdb/meta/raft.db
influxd 457 root  mem    REG     252,2       95 461891 /root/.influxdb/data/test/default/2/000000001-000000001.tsm
influxd 457 root  mem    REG     252,2     7396 461888 /root/.influxdb/data/_internal/monitor/1/000000001-000000001.tsm
influxd 457 root    6uW  REG     252,2    65536 461643 /root/.influxdb/meta/raft.db
influxd 457 root   11r   REG     252,2     7396 461888 /root/.influxdb/data/_internal/monitor/1/000000001-000000001.tsm
influxd 457 root   12uW  REG     252,2   131072 461886 /root/.influxdb/data/_internal.bak/monitor/1
influxd 457 root   13u   REG     252,2     1898 461884 /root/.influxdb/wal/_internal.bak/monitor/1/000000.meta
influxd 457 root   14r   REG     252,2       95 461891 /root/.influxdb/data/test/default/2/000000001-000000001.tsm
influxd 457 root   15uW  REG     252,2    65536 461887 /root/.influxdb/data/test.bak/default/2
influxd 457 root   16u   REG     252,2       72 461893 /root/.influxdb/wal/test.bak/default/2/000000.meta

$ ls -l /root/.influxdb/data/test.bak/default/2 /root/.influxdb/data/test/default/2/000000001-000000001.tsm 
-rw-r--r-- 1 root root 65536 Jan 28 09:44 /root/.influxdb/data/test.bak/default/2
-rw-r--r-- 1 root root    95 Jan 28 09:43 /root/.influxdb/data/test/default/2/000000001-000000001.tsm

Stop server. remove .bak file and restart it:

$ kill `pgrep influxd`
$ rm -fr ~/.influxdb/data/*.bak
$ ./usr/bin/influxd

New inserted value is lost:

$ ./usr/bin/influx
Connected to http://localhost:8086 version v0.10.0-beta2
InfluxDB shell v0.10.0-beta2
> use test
Using database test
> select * from metric
name: metric
------------
time                    value
1453974167488955239     42

Backup now works:

$ ./usr/bin/influxd backup -database test /tmp/backup2
2016/01/28 09:48:33 backing up db=test since 0001-01-01 00:00:00 +0000 UTC
2016/01/28 09:48:33 backing up metastore to /tmp/backup2/meta.00
2016/01/28 09:48:33 backing up db=test rp=default shard=2 to /tmp/backup2/test.default.00002.00 since 0001-01-01 00:00:00 +0000 UTC
2016/01/28 09:48:33 backup complete

$ ls -l /tmp/backup2
total 8
-rw-r--r-- 1 root root  289 Jan 28 09:48 meta.00
-rw-r--r-- 1 root root 2048 Jan 28 09:48 test.default.00002.00

The text was updated successfully, but these errors were encountered:

joelegasse · 2016-01-28T18:54:38Z

@PierreF Thank you for reporting this with the amount of detail that you did. We've updated the conversion tool to backup the databases to a separate directory.

The problem was the backed-up directories were not being ignored because foo.bak could be a valid name for a database. The shard data then reported itself as belonging to the original database name. That's why your new points were getting written to the "backup" directory.

The tool will no longer allow databases to be backed up underneath the data directory, and the instructions/examples have been updated accordingly.

joelegasse self-assigned this Jan 28, 2016

joelegasse mentioned this issue Jan 28, 2016

Skip backup directories created by influx_tsm #5471

Closed

jwilder mentioned this issue Jan 28, 2016

Update influx_tsm to use a backup directory #5472

Merged

joelegasse changed the title ~~Convertion from bz1 to tsm don't work as described~~ Convertion from bz1 to tsm doesn't work as described Jan 28, 2016

joelegasse closed this as completed Jan 28, 2016

jwilder added this to the 0.10.0 milestone Feb 1, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convertion from bz1 to tsm doesn't work as described #5469

Convertion from bz1 to tsm doesn't work as described #5469

PierreF commented Jan 28, 2016

joelegasse commented Jan 28, 2016

Convertion from bz1 to tsm doesn't work as described #5469

Convertion from bz1 to tsm doesn't work as described #5469

Comments

PierreF commented Jan 28, 2016

joelegasse commented Jan 28, 2016