[0.9.3-rc2] Unable to rapidly drop/recreate measurement

[jacek213]

This is related to #2955.

Below you can find a complete reproduction of my steps for 0.9.2. I'm unable to drop measurement (ERR: database not open) and after recreating I get error while executing basic SELECT clause.

InfluxDB shell 0.9.2
> CREATE DATABASE drop_test
> USE drop_test
Using database drop_test
> INSERT response_times,server=honolulu value=44.44
> SELECT * FROM response_times
name: response_times
tags: server=honolulu
time                            value
----                            -----
2015-08-04T14:45:21.546067507Z  44.44

> DROP MEASUREMENT response_times
ERR: database not open
> SELECT * FROM response_times
ERR: measurement not found: "drop_test"."default".response_times
> SHOW SERIES
> SELECT * FROM response_times
ERR: measurement not found: "drop_test"."default".response_times
> INSERT response_times,server=honolulu value=44.44
> SELECT * from response_times
ERR: found FROM, expected identifier, string, number, bool at line 1, char 9

[beckettsean]

@jacekgrzybowski I cannot reproduce this error on my setup. I'm running InfluxDB 0.9.2 installed via Homebrew on OS X 10.10.4 with an unaltered generated config file.

> create database drop_test
> use drop_test
Using database drop_test
> INSERT response_times,server=honolulu value=44.44
> SELECT * FROM response_times
name: response_times
tags: server=honolulu
time                value
----                -----
2015-08-04T18:56:41.558963517Z  44.44

> DROP MEASUREMENT response_times
> SELECT * FROM response_times
ERR: measurement not found: "drop_test"."default".response_times
> SHOW SERIES
> SELECT * FROM response_times
ERR: measurement not found: "drop_test"."default".response_times
> INSERT response_times,server=honolulu value=44.44
> SELECT * from response_times
name: response_times
tags: server=honolulu
time                value
----                -----
2015-08-04T18:57:11.097944281Z  44.44

How did you install InfluxDB? Did you make any changes to the configuration file?

[jacek213]

@beckettsean after a fresh system startup this situation hadn't occured anymore. However, after I ran my units tests a couple of times (before each test I initialize series by inserting a non relevant value and after each one I run DROP MEASUREMENT) it happened again. So imo it happens when I initialize and drop series over and over in small time intervals (like in #2955). Also sometimes localhost:8086 stops responding and I need to restart influxdb service. Installed on Ubuntu 14.04 LTS from .deb package downloaded from influxdb.com.

[beckettsean]

So imo it happens when I initialize and drop series over and over in small time intervals (like in #2955).

How small are the time intervals? InfluxDB wasn't designed to handle frequent deletes, it's not part of the core use case. Perhaps you can fuzz the database or measurement name in each test run so they aren't identical?

[beckettsean]

@jacekgrzybowski can you provide a script to reproduce? We can't seem to get this to happen.

[erikogenvik]

I'm also getting this error on 0.9.2.
Time seems to have no effect; I'm still unable to drop measurements even after many days.

Restarting the database fixes it.

These lines will always reproduce it for me.

curl -G http://localhost:8086/query --data-urlencode "q=DROP DATABASE droptest"
curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE droptest"
curl -i -XPOST 'http://localhost:8086/write?db=droptest' --data-binary 'test_point,tag1=one value=1'
curl -G http://localhost:8086/query --data-urlencode "db=droptest"  --data-urlencode "q=DROP SERIES FROM test_point"

The output from the final command is:

{"results":[{"error":"database not open"}]}

[beckettsean]

reproduced in 0.9.2 and 0.9.3-rc2. The latter has a more detailed error message:

curl -G http://localhost:8086/query --data-urlencode "q=DROP DATABASE droptest"; echo
{"results":[{}]}
curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE droptest"; echo
{"results":[{}]}
curl -i -XPOST 'http://localhost:8086/write?db=droptest' --data-binary 'test_point,tag1=one value=1'; echo
HTTP/1.1 204 No Content
curl -G http://localhost:8086/query --data-urlencode "db=droptest"  --data-urlencode "q=DROP SERIES FROM test_point"; echo
{"results":[{"error":"open /var/opt/influxdb/wal/droptest/default/122/000000.meta: no such file or directory"}]}

The error was consistent through multiple cycles of drop and recreate, always referencing /var/opt/influxdb/wal/droptest/default/122/000000.meta

[beckettsean]

After restarting the daemon and using DROP MEASUREMENT I get a slightly different error message (/124 vs /122):

curl -G http://localhost:8086/query --data-urlencode "db=droptest"  --data-urlencode "q=DROP MEASUREMENT test_point"; echo
{"results":[{"error":"open /var/opt/influxdb/wal/droptest/default/124/000000.meta: no such file or directory"}]}

[beckettsean]

@erikogenvik I just realized that the examples we've been working on recently are using DROP SERIES not DROP MEASUREMENT. The DROP SERIES issue already exists: #3087.

Cannot repro the original DROP MEASUREMENT issue, closing.

[jacek213]

@beckettsean it happened in my unit test, but I was able to reproduce it in a way I posted in the opening post of this issue. However here's a sample of my ruby unit tests which use influxdb-ruby client lib.

require 'test_helper'

class ServiceStatusTest < ActiveSupport::TestCase


  setup do
    @service = services(:six)
    @influx = InfluxSource.connect
    initialize_series
  end

  teardown do
    cleanup
  end

  test "if no influx data for dc, status data should be blank" do
    assert_equal Hash.new, @service.status_data('fo.foobar.dc1')
  end

  test "if no influx data for dc, state should be unknown" do
    assert_equal :unknown, @service.state('fo.foobar.dc1')
  end


  test "status data should return data for latest point if no datacenter specified" do
    seed_influx
    expected_result = {"time"=>0, "value"=>0.033333333, "status"=>"OK", "code"=>200, "result"=>"success"}
    assert_equal expected_result, @service.status_data
  end

  test "status data should return latest data for a specified datacenter" do
    seed_influx
    expected_result = {"time"=>0, "value"=>0.022225555, "status"=>"No\\ Content", "code"=>204, "result"=>"success"}
    assert_equal expected_result, @service.status_data('us.new-york.dc3')
  end

 # ...

  private

  def seed_influx
    @influx.write_points([
      {
        series: "response_times",
        tags: {
          service_id: @service.id,
          datacenter_id: 'pl.warsaw.dc1'
        },
        values: {
          value: 0.011111111,
          status: "Service unavailable",
          code: 503,
          result: 'failure'
        },
        timestamp: 1416388000
      }, 
      # ... some more data
    ])
  end

  def initialize_series
    @influx.write_point('response_times', tags: {service_id: 0}, values: {value: 0})
  end

  def cleanup
    @influx.query "DROP MEASUREMENT response_times"
  end
end

What happens here is:

1. initialize_series is called before running each test - some non-relevant data is saved just to initialize the measurement.
2. some tests use seed_influx to save some data, some don't.
3. after each test cleanup is called which drops the measurement.

I was not the only one encoutering database not open error. It happened to my coworker as well, and I had to stub all the interaction with db.

[beckettsean]

Thanks for the clarification, @jacekgrzybowski. I'll reopen the issue, but InfluxDB is not optimized for deletes, and certainly not for rapidly deleting and then recreating a measurement. I do understand the use case for automated testing but it's not an actual real-world issue. This might be something we don't fix for a while.

[cloud-rocket]

+1
X-Influxdb-Version:0.9.4.2

[jsternberg]

We have had a lot of work since this issue was filed. I'm going to close this as the issue may not be relevant anymore. Please try again with a newer version of InfluxDB and either comment here or open a new issue if this is still happening. Thank you.