incorrect X-axis value plotted

[ghost]

In my case, the start time is set to 2011/02/06-12:00:00 and the To is "now".

However, in the result graph, the X-axis is plotted like 2:00, 3:00, 4:00 instead of 12:00, 13:00 and 14:00.

[tsuna]

What timezone are you in? If the server on which the TSD is running isn't currently set to use your local time, you need to tell the TSD which time zone to use in order to render timestamps. You do so by running this command once in the root directory of OpenTSDB:
echo 'export TZ=YOURTIMEZONE' >>src/tsdb.local
For instance, if you're in California, you need to run:
echo 'export TZ=PST8PDT' >>src/tsdb.local
The valid names for timezones can typically be found under /usr/share/zoneinfo

[ghost]

My timezone is GMT+8 (China，Shanhai), so I've done this:

$ echo 'export TZ=Asia/Shanghai' >> src/tsdb.local

And I've verified my timezone by using zdump command:
# zdump /etc/localtime
/etc/localtime Mon Feb 7 14:31:22 2011 CST

# zdump /usr/share/zoneinfo/Asia/Shanghai
/usr/share/zoneinfo/Asia/Shanghai  Mon Feb  7 14:31:15 2011 CST

But still the graph seems 10 hours earlier than my local time although I've restarted both opentsdb and hbase.

[tsuna]

If you look at the logs of the TSD, do the timestamps appear in your local timezone, or are they also incorrect? I suspect that they will be incorrect too, which means that the TSD didn't pick up the TZ change, in which case I'm going to ask you how are you starting the TSD?

If the timestamps in the log are correct, but the graphs are still off, then there's another problem somewhere else.

[ghost]

The timestamps in TSD are correct:
>TsdbQuery: TsdbQuery(start_time=1297051200, end_time=1297062331)

which starts from Mon Feb 07 2011 12:00:00 GMT+0800 (China Standard Time), end at Mon Feb 07 2011 15:05:31 GMT+0800 (China Standard Time).

Since only the graph is incorrect, is this related to Gnuplot？

[tsuna]

Sorry, I wasn't clear enough. I was asking you whether the time on each log line in the TSD's log was correct.
2011-02-06 20:47:38,847 INFO [main] TSDMain: Starting.
For instance. Do those timestamps look correct?

Note: the UNIX timestamps (such as start_time=1297051200) cannot possibly be incorrect, since they're UNIX timestamps they don't have a timezone, they're always based on UTC.

[ghost]

Okay, the TSD's log is correct.

2011-02-07 15:45:12,753 INFO [New I/O server worker #1-4] TsdbQuery

Thanks for note :)

[tsuna]

OK, this is really weird then. When you make a query, if you look in the cache directory of the TSD, you should see a bunch of .gnuplot and .dat files. Try to locate the most recently created .dat file, it should correspond to your query. In this file, you'll see lines that look like this:
1297033476 0.0
Normally, the timestamp in the .dat file should be correct, for your local timezone. Given what you see, I expect that the timestamps in the .dat files are in UTC, which is why you're seeing the wrong time on the graphs. But I'd like to make sure that this is indeed the problem.

Also I'd like to know what's the output of gnuplot --version for you. Personally I'm using 4.4 patchlevel 2, but I've also tested OpenTSDB with Gnuplot version 4.2 in the past.

[ghost]

Sorry, my gnuplot is a very old one: gnuplot 4.0 patchlevel 0 which is installed from Centos's base repository.

The time stamp in the .dat file is incorrect. For instance,

• Start: 2011/02/07-16:00:00 (GMT +8)
  head -1 c9793b75_0.dat
  1297093200 0 # 2011/02/07 23:40:00 (GMT+0800)
• End: 2011/02/07-16:17:06 (GMT +8)
  tail -1 c9793b75_0.dat
  1297095428 0 # 2011/02/08 00:17:08 (GMT+0800)

Now I'll try to upgrade gnuplot to 4.4 patchleve 2.

[ghost]

Problem solved. It's actually a gnuplot issue. On the website it reads:

You need to have Gnuplot (custom open-source license) installed in your PATH version 4.2 minimum, 4.4 recommended.

So It's my fault to neglect warnings, I just proved that 4.0 is incompatible with TSD, lol. Thanks for you time.

[tsuna]

I think the timestamp in the file is correct, because 1297093200 is Mon Feb 7 15:40:00 2011 in CST (your time zone).

Let me explain what's going on under the hood: timestamps are always in UTC, so whenever the TSD receives and stores a timestamp, it doesn't change it, because no adjustment is needed if your data source is providing a correct UNIX timestamp.

When you make a query, the TSD retrieves the data points and passes them on to Gnuplot. The problem is that you can't tell Gnuplot what timezone to render the timestamps in, it always renders them in UTC. So the TSD has to adjust the timestamps so that they're rendered correctly for your time zone. This happens in net.opentsdb.graph.Plot.

So, let's recap:

• Your local timezone is CST (UTC + 8).
• The log lines in the TSD's logs appear in CST.
• The .dat files correctly contain timestamps that have been adjusted to be in CST.
• The graph generated by Gnuplot shows the timestamps in ... what? In UTC?

[tsuna]

So, just to be clear, did you upgrade Gnuplot and the problem went away? Maybe Gnuplot changed the way they were handling timestamps?

Please confirm whether upgrading Gnuplot fixed the issue for you.

[ghost]

Please confirm whether upgrading Gnuplot fixed the issue for you.

Confirm. After upgrading from 4.0 patchlevel 0 to 4.4 patchlevel 2, the issue is fixed.

Your local timezone is CST (UTC + 8).

Correct.

The log lines in the TSD's logs appear in CST.

Correct.

The .dat files correctly contain timestamps that have been adjusted to be in CST.

No, 1297093200 is Mon, 07 Feb 2011 15:40:00 GMT, not CST. As you can verify it from http://www.epochconverter.com

The graph generated by Gnuplot shows the timestamps in ... what? In UTC?

Actually, the generated graph is very strange, I've uploaded an incorrect one here: http://op.sdo.com/download/gnuplot-graph.png.

If it is an timezone issue, then the wrong graph should be 8 hours earlier than the given input value. However, the different is 10 hours which is peculiar.

[tsuna]

OK, I know what this problem is. You're using a buggy version of Gnuplot. I've had this problem in the past. The version you have is clipping the first character off the label of the X axis.

I'm trying to find the reference to the bug report. Please upgrade your version of Gnuplot to one that doesn't have this bug.