-
Notifications
You must be signed in to change notification settings - Fork 0
/
local-search.xml
639 lines (308 loc) · 432 KB
/
local-search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>GitHub清空提交记录</title>
<link href="/2022/10/19/GitHub%E6%B8%85%E7%A9%BA%E6%8F%90%E4%BA%A4%E8%AE%B0%E5%BD%95/"/>
<url>/2022/10/19/GitHub%E6%B8%85%E7%A9%BA%E6%8F%90%E4%BA%A4%E8%AE%B0%E5%BD%95/</url>
<content type="html"><![CDATA[<p>各位在github上维护自己编写的代码工程仓库时，想必遇到过仓库内文件泄漏隐私信息、commits泄漏邮箱信息等敏感头疼问题，但是由于这些仓库收获了很多stars，不想删除后重建新仓库。为了解决这些问题，这里有两个办法：</p><p>方法一：强制回退法</p><ol><li><p>克隆仓库，但记住不可以加–depth=1这类选项；</p></li><li><p>使用 git log 命令查询要回滚的 commit_id；</p></li><li><p>查找最早一次提交到的commit_id；</p></li><li><p>备份原仓库目录下所有文件及文件夹(.git文件夹除外)；</p></li><li><p>强制还原，执行命令：git reset –hard commit_id，之后 HEAD 就会指向此次的提交记录；</p></li><li><p>删除仓库目录下所有文件和文件夹，去除你备份的文件中包含隐私信息的部分，并还原至原仓库目录下；</p></li><li><p>执行提交新commit前的常规操作：git add . 和 git commit -m “init:剔除敏感信息”； ## 自己更换备注</p></li><li><p>强制推送到github远程仓库：git push origin HEAD –force；</p></li><li><p>此时只剩下第一次提交和本次提交记录，基本达到清空目的：</p></li></ol><p>此方法速度比较快，但缺点很明显：第一次commit记录无法清除。</p><p> 方法二：分支替换法</p><ol><li><p>同方法一克隆仓库；</p></li><li><p>将当前分支指向一个空分支 latest_branch：git checkout –orphan latest_branch；</p></li><li><p>备份原仓库目录下所有文件及文件夹(.git文件夹除外)，之后清空该目录(.git文件夹除外)；</p></li><li><p>去除你备份的文件中包含隐私信息的部分，并还原至原仓库目录下，最后重新添加：git add -A；</p></li><li><p>提交更改：git commit -am “init:剔除敏感信息”； ## 自己更换备注</p></li><li><p>删除原分支：git branch -D master （新版的github默认分支是main了，请根据实际情况更改，下同）；</p></li><li><p>将现有临时分支切换到主分支：git branch -m master；</p></li><li><p>强制推送到远程：git push -f origin master –force</p></li></ol><p><a href="https://blog.csdn.net/u012783994/article/details/126339284">感谢作者</a></p>]]></content>
<categories>
<category>开源相关</category>
</categories>
<tags>
<tag>GitHub</tag>
</tags>
</entry>
<entry>
<title>数据迁移脚本案例</title>
<link href="/2022/09/30/%E6%95%B0%E6%8D%AE%E8%BF%81%E7%A7%BB%E8%84%9A%E6%9C%AC%E6%A1%88%E4%BE%8B/"/>
<url>/2022/09/30/%E6%95%B0%E6%8D%AE%E8%BF%81%E7%A7%BB%E8%84%9A%E6%9C%AC%E6%A1%88%E4%BE%8B/</url>
<content type="html"><![CDATA[<p>使用Python脚本，将MySQL数据迁移至TDengine</p><span id="more"></span><p>项目背景：</p><p>公司调研过程时序数据库，做技术储备。最开始调研TDengine2.4，随时间推进官方升级2.6，在2.6上做了数据迁移，一批手机上传上来的定位数据，使用DATAX，使用mysqlreader加淘思官方提供的tdenginewriter插件，做好字段对应关系，写入数据，按阿里DATAX效率，开3个管道速度也并不快，走内网平均迁移速递大概在5000条/秒；到最近，涛思升级TDengine为3.0版本，很多附属工具基本重构，支持了流式计算，增加了社区版的数据删除等功能，这次使用DATAX适配失败，可能是改的太多了，索性尝试先使用python脚本迁移下。</p><p>说干就干，官方提供了<a href="https://docs.taosdata.com/develop/insert-data/high-volume/">快速迁移方案</a>，但是数据源是虚拟出来的，并没有时间主键，而且拆分子表是按量拆分然后平均分配，索性直接放弃，但是里面的多线程写入还是要参考的。</p><p>由于是python初学者，类和一些库函数使用还不是很熟，简简单单一把梭，一条流水从头撸到尾。迁移也是尝试了很多方法，遇到了很多问题这里逐一介绍。</p><h2 id="TDengine连接建立"><a href="#TDengine连接建立" class="headerlink" title="TDengine连接建立"></a>TDengine连接建立</h2><p><a href="https://docs.taosdata.com/develop/connect/">官方提供文档</a>，两种连接方式，原生客户端，restful方式。</p><h3 id="taosc-本地客户端方式"><a href="#taosc-本地客户端方式" class="headerlink" title="taosc 本地客户端方式"></a>taosc 本地客户端方式</h3><h4 id="安装驱动"><a href="#安装驱动" class="headerlink" title="安装驱动"></a>安装驱动</h4><p>安装包见官网 <a href="https://www.taosdata.com/assets-download/3.0/TDengine-client-3.0.1.4-Linux-x64.tar.gz">Linux</a>|<a href="https://www.taosdata.com/assets-download/3.0/TDengine-client-3.0.1.1.4-Windows-x64.exe">Windows</a></p><p>win安装，linux解压，执行 <code>chmod+x install_client.sh && install_client.sh</code></p><h4 id="配置客户端"><a href="#配置客户端" class="headerlink" title="配置客户端"></a>配置客户端</h4><blockquote><p>前提是与服务端的6030端口，开启TCP和UDP的放行，<font color=red> 如果是集群服务端，所有FQDN都需要添加端口两个协议的放行</font></p></blockquote><ul><li>修改cfg配置文件(win: C:\TDengine\cfg\taos.cfg linux：/etc/taos/taos.cfg) 修改 firstEP 为TDengine服务端的FirstEP</li></ul><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs ruby"><span class="hljs-comment"># The end point of the first dnode in the cluster to be connected to when </span><br><span class="hljs-string">`taosd`</span> <span class="hljs-keyword">or</span> <span class="hljs-string">`taos`</span> is started<br>firstEp <span class="hljs-symbol">tdengine1:</span><span class="hljs-number">6030</span><br></code></pre></td></tr></table></figure><ul><li>修改host文件，添加FQDN解析映射</li></ul><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">vim</span> /etc/hosts<br><span class="hljs-attribute">192</span>.<span class="hljs-number">168</span>.<span class="hljs-number">1</span>.xxx tdengine1<br><span class="hljs-attribute">192</span>.<span class="hljs-number">168</span>.<span class="hljs-number">1</span>.xxx tdengine2<br><span class="hljs-attribute">192</span>.<span class="hljs-number">168</span>.<span class="hljs-number">1</span>.xxx tdengine3<br></code></pre></td></tr></table></figure><ul><li>验证连接</li></ul><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">ping</span> tdengine1<br><span class="hljs-attribute">ping</span> tdengine2<br><span class="hljs-attribute">ping</span> tdengine3<br><span class="hljs-attribute">telnet</span> tdengine1 <span class="hljs-number">6030</span><br><span class="hljs-attribute">telnet</span> tdengine2 <span class="hljs-number">6030</span><br><span class="hljs-attribute">telnet</span> tdengine3 <span class="hljs-number">6030</span><br></code></pre></td></tr></table></figure><ul><li>taosc验证</li></ul><figure class="highlight crystal"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs crystal"><span class="hljs-variable">$ </span>taos -h tdengine1 -u root -p<br><br>taos > show databases;<br><br></code></pre></td></tr></table></figure><h4 id="安装python库"><a href="#安装python库" class="headerlink" title="安装python库"></a>安装python库</h4><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip uninstall taos taospy<br>pip <span class="hljs-keyword">install</span> taospy<br></code></pre></td></tr></table></figure><p>此种安装方法最高版本2.4.x，使用<code>pip_search</code>查看最高2.5.1(2022.09.30)，还是使用git源直接安装靠谱：</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">pip install git+https:<span class="hljs-regexp">//gi</span>thub.com<span class="hljs-regexp">/taosdata/</span>taos-connector-python.git<br></code></pre></td></tr></table></figure><p>但是吧，github方式国内连接不畅，可以<a href="https://github.com/taosdata/taos-connector-python/releases/download/v2.6.4/taospy-2.6.4-py3-none-any.whl">下载包</a>后来本地安装。</p><blockquote><p>注意：要求python版本3.8以上，非官方文档上的3.6以上。</p></blockquote><h4 id="测试连接"><a href="#测试连接" class="headerlink" title="测试连接"></a><a href="https://github.com/taosdata/TDengine/blob/main/docs/examples/python/connect_example.py">测试连接</a></h4><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs routeros">import taos<br><br>def test_connection():<br> # all parameters are optional.<br> # <span class="hljs-keyword">if</span> database is specified,<br> # then it must exist.<br> conn = taos.connect(<span class="hljs-attribute">host</span>=<span class="hljs-string">"localhost"</span>,<br> <span class="hljs-attribute">port</span>=6030,<br> <span class="hljs-attribute">user</span>=<span class="hljs-string">"root"</span>,<br> <span class="hljs-attribute">password</span>=<span class="hljs-string">"taosdata"</span>,<br> <span class="hljs-attribute">database</span>=<span class="hljs-string">"log"</span>)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'client info:'</span>, conn.client_info)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'server info:'</span>, conn.server_info)<br> conn.close()<br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br> test_connection()<br></code></pre></td></tr></table></figure><h3 id="restful-方式"><a href="#restful-方式" class="headerlink" title="restful 方式"></a>restful 方式</h3><p>万能连接大法，能发送GET、POST请求就行。</p><blockquote><p>与原生连接器的一个区别是，RESTful 接口是无状态的，因此 USE db_name 指令没有效果，所有<br>对表名、超级表名的引用都需要指定数据库名前缀。支持在 RESTful URL 中指定 db_name，这时<br>如果 SQL 语句中没有指定数据库名前缀的话，会使用 URL 中指定的这个 db_name。</p></blockquote><ul><li>确认服务端开启 taosadapter</li></ul><p>firstEP服务端运行</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">systemctl status taosadapter</span><br></code></pre></td></tr></table></figure><p>确保开放6041</p><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-attribute">lsof</span> -nP -iTCP -sTCP:LISTEN <br></code></pre></td></tr></table></figure><ul><li>测试连接</li></ul><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">curl</span> -u root:password http://<span class="hljs-number">192</span>.<span class="hljs-number">168</span>.<span class="hljs-number">1</span>.xxx:<span class="hljs-number">6041</span>/rest/sql -d <span class="hljs-string">"select server_version()"</span><br></code></pre></td></tr></table></figure><blockquote><p>HTTP 响应码<br>response code 说明<br>200 正确返回和 C 接口错误返回<br>400 参数错误返回<br>401 鉴权失败<br>404 接口不存在<br>500 内部错误<br>503 系统资源不足</p></blockquote><ul><li><p>安装python包</p><p>同上一章</p></li><li><p><a href="https://github.com/taosdata/TDengine/blob/3.0/docs/examples/python/connect_rest_examples.py">测试连接</a></p></li></ul><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-keyword">from</span> taosrest import connect, TaosRestConnection, TaosRestCursor<br><br>conn: TaosRestConnection = connect(<span class="hljs-attribute">url</span>=<span class="hljs-string">"http://localhost:6041"</span>,<br> <span class="hljs-attribute">user</span>=<span class="hljs-string">"root"</span>,<br> <span class="hljs-attribute">password</span>=<span class="hljs-string">"taosdata"</span>,<br> <span class="hljs-attribute">timeout</span>=30)<br><br></code></pre></td></tr></table></figure><h2 id="源数据batch"><a href="#源数据batch" class="headerlink" title="源数据batch"></a>源数据batch</h2><h3 id="拆分、循环方法"><a href="#拆分、循环方法" class="headerlink" title="拆分、循环方法"></a>拆分、循环方法</h3><p>因源数据有上亿级别数据，不可能一个<code>SELECT</code>撸到低，首先源表必须加上自增主键，或者带索引的可以作为分片键的字段(可以现加)，然后取个最大值，用for循环分批次去读取，如果数据处理模块、函数是要分多线程来取数据，需要有个全局变量记录不同线程之间取完数据后的增量值。</p><h2 id="子表名称拼接"><a href="#子表名称拼接" class="headerlink" title="子表名称拼接"></a>子表名称拼接</h2><p>这里会考虑将源表代表设备、人等可分子表的关键键值，作为分组条件，取出各个唯一值，全量再去循环每个子表的固定batch，分批拼接<code>INSERT</code>语句后写入。这里结合上面的分批取源表数据，会有嵌套循环的场景，</p><ul><li>先从全量数据取出固定分组，然后循环每个分组内的Id做for循环，这里好处是可以保证子循环里Id可以连续，不会按自增来跑空；</li><li>先取全量Id的连续值，在子循环里再找分组，这里好处是分组的压力对源库压力小，不至于直接给上亿条数据直接<code>GROUP BY </code>；</li><li>还有一种是先取全量Id连续值，子循环里拆分组，这个分组之后，由于有设备、人等键的限制，Id是不连续的，那就再次来个for循环遍历分组内的结果，直接拼出来<code>INSERT</code>语句。</li></ul><p>这里各种循环，没得用函数，也没得分文件，最开始需求就是要迁移数据，有思路更清晰的应该能更好的给模块化下。</p><h2 id="TDengine语句拼接"><a href="#TDengine语句拼接" class="headerlink" title="TDengine语句拼接"></a>TDengine语句拼接</h2><p>TDengine对标签数据和子表内数据写入，和正常的SQL标准内的还稍微有些区别，和其他时序类的应该很相似，这里在<a href="https://docs.taosdata.com/taos-sql/insert/">官方文档</a>里有段</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">INSERT</span> INTO d21001 USING meters TAGS ('California.SanFrancisco', <span class="hljs-number">2</span>) VALUES ('<span class="hljs-number">2021</span>-<span class="hljs-number">07</span>-<span class="hljs-number">13</span> <span class="hljs-number">14</span>:<span class="hljs-number">06</span>:<span class="hljs-number">34</span>.<span class="hljs-number">630</span>', <span class="hljs-number">10</span>.<span class="hljs-number">2</span>, <span class="hljs-number">219</span>, <span class="hljs-number">0</span>.<span class="hljs-number">32</span>) ('<span class="hljs-number">2021</span>-<span class="hljs-number">07</span>-<span class="hljs-number">13</span> <span class="hljs-number">14</span>:<span class="hljs-number">06</span>:<span class="hljs-number">35</span>.<span class="hljs-number">779</span>', <span class="hljs-number">10</span>.<span class="hljs-number">15</span>, <span class="hljs-number">217</span>, <span class="hljs-number">0</span>.<span class="hljs-number">33</span>)<br> <span class="hljs-attribute">d21002</span> USING meters (groupId) TAGS (<span class="hljs-number">2</span>) VALUES ('<span class="hljs-number">2021</span>-<span class="hljs-number">07</span>-<span class="hljs-number">13</span> <span class="hljs-number">14</span>:<span class="hljs-number">06</span>:<span class="hljs-number">34</span>.<span class="hljs-number">255</span>', <span class="hljs-number">10</span>.<span class="hljs-number">15</span>, <span class="hljs-number">217</span>, <span class="hljs-number">0</span>.<span class="hljs-number">33</span>)<br> <span class="hljs-attribute">d21003</span> USING meters (groupId) TAGS (<span class="hljs-number">2</span>) (ts, current, phase) VALUES ('<span class="hljs-number">2021</span>-<span class="hljs-number">07</span>-<span class="hljs-number">13</span> <span class="hljs-number">14</span>:<span class="hljs-number">06</span>:<span class="hljs-number">34</span>.<span class="hljs-number">255</span>', <span class="hljs-number">10</span>.<span class="hljs-number">27</span>, <span class="hljs-number">0</span>.<span class="hljs-number">31</span>);<br></code></pre></td></tr></table></figure><p>需要拿出标签数据写在<code>INSERT</code>段里，然后<code>VALUES</code>值按行写在括号内。</p><h2 id="简单日志"><a href="#简单日志" class="headerlink" title="简单日志"></a>简单日志</h2><p>上面讲到有实际已经到了三个循环，可以用全局变量累计每个子循环的时间、返回行数的累计，计算出平均迁移速度，返回给前台。</p><h2 id="后台运行"><a href="#后台运行" class="headerlink" title="后台运行"></a>后台运行</h2><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs vim">nohup <span class="hljs-keyword">python3</span>.<span class="hljs-number">9</span> mysql2tdnative.<span class="hljs-keyword">py</span> &<br></code></pre></td></tr></table></figure><h2 id="实际脚本"><a href="#实际脚本" class="headerlink" title="实际脚本"></a>实际脚本</h2><p>放托管平台了</p><p><a href="https://github.com/redgreat/mysql2td">GitHub</a></p><p><a href="https://gitee.com/rubygreat/mysql2td">Gitee</a></p><p><a href="https://codeberg.org/wangcw/mysql2td_codeberg">Codeberg</a></p><p>本来想写一堆，写着写着忘词了~</p>]]></content>
<categories>
<category>ETL工具</category>
</categories>
<tags>
<tag>ETL</tag>
<tag>数据迁移</tag>
</tags>
</entry>
<entry>
<title>数据库压测工具HannerDB</title>
<link href="/2022/09/19/%E6%95%B0%E6%8D%AE%E5%BA%93%E5%8E%8B%E6%B5%8B%E5%B7%A5%E5%85%B7hammerDB/"/>
<url>/2022/09/19/%E6%95%B0%E6%8D%AE%E5%BA%93%E5%8E%8B%E6%B5%8B%E5%B7%A5%E5%85%B7hammerDB/</url>
<content type="html"><![CDATA[<p>又一款数据库压测工具</p><span id="more"></span><h2 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h2><p>上次因公司业务需要使用新数据库，为保持换上之后业务稳定运行，需要压测对比结果，找了两款并没很好的测出想要的统一对比指标，偶尔发现这款数据库压测工具，方便好用。</p><p><a href="https://www.hammerdb.com/">软件官网</a></p><p><a href="https://github.com/TPC-Council/HammerDB">GitHub</a></p><p><a href="https://www.hammerdb.com/docs/index.html">标准官方文档</a>，内容相当丰富</p><p>支持win和linux平台，支持<a href="https://github.com/webysther/hammerdb-docker">docker部署</a>，linux和docker最终还是需要界面展示(xwindow等)支持，所以这里干脆直接用win版测试，如果需要测试的数据库服务只有内网环境，又没有可以用来测试的win服务器，用后面两种比较合适。</p><p>支持压测数据库类型：MySQL，MariaDB，PostgreSQL，Redis，MSSQLServer，Oracle.Db2</p><p>这里只压测MySQL做为测试。</p><p>压测不同类型数据库，需要有统一的测试指标吧，运行同样的SQL，同等数据量，同等并发、用户量等，这里介绍下测试基准 TPC比较重要的两项：</p><h2 id="压测基准"><a href="#压测基准" class="headerlink" title="压测基准"></a>压测基准</h2><blockquote><p><strong>TPC-C:</strong></p><p>TPC Benchmark C于1992年7月获得批准，是一个在线事务处理（OLTP）基准。 与TPC-A等以前的OLTP基准测试相比，TPC-C更复杂，因为它具有多种事务类型，更复杂的数据库和整体执行结构。 TPC-C涉及五个不同类型和复> 杂性的并发事务的混合，这些事务可以在线执行或排队等待延迟执行。 该数据库由九种类型的表组成，具有广泛的记录和人口规模。 TPC-C以每分钟事务数（tpmC）衡量。 虽然基准描述了批发供应商的活动，但TPC-C并不限于任何特定业务领域的活动，而是代表必须管理，销售或分销产品或服务的任何行业。<br>百科介绍：<a href="https://baike.baidu.com/item/TPCC/10023747?fr=aladdin">https://baike.baidu.com/item/TPCC/10023747?fr=aladdin</a></p><p><strong>TPC-H:</strong></p><p>TPC Benchmark™H（TPC-H）是决策支持基准。 它由一套面向业务的即时查询（ad-hoc）和并发数据修改组成。 选择查询和填充数据库的数据具有广泛的行业相关性。 该基准测试说明了决策支持系统，该系统可检查大量数> 据，执行高度复杂的查询并为关键业务问题提供答案。 TPC-H报告的性能指标称为TPC-H每小时综合查询性能指标（QphH @ Size），它反映了系统处理查询能力的多个方面。 这些方面包括执行查询时所选择的数据库大小，单> 个流提交查询时的查询处理能力以及多个并发用户提交查询时的查询吞吐量。 TPC-H价格/性能指标</p><p>TPC-H是一款面向商品零售业的决策支持系统测试基准，它定义了8张表，22个查询，遵循SQL92；TPC-H基准的数据库模式遵循第三范式。新兴的数据仓库开始采用新的模型，如星型模型、雪花模型。TPC-H已经不能精准反映当> 今数据库系统的真实性能。为此，TPC组织推出了新一代的面向决策应用的TPC-DS 基准。</p></blockquote><p>TPC还有一系列的针对不同业务场景测试基准，<a href="https://www.tpc.org/tpcc/">TPC官网</a>可以看到定义：</p><p><img src="https://www.wongcw.cn/static/img/bcaaa31f8a517cded7e1635780f14750.clipboard-2022-09-19.png"></p><p>由于TPC定义的测试基准比较复杂，这里HammerDB的 <a href="https://www.hammerdb.com/docs/ch03s03.html">TPROC-C</a>、<a href="https://www.hammerdb.com/docs/ch11s01.html">TPROC-H</a> 基准都是TPC基准上为更方便测试、更低的成本实现的。</p><p><img src="https://www.wongcw.cn/static/img/9e2b3e4433997aa3885f0cc79d3ab602.clipboard-2022-09-19.png"></p><p>了解了上面的测试基准，才能更好的输入测试参数来达到真实模拟测试效果以及每个输入。</p><h2 id="开测"><a href="#开测" class="headerlink" title="开测"></a>开测</h2><p><strong><a href="https://www.hammerdb.com/docs/ch02s01.html">Building the Schema</a></strong></p><h3 id="建立数据库连接"><a href="#建立数据库连接" class="headerlink" title="建立数据库连接"></a>建立数据库连接</h3><p><img src="https://www.wongcw.cn/static/img/e3acd24e2e38a82e175fded70f2534d9.clipboard-2022-09-19.png"></p><p>这里的虚拟用户 VisualUser 数量，等实际压测结果会比设定的多一个。注意用户一定要有建库建表权限。</p><h3 id="建库、建表、建立测试数据"><a href="#建库、建表、建立测试数据" class="headerlink" title="建库、建表、建立测试数据"></a>建库、建表、建立测试数据</h3><p><img src="https://www.wongcw.cn/static/img/cc1cd4ffce8dbacb52bb812e9fb1e8e8.clipboard-2022-09-19.png"></p><p>点机开始之后，这里可以看到每个用户进程正在执行的事务</p><p><img src="https://www.wongcw.cn/static/img/37b83dc9f3e9f34f42c73023ef3a5655.clipboard-2022-09-19.png"></p><p>这里可以看到创建 Warehouse 的 tpm</p><p><img src="https://www.wongcw.cn/static/img/6683108eb0b2fbe40c8a66fd95c27c73.clipboard-2022-09-19.png"></p><p>创建成功</p><p><img src="https://www.wongcw.cn/static/img/81a0fd6c4a531b4b8d898aa746d5b1b3.clipboard-2022-09-19.png"></p><p>根据大佬给的脚本可以看到创建成功后的库数据</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">select</span> a.<span class="hljs-built_in">SCHEMA_NAME</span>, a.DEFAULT_CHARACTER_SET_NAME,a.DEFAULT_COLLATION_NAME,<br>sum(table_rows) <span class="hljs-keyword">as</span> <span class="hljs-string">'记录数'</span>,<br>sum(<span class="hljs-keyword">truncate</span>(data_length/<span class="hljs-number">1024</span>/<span class="hljs-number">1024</span>, <span class="hljs-number">2</span>)) <span class="hljs-keyword">as</span> <span class="hljs-string">'数据容量(MB)'</span>,<br>sum(<span class="hljs-keyword">truncate</span>(index_length/<span class="hljs-number">1024</span>/<span class="hljs-number">1024</span>, <span class="hljs-number">2</span>)) <span class="hljs-keyword">as</span> <span class="hljs-string">'索引容量(MB)'</span>,<br>sum(<span class="hljs-keyword">truncate</span>((data_length+index_length)/<span class="hljs-number">1024</span>/<span class="hljs-number">1024</span>, <span class="hljs-number">2</span>)) <span class="hljs-keyword">as</span> <span class="hljs-string">'总大小(MB)'</span>,<br>sum(<span class="hljs-keyword">truncate</span>(max_data_length/<span class="hljs-number">1024</span>/<span class="hljs-number">1024</span>, <span class="hljs-number">2</span>)) <span class="hljs-keyword">as</span> <span class="hljs-string">'最大值(MB)'</span>,<br>sum(<span class="hljs-keyword">truncate</span>(data_free/<span class="hljs-number">1024</span>/<span class="hljs-number">1024</span>, <span class="hljs-number">2</span>)) <span class="hljs-keyword">as</span> <span class="hljs-string">'空闲空间(MB)'</span><br><span class="hljs-keyword">from</span> INFORMATION_SCHEMA.SCHEMATA a<br><span class="hljs-keyword">left outer join</span> information_schema.<span class="hljs-keyword">tables</span> b<br><span class="hljs-keyword">on</span> a.<span class="hljs-built_in">SCHEMA_NAME</span>=b.TABLE_SCHEMA<br><span class="hljs-keyword">where</span> a.<span class="hljs-built_in">SCHEMA_NAME</span>=<span class="hljs-string">'tpcc'</span><br><span class="hljs-keyword">group</span> <span class="hljs-keyword">by</span> a.<span class="hljs-built_in">SCHEMA_NAME</span>, a.DEFAULT_CHARACTER_SET_NAME,a.DEFAULT_COLLATION_NAME<br><span class="hljs-keyword">order</span> <span class="hljs-keyword">by</span> sum(data_length) <span class="hljs-keyword">desc</span>, sum(index_length) <span class="hljs-keyword">desc</span>;<br></code></pre></td></tr></table></figure><h3 id="查看服务器主机性能"><a href="#查看服务器主机性能" class="headerlink" title="查看服务器主机性能"></a>查看服务器主机性能</h3><p>这里用了htop , 可以用其他工具</p><p><img src="https://www.wongcw.cn/static/img/9a12fbe882debd107f30930eb16092d6.clipboard-2022-09-19.png"></p><p><strong><a href="https://www.hammerdb.com/docs/ch02s02.html">Run a Test Workload</a></strong></p><h3 id="开启正儿八经压测"><a href="#开启正儿八经压测" class="headerlink" title="开启正儿八经压测"></a>开启正儿八经压测</h3><p><img src="https://www.wongcw.cn/static/img/a30451050547b1910b1b9097d7735b3a.clipboard-2022-09-19.png"></p><p>两个选项，分别对应直接跑完脚本，一个是跑固定时间结束，选完之后点击Load加载脚本，点击Run Virtual Users 图标开始测试，我的虚拟机性能不高，这个测试客户端已经卡的不行了，但服务器性资源占用倒不高。虚拟用户创建可以在Driver Script 下面的菜单新建。</p><p><img src="https://www.wongcw.cn/static/img/6100988b30e823d6dd0bdf3a5ee8b412.clipboard-2022-09-19.png"></p><p>官网最新win版压测MySQL驱动会有些问题，据说是Tcl的版本与MySQL驱动依赖包版本不对应，需要更换MySQL驱动dll, HammerDB-4.5\lib\mysqltcl3.052\libmysql.dll，报错界面忘记截图，具体下载位置自行百度。后来发现<a href="https://www.hammerdb.com/docs/ch01s10.html#d0e628">HammerDB官方文档</a>已经写的很清楚了。</p>]]></content>
<categories>
<category>数据库压测</category>
</categories>
<tags>
<tag>MySQL</tag>
<tag>数据库压测</tag>
</tags>
</entry>
<entry>
<title>骑行分享</title>
<link href="/2022/09/17/%E9%AA%91%E8%A1%8C%E5%88%86%E4%BA%AB-2022-09-17/"/>
<url>/2022/09/17/%E9%AA%91%E8%A1%8C%E5%88%86%E4%BA%AB-2022-09-17/</url>
<content type="html"><![CDATA[<p>分享骑行照片</p><span id="more"></span><p>为了通勤买了个山地车，最近家里疫情，一直没回家，看着离海边还不远，每到周末，就骑起来自行车到海边看日出，由于非常业余也没有专业护具，年龄大了骑行速度也很慢，青岛又没有连续的宽敞自行车道，经过两周末两次的探索，总结出一条机动车较少的通往海边的骑行路线。</p><p>今早在海边还被路人鄙视没有护具，确实不安全，其实我有个骑行包最开始通勤的时候带着头盔，无奈头大，买的头盔太小了，今天停了路人叨叨的觉得也确实有道理，等后面添置上买个大的，安全第一。</p><p>下面就来到分享阶段，设备捉急，只能说凑合看~，多年前的诺记808早已过时，现在带着的是被通勤骑行时摔坏的果X，屏幕都摔稀碎了，马上准备换个二手的阿为P40PRO，眼馋好久了一直也没降价，话不多说来正文吧：</p><h4 id="第一天去程"><a href="#第一天去程" class="headerlink" title="第一天去程"></a>第一天去程</h4><p><img src="https://www.wongcw.cn/static/img/68f8a443231a04ceba63d7d510288399.IMG_1109.jpg"></p><h4 id="第一天日出"><a href="#第一天日出" class="headerlink" title="第一天日出"></a>第一天日出</h4><p>其实起的比较早，天还很黑，就没出发，等出去的时候已经5:20左右了</p><p><img src="https://www.wongcw.cn/static/img/f981631d670897023e3e43f9cc930275.IMG_1110.JPG"></p><p><img src="https://www.wongcw.cn/static/img/b64432d1d49d889710a4c0b97bbfc882.IMG_1114.JPG"></p><p><img src="https://www.wongcw.cn/static/img/178faac3bfd52662d4d6c318478f31fa.IMG_1116.jpg"></p><h4 id="第一天返程"><a href="#第一天返程" class="headerlink" title="第一天返程"></a>第一天返程</h4><p><img src="https://www.wongcw.cn/static/img/44a299cca6e9d3dce3c95ee2abd9d130.IMG_1119.PNG"></p><p>第一次骑车出远门，回家直接躺平，还好没有啥后遗症。</p><h4 id="第二次去程"><a href="#第二次去程" class="headerlink" title="第二次去程"></a>第二次去程</h4><p><img src="https://www.wongcw.cn/static/img/d141744d446e816927fc9459be813ad0.%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20220918102900.jpg"></p><h4 id="第二次日出"><a href="#第二次日出" class="headerlink" title="第二次日出"></a>第二次日出</h4><p><img src="https://www.wongcw.cn/static/img/3060704b24a867d9dc27ea3ba70bbfa3.IMG_1177.JPG"></p><p><img src="https://www.wongcw.cn/static/img/6e51632015e0ac1aadc6124e55641101.IMG_1179.JPG"></p><p><img src="https://www.wongcw.cn/static/img/d58b2c75bf4da13286c06943a37d8fb0.IMG_1184.JPG"></p><h4 id="第二次返程"><a href="#第二次返程" class="headerlink" title="第二次返程"></a>第二次返程</h4><p><img src="https://www.wongcw.cn/static/img/5c40267e9151ba730127913d385e1012.%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20220918102853.jpg"></p><p><img src="https://www.wongcw.cn/static/img/7ee55c38d2712ffbf46b99b30e4f3e2e.IMG_1189.JPG"></p><p>下次要添加装备了。</p><p>在此也记录下2022年初秋本次疫情下的即墨，中秋节只放行人进城不让出，节后马上静默状态，到现在也回不去家，希望人类早日战胜小小的罪恶的病毒。</p>]]></content>
<categories>
<category>骑行分享</category>
</categories>
<tags>
<tag>骑行分享</tag>
<tag>日出</tag>
</tags>
</entry>
<entry>
<title>Docker学习笔记</title>
<link href="/2022/09/17/Docker%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/"/>
<url>/2022/09/17/Docker%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/</url>
<content type="html"><![CDATA[<span id="more"></span><p>最近又有时间研究自己的博客了，作为一个DBA，对容器部署基本不感兴趣的，但是有一些应用，或者试验环境，用Docker搭建真的很方便，很容易维护，这不，要恶补补容器知识了，这里只记录下常用的运维语句吧，后面再研究怎么优化配置，打包自己镜像。</p><p>官方介绍：<br>Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. The Docker client and daemon can run on the same system, or you can connect a Docker client to a remote Docker daemon. The Docker client and daemon communicate using a REST API, over UNIX sockets or a network interface. Another Docker client is Docker Compose, that lets you work with applications consisting of a set of containers.<br><img src="https://qiniublogimg.wongcw.cn/202209180955388.svg+xml"></p><h4 id="安装Docker"><a href="#安装Docker" class="headerlink" title="安装Docker"></a>安装Docker</h4><figure class="highlight livecodeserver"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs livecodeserver">curl -fsSL <span class="hljs-keyword">https</span>://<span class="hljs-built_in">get</span>.docker.com | bash -s docker <span class="hljs-comment">--mirror Aliyun</span><br></code></pre></td></tr></table></figure><h4 id="更换Docker存储卷位置"><a href="#更换Docker存储卷位置" class="headerlink" title="更换Docker存储卷位置"></a>更换Docker存储卷位置</h4><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs gradle">vim <span class="hljs-regexp">/etc/</span>docker/daemon.json<br><br>{<br><span class="hljs-string">"registry-mirrors"</span>: [<span class="hljs-string">"https://hub.docker.com/"</span>],<br><span class="hljs-string">"graph"</span>:<span class="hljs-string">"/app/docker"</span>,<br><span class="hljs-string">"live-restore"</span>: <span class="hljs-keyword">true</span><br>}<br></code></pre></td></tr></table></figure><h4 id="启动"><a href="#启动" class="headerlink" title="启动"></a>启动</h4><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs crmsh">systemctl <span class="hljs-literal">start</span> docker<br></code></pre></td></tr></table></figure><h4 id="查看docker内容器"><a href="#查看docker内容器" class="headerlink" title="查看docker内容器"></a>查看docker内容器</h4><figure class="highlight handlebars"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs handlebars"><span class="language-xml">docker ps --format "table </span><span class="hljs-template-variable">{{<span class="hljs-name">.ID</span>}}</span><span class="language-xml">\t</span><span class="hljs-template-variable">{{<span class="hljs-name">.Names</span>}}</span><span class="language-xml">\t</span><span class="hljs-template-variable">{{<span class="hljs-name">.Ports</span>}}</span><span class="language-xml">\t</span><span class="hljs-template-variable">{{<span class="hljs-name">.Status</span>}}</span><span class="language-xml">"</span><br></code></pre></td></tr></table></figure><h4 id="查询镜像"><a href="#查询镜像" class="headerlink" title="查询镜像"></a>查询镜像</h4><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">docker search mongo</span><br></code></pre></td></tr></table></figure><h4 id="查看镜像版本"><a href="#查看镜像版本" class="headerlink" title="查看镜像版本"></a>查看镜像版本</h4><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs awk">curl https:<span class="hljs-regexp">//</span>registry.hub.docker.com<span class="hljs-regexp">/v1/</span>repositories<span class="hljs-regexp">/${docker_img}/</span>tags | python3 -m json.tool | more<br>curl https:<span class="hljs-regexp">//</span>registry.hub.docker.com<span class="hljs-regexp">/v1/</span>repositories<span class="hljs-regexp">/elasticsearch/</span>tags| tr -d <span class="hljs-string">'[\[\]" ]'</span> | tr <span class="hljs-string">'}'</span> <span class="hljs-string">'\n'</span>| awk -F: -v image=<span class="hljs-string">'elasticsearch'</span> <span class="hljs-string">'{if(NR!=NF && $3 != ""){printf("%s:%s\n",image,$3)}}'</span><br></code></pre></td></tr></table></figure><h4 id="查看版本-基本信息"><a href="#查看版本-基本信息" class="headerlink" title="查看版本/基本信息"></a>查看版本/基本信息</h4><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-attribute">docker</span> -v<br>docker <span class="hljs-literal">info</span><br></code></pre></td></tr></table></figure><h4 id="拉取镜像"><a href="#拉取镜像" class="headerlink" title="拉取镜像"></a>拉取镜像</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> pull apache/incubator-doris:build-env-<span class="hljs-number">1</span>.<span class="hljs-number">2</span><br></code></pre></td></tr></table></figure><h4 id="查看已下载镜像"><a href="#查看已下载镜像" class="headerlink" title="查看已下载镜像"></a>查看已下载镜像</h4><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">docker images</span><br></code></pre></td></tr></table></figure><h4 id="删除镜像"><a href="#删除镜像" class="headerlink" title="删除镜像"></a>删除镜像</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> rmi apache/incubator-doris:build-env-<span class="hljs-number">1</span>.<span class="hljs-number">2</span><br></code></pre></td></tr></table></figure><h4 id="查看容器"><a href="#查看容器" class="headerlink" title="查看容器"></a>查看容器</h4><figure class="highlight mel"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs mel">docker <span class="hljs-keyword">container</span> <span class="hljs-keyword">ls</span> -all<br></code></pre></td></tr></table></figure><h4 id="查看容器运行状态，相当于top"><a href="#查看容器运行状态，相当于top" class="headerlink" title="查看容器运行状态，相当于top"></a>查看容器运行状态，相当于top</h4><figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs fsharp">docker stats<br>docker <span class="hljs-keyword">top</span> {<span class="hljs-built_in">id</span>}<br></code></pre></td></tr></table></figure><h4 id="开启容器"><a href="#开启容器" class="headerlink" title="开启容器"></a>开启容器</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> start doris-<span class="hljs-number">014</span><br></code></pre></td></tr></table></figure><h4 id="关闭容器"><a href="#关闭容器" class="headerlink" title="关闭容器"></a>关闭容器</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> stop doris-<span class="hljs-number">014</span><br></code></pre></td></tr></table></figure><h4 id="重启容器"><a href="#重启容器" class="headerlink" title="重启容器"></a>重启容器</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> restart doris-<span class="hljs-number">014</span><br></code></pre></td></tr></table></figure><h4 id="删除容器"><a href="#删除容器" class="headerlink" title="删除容器"></a>删除容器</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> rm doris-<span class="hljs-number">014</span><br></code></pre></td></tr></table></figure><h4 id="连接容器-正在运行"><a href="#连接容器-正在运行" class="headerlink" title="连接容器(正在运行)"></a>连接容器(正在运行)</h4><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> attach doris-<span class="hljs-number">014</span><br></code></pre></td></tr></table></figure><h4 id="连接容器-正在运行-1"><a href="#连接容器-正在运行-1" class="headerlink" title="连接容器(正在运行)"></a>连接容器(正在运行)</h4><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">docker exec -it 容器ID <span class="hljs-regexp">/bin/</span>bash<br></code></pre></td></tr></table></figure><h4 id="退出（默认直接关闭）"><a href="#退出（默认直接关闭）" class="headerlink" title="退出（默认直接关闭）"></a>退出（默认直接关闭）</h4><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk"><span class="hljs-keyword">exit</span><br></code></pre></td></tr></table></figure><h4 id="删除所有容器"><a href="#删除所有容器" class="headerlink" title="删除所有容器"></a>删除所有容器</h4><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">docker rm <span class="hljs-constructor">$(<span class="hljs-params">docker</span> <span class="hljs-params">ps</span> -<span class="hljs-params">a</span> -<span class="hljs-params">q</span>)</span><br></code></pre></td></tr></table></figure><h4 id="删除所有镜像"><a href="#删除所有镜像" class="headerlink" title="删除所有镜像"></a>删除所有镜像</h4><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">docker rmi <span class="hljs-constructor">$(<span class="hljs-params">docker</span> <span class="hljs-params">images</span> -<span class="hljs-params">q</span>)</span><br></code></pre></td></tr></table></figure><h4 id="清理docker磁盘"><a href="#清理docker磁盘" class="headerlink" title="清理docker磁盘"></a>清理docker磁盘</h4><figure class="highlight livecodeserver"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs livecodeserver">docker <span class="hljs-keyword">system</span> prune -<span class="hljs-keyword">a</span><br></code></pre></td></tr></table></figure><h4 id="查看挂在目录"><a href="#查看挂在目录" class="headerlink" title="查看挂在目录"></a>查看挂在目录</h4><figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs dockerfile">docker <span class="hljs-keyword">volume</span><span class="language-bash"> <span class="hljs-built_in">ls</span></span><br>docker <span class="hljs-keyword">volume</span><span class="language-bash"> inspect</span><br></code></pre></td></tr></table></figure><h4 id="查看volumes目录大小"><a href="#查看volumes目录大小" class="headerlink" title="查看volumes目录大小"></a>查看volumes目录大小</h4><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">du -sh <span class="hljs-regexp">/var/</span>lib<span class="hljs-regexp">/docker/</span>volumes/<br></code></pre></td></tr></table></figure><h4 id="清理volumes目录下不同的volume"><a href="#清理volumes目录下不同的volume" class="headerlink" title="清理volumes目录下不同的volume"></a>清理volumes目录下不同的volume</h4><figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs dockerfile">docker <span class="hljs-keyword">volume</span><span class="language-bash"> <span class="hljs-built_in">ls</span> -f dangling=<span class="hljs-literal">true</span> | awk <span class="hljs-string">'{ print $2 }'</span> | xargs docker volume <span class="hljs-built_in">rm</span></span><br></code></pre></td></tr></table></figure><h4 id="查看清理后的volumes目录大小"><a href="#查看清理后的volumes目录大小" class="headerlink" title="查看清理后的volumes目录大小"></a>查看清理后的volumes目录大小</h4><p>/var/lib/docker/tmp清理<br>有一次制作镜像，一口气后台制作8个镜像，结果，tmp目录瞬间涨到12G。虚机硬盘分配的空间不大，导致了磁盘爆满，还好能直接删除tmp目录下所有的文件目录。</p><h4 id="容器清理"><a href="#容器清理" class="headerlink" title="容器清理"></a>容器清理</h4><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs routeros">sudo docker ps --filter <span class="hljs-attribute">status</span>=dead --filter <span class="hljs-attribute">status</span>=exited -aq | xargs -r sudo docker rm -v<br></code></pre></td></tr></table></figure><h4 id="镜像清理"><a href="#镜像清理" class="headerlink" title="镜像清理"></a>镜像清理</h4><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-attribute">sudo</span> docker images --<span class="hljs-literal">no</span>-trunc | grep <span class="hljs-string">'<none>'</span> | awk <span class="hljs-string">'{ print <span class="hljs-variable">$3</span> }'</span> | xargs -r sudo docker rmi<br></code></pre></td></tr></table></figure><h4 id="查看所有docker存储卷"><a href="#查看所有docker存储卷" class="headerlink" title="查看所有docker存储卷"></a>查看所有docker存储卷</h4><figure class="highlight livecodeserver"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs livecodeserver">sudo docker <span class="hljs-keyword">system</span> df<br>docker <span class="hljs-keyword">system</span> prune<br>docker <span class="hljs-keyword">system</span> prune -<span class="hljs-keyword">a</span><br></code></pre></td></tr></table></figure><h4 id="删除所有关闭的容器："><a href="#删除所有关闭的容器：" class="headerlink" title="删除所有关闭的容器："></a>删除所有关闭的容器：</h4><figure class="highlight powershell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs powershell">docker <span class="hljs-built_in">ps</span> <span class="hljs-literal">-a</span> | grep <span class="hljs-keyword">Exit</span> | cut <span class="hljs-literal">-d</span> <span class="hljs-string">' '</span> <span class="hljs-operator">-f</span> <span class="hljs-number">1</span> | xargs docker <span class="hljs-built_in">rm</span><br></code></pre></td></tr></table></figure><h4 id="删除所有dangling镜像（即无tag的镜像）："><a href="#删除所有dangling镜像（即无tag的镜像）：" class="headerlink" title="删除所有dangling镜像（即无tag的镜像）："></a>删除所有dangling镜像（即无tag的镜像）：</h4><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">docker rmi <span class="hljs-constructor">$(<span class="hljs-params">docker</span> <span class="hljs-params">images</span> | <span class="hljs-params">grep</span> <span class="hljs-string">"^<none>"</span> | <span class="hljs-params">awk</span> <span class="hljs-string">"{print $3}"</span>)</span><br></code></pre></td></tr></table></figure><h4 id="删除所有dangling数据卷（即无用的Volume）："><a href="#删除所有dangling数据卷（即无用的Volume）：" class="headerlink" title="删除所有dangling数据卷（即无用的Volume）："></a>删除所有dangling数据卷（即无用的Volume）：</h4><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">docker volume rm <span class="hljs-constructor">$(<span class="hljs-params">docker</span> <span class="hljs-params">volume</span> <span class="hljs-params">ls</span> -<span class="hljs-params">qf</span> <span class="hljs-params">dangling</span>=<span class="hljs-params">true</span>)</span><br></code></pre></td></tr></table></figure><h4 id="docker镜像自动更新"><a href="#docker镜像自动更新" class="headerlink" title="docker镜像自动更新"></a>docker镜像自动更新</h4><figure class="highlight livescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs livescript">docker pull containrrr/watchtower<br>docker run -d <span class="hljs-string">\</span><br> --name watchtower <span class="hljs-string">\</span><br> -v /<span class="hljs-keyword">var</span>/run/docker.sock:/<span class="hljs-keyword">var</span>/run/docker.sock <span class="hljs-string">\</span><br> containrrr/watchtower <span class="hljs-string">\</span><br> --cleanup <span class="hljs-string">\</span><br> --schedule <span class="hljs-string">"0 0 4 * * *"</span> <span class="hljs-string">\</span><br> --include-restarting<br></code></pre></td></tr></table></figure><h4 id="查看docker日志"><a href="#查看docker日志" class="headerlink" title="查看docker日志"></a>查看docker日志</h4><figure class="highlight powershell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs powershell">docker logs <span class="hljs-literal">--tail</span>=<span class="hljs-number">1000</span> 容器名称<br>docker<span class="hljs-literal">-compose</span> <span class="hljs-operator">-f</span> docker<span class="hljs-literal">-compose-app</span>.yml logs <span class="hljs-operator">-f</span><br>docker compose <span class="hljs-operator">-f</span> docker<span class="hljs-literal">-compose</span>.yml logs <span class="hljs-operator">-f</span><br></code></pre></td></tr></table></figure><h4 id="启动镜像"><a href="#启动镜像" class="headerlink" title="启动镜像"></a>启动镜像</h4><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">docker-compose up -d</span><br></code></pre></td></tr></table></figure><h4 id="关闭"><a href="#关闭" class="headerlink" title="关闭"></a>关闭</h4><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">docker-compose down</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>运维开发</category>
</categories>
<tags>
<tag>DevOps</tag>
<tag>运维</tag>
<tag>Docker</tag>
</tags>
</entry>
<entry>
<title>OracleCloud个人使用案例指导</title>
<link href="/2022/09/17/OracleCloud%E4%B8%AA%E4%BA%BA%E5%BA%94%E7%94%A8%E6%A1%88%E4%BE%8B/"/>
<url>/2022/09/17/OracleCloud%E4%B8%AA%E4%BA%BA%E5%BA%94%E7%94%A8%E6%A1%88%E4%BE%8B/</url>
<content type="html"><![CDATA[<p>注册OracleCloud后，个人建站前的一些准备</p><span id="more"></span><h2 id="注册"><a href="#注册" class="headerlink" title="注册"></a>注册</h2><h2 id="免费范围"><a href="#免费范围" class="headerlink" title="免费范围"></a>免费范围</h2><h2 id="云数据库"><a href="#云数据库" class="headerlink" title="云数据库"></a>云数据库</h2><h2 id="实例Instance"><a href="#实例Instance" class="headerlink" title="实例Instance"></a>实例Instance</h2><h3 id="ssh连接"><a href="#ssh连接" class="headerlink" title="ssh连接"></a>ssh连接</h3><p>#!/bin/bash<br>echo root:Mm@19890425 |sudo chpasswd root<br>sudo sed -i ‘s/PermitRootLogin no/PermitRootLogin yes/g’ /etc/ssh/sshd_config;<br>sudo sed -i ‘s/PasswordAuthentication no/PasswordAuthentication yes/g’ /etc/ssh/sshd_config;<br>sudo service sshd restart</p><p>echo root:Mm@20190601 |sudo chpasswd root<br>systemctl restart sshd</p><h3 id="新建用户"><a href="#新建用户" class="headerlink" title="新建用户"></a>新建用户</h3><p>groupadd wangcw<br>useradd trojan -m -g wangcw -s /bin/bash -d /home/trojan<br>passwd trojan<br>/sbin/usermod -aG sudo trojan<br>vim /etc/sudoers<br>trojan ALL=(ALL) ALL</p><p>useradd wangjin -m -g wangcw -s /bin/bash -d /home/wangjin<br>passwd wangjin</p><p>userdel -rZ trojan</p><h3 id="防火墙设置"><a href="#防火墙设置" class="headerlink" title="防火墙设置"></a>防火墙设置</h3><p>systemctl unmask firewalld #执行命令，即可实现取消服务的锁定<br>systemctl mask firewalld # 下次需要锁定该服务时执行<br>systemctl start firewalld.service #启动防火墙<br>systemctl stop firewalld.service #停止防火墙<br>systemctl reloadt firewalld.service #重载配置<br>systemctl restart firewalld.service #重启服务<br>systemctl status firewalld.service #显示服务的状态<br>systemctl enable firewalld.service #在开机时启用服务<br>systemctl disable firewalld.service #在开机时禁用服务<br>systemctl is-enabled firewalld.service #查看服务是否开机启动<br>systemctl list-unit-files|grep enabled #查看已启动的服务列表<br>systemctl –failed #查看启动失败的服务列表</p><p>firewall-cmd</p><p>firewall-cmd –state #查看防火墙状态<br>firewall-cmd –reload #更新防火墙规则<br>firewall-cmd –state #查看防火墙状态<br>firewall-cmd –reload #重载防火墙规则<br>firewall-cmd –list-ports #查看所有打开的端口<br>firewall-cmd –list-services #查看所有允许的服务<br>firewall-cmd –get-services #获取所有支持的服务</p><p>#区域相关<br>firewall-cmd –list-all-zones #查看所有区域信息<br>firewall-cmd –get-active-zones #查看活动区域信息<br>firewall-cmd –set-default-zone=public #设置public为默认区域<br>firewall-cmd –get-default-zone #查看默认区域信息<br>firewall-cmd –zone=public –add-interface=eth0 #将接口eth0加入区域public</p><p>#接口相关<br>firewall-cmd –zone=public –remove-interface=eth0 #从区域public中删除接口eth0<br>firewall-cmd –zone=default –change-interface=eth0 #修改接口eth0所属区域为default<br>firewall-cmd –get-zone-of-interface=eth0 #查看接口eth0所属区域</p><p>#端口控制<br>firewall-cmd –add-port=80/tcp –permanent #永久添加80端口例外(全局)<br>firewall-cmd –remove-port=80/tcp –permanent #永久删除80端口例外(全局)<br>firewall-cmd –add-port=65001-65010/tcp –permanent #永久增加65001-65010例外(全局)<br>firewall-cmd –zone=public –add-port=80/tcp –permanent #永久添加80端口例外(区域public)<br>firewall-cmd –zone=public –remove-port=80/tcp –permanent #永久删除80端口例外(区域public)<br>firewall-cmd –zone=public –add-port=65001-65010/tcp –permanent #永久增加65001-65010例外(区域public)<br>firewall-cmd –query-port=8080/tcp # 查询端口是否开放<br>firewall-cmd –permanent –add-port=80/tcp # 开放80端口<br>firewall-cmd –permanent –remove-port=8080/tcp # 移除端口<br>firewall-cmd –reload #重启防火墙(修改配置后要重启防火墙)</p><p>改用iptables.service</p><p>yum install iptables-services #安装iptables<br>systemctl stop firewalld.service #停止firewalld<br>systemctl mask firewalld.service #禁止自动和手动启动firewalld<br>systemctl start iptables.service #启动iptables<br>systemctl start ip6tables.service #启动ip6tables<br>systemctl enable iptables.service #设置iptables自启动<br>systemctl enable ip6tables.service #设置ip6tables自启动</p><p>firewall-cmd –add-port=80/tcp –permanent<br>firewall-cmd –add-port=443/tcp –permanent<br>firewall-cmd –add-port=8080/tcp –permanent<br>firewall-cmd –add-port=8024/tcp –permanent<br>firewall-cmd –add-port=27017/tcp –permanent</p><h2 id="存储"><a href="#存储" class="headerlink" title="存储"></a>存储</h2><h3 id="附赠数据盘挂载"><a href="#附赠数据盘挂载" class="headerlink" title="附赠数据盘挂载"></a>附赠数据盘挂载</h3><p>lsblk<br>fdisk /dev/sdb<br>n<br>p<br>#1<br>t<br>#1<br>8e<br>w</p><p>fdisk -l<br>pvcreate /dev/sdb1<br>pvdisplay /dev/sdb1 -v<br>pvs</p><p>vgcreate lvm_wangjin /dev/sdb1<br>vgdisplay lvm_wangjin -v<br>vgs<br>vgrename lvm_wangjin vg_wangjin<br>vgscan</p><p>lvcreate -L 49.5G -n lv_wangjin vg_wangjin<br>lvdisplay<br>lvs<br>lvscan</p><p>mkfs.ext4 /dev/vg_wangjin/lv_wangjin</p><p>mkdir /app<br>mount /dev/mapper/vg_wangjin-lv_wangjin /app</p><p>vim /etc/fstab<br>/dev/mapper/vg_wangjin-lv_wangjin /app ext4 defaults 0 0</p><p>umount /app<br>lvremove<br>vgremove<br>pvremove </p><h3 id="附赠免费对象存储-object-storage-挂载"><a href="#附赠免费对象存储-object-storage-挂载" class="headerlink" title="附赠免费对象存储(object storage)挂载"></a>附赠免费对象存储(object storage)挂载</h3><p>使用这个小工具<br> <a href="https://github.com/s3fs-fuse/s3fs-fuse">https://github.com/s3fs-fuse/s3fs-fuse</a><br>wget <a href="https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/s/s3fs-fuse-1.91-1.el8.x86_64.rpm">https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/s/s3fs-fuse-1.91-1.el8.x86_64.rpm</a></p><p>sudo dnf install fuse<br>sudo dnf install compat-openssl11<br>rpm -ivh s3fs-fuse-1.91-1.el8.x86_64.rpm</p><p>API鉴权密码<br>/etc/passwd-s3fs<br>接口地址<br><a href="https://docs.oracle.com/en-us/iaas/api/#/en/s3objectstorage/20160918/">https://docs.oracle.com/en-us/iaas/api/#/en/s3objectstorage/20160918/</a></p><p>挂载<br>s3fs filewong /objectstorage -o passwd_file=/etc/passwd-s3fs -o url=<a href="https://cnggzacz1yx9.compat.objectstorage.ap-seoul-1.oraclecloud.com/">https://cnggzacz1yx9.compat.objectstorage.ap-seoul-1.oraclecloud.com/</a> -o use_path_request_style</p><p>测试<br>touch test.txt 界面查看</p><p>开机挂载<br>vim /etc/fstab # 这个不好使<br>vim /etc/rc.local<br>s3fs filewong /objectstorage -o passwd_file=/etc/passwd-s3fs -o url=<a href="https://cnggzacz1yx9.compat.objectstorage.ap-seoul-1.oraclecloud.com/">https://cnggzacz1yx9.compat.objectstorage.ap-seoul-1.oraclecloud.com/</a> -o use_path_request_style</p><h2 id="负载均衡"><a href="#负载均衡" class="headerlink" title="负载均衡"></a>负载均衡</h2><h2 id="网络"><a href="#网络" class="headerlink" title="网络"></a>网络</h2><h3 id="白名单设置"><a href="#白名单设置" class="headerlink" title="白名单设置"></a>白名单设置</h3><h2 id="API-OCI-Cli"><a href="#API-OCI-Cli" class="headerlink" title="API-OCI Cli"></a>API-OCI Cli</h2><h2 id="Oracle-OCI重新验证"><a href="#Oracle-OCI重新验证" class="headerlink" title="Oracle OCI重新验证"></a>Oracle OCI重新验证</h2><p>Try out your newly registered credentials with the following example command:</p><p>oci iam region list –config-file C:\Users\王存伟.oci\config –profile DEFAULT</p><p>$T for a tenancy OCID<br>$C for a compartment OCID</p><p>T=ocid1.tenancy.oc1..xxx<br>C=ocid1.compartment.oc1..xxx</p><h2 id="OCI-获取namespace"><a href="#OCI-获取namespace" class="headerlink" title="OCI 获取namespace"></a>OCI 获取namespace</h2><p>oci os ns get</p><h2 id="查看租户信息"><a href="#查看租户信息" class="headerlink" title="查看租户信息"></a>查看租户信息</h2><p>oci iam compartment list -c ocid1.tenancy.oc1..</p><h2 id="获取网络"><a href="#获取网络" class="headerlink" title="获取网络"></a>获取网络</h2><p>oci network vcn list -c ocid1.tenancy.oc1..</p><h2 id="白名单"><a href="#白名单" class="headerlink" title="白名单"></a>白名单</h2><p>oci network security-list list -c ocid1.tenancy.oc1..</p><h2 id="创建防火墙-Gateway"><a href="#创建防火墙-Gateway" class="headerlink" title="创建防火墙 Gateway"></a>创建防火墙 Gateway</h2><p>oci network internet-gateway create -c <compartment_id> –is-enabled <true> –vcn-id <vcn_id> –display-name <gateway_display_name></p><h2 id="Route-Table"><a href="#Route-Table" class="headerlink" title="Route Table"></a>Route Table</h2><p>oci network route-table list -h, oci network route-table update -h</p><p>oci network route-table list -c <compartment_id> –vcn-id <vcn_id><br>oci network route-table update –rt-id <route_table_id> –route-rules “[{“cidrBlock”:”<0.0.0.0/0>”,”networkEntityId”:”<internet_gateway_id>“}]<br>WARNING: Updates to route-rules will replace any existing values. Are you sure you want to continue? [y/N]: y</p><h2 id="查看计算实例"><a href="#查看计算实例" class="headerlink" title="查看计算实例"></a>查看计算实例</h2><p>oci compute instance list -c ocid1.tenancy.oc1..</p><h2 id="列出-Bucuet"><a href="#列出-Bucuet" class="headerlink" title="列出 Bucuet"></a>列出 Bucuet</h2><p>oci os bucket list -ns picwonng –compartment-id $C<br>oci os bucket list -ns picwonng -c ocid1.compartment.oc1..</p><h2 id=""><a href="#" class="headerlink" title=""></a></h2><p>To list users and limit the output, run the following command.<br>oci iam user list –compartment-id $T –limit 5</p><p>To add a user to a group, run the following command.<br>oci iam group add-user –user-id ocid1.user.oc1..</p><p>oci compute image list -c ocid1.compartment.oc1.. –output table –query “data [*].{ImageName:"display-name", OCID:id}”</p>]]></content>
<categories>
<category>运维开发</category>
</categories>
<tags>
<tag>DevOps</tag>
<tag>运维</tag>
<tag>Docker</tag>
<tag>OracleCloud</tag>
</tags>
</entry>
<entry>
<title>rhel9安装部署mysql8</title>
<link href="/2022/08/10/rhel9%E5%AE%89%E8%A3%85mysql8/"/>
<url>/2022/08/10/rhel9%E5%AE%89%E8%A3%85mysql8/</url>
<content type="html"><![CDATA[<h3 id="安装前准备"><a href="#安装前准备" class="headerlink" title="安装前准备"></a>安装前准备</h3><h4 id="CRT连接"><a href="#CRT连接" class="headerlink" title="CRT连接"></a>CRT连接</h4><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-attribute">vim</span> /etc/ssh/sshd_config<br>LoginGraceTime <span class="hljs-number">0</span><br>PermitRootLogin <span class="hljs-literal">yes</span><br>StrictModes <span class="hljs-literal">yes</span><br>UseDNS <span class="hljs-literal">no</span><br>MaxSessions <span class="hljs-number">50</span><br><br><span class="hljs-comment">#重启 sshd 服务</span><br>systemctl restart sshd.service<br></code></pre></td></tr></table></figure><h4 id="主机名、时区"><a href="#主机名、时区" class="headerlink" title="主机名、时区"></a>主机名、时区</h4><figure class="highlight dsconfig"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs dsconfig"><span class="hljs-string">timedatectl</span> <span class="hljs-built_in">set-timezone</span> <span class="hljs-string">"Asia/Shanghai"</span> <br><span class="hljs-string">timedatectl</span> <span class="hljs-string">status</span>|<span class="hljs-string">grep</span> <span class="hljs-string">Local</span><br><span class="hljs-string">hostnamectl</span> <span class="hljs-built_in">set-hostname</span> <span class="hljs-string">rhel9</span><br><span class="hljs-string">vim</span> /<span class="hljs-string">etc</span>/<span class="hljs-string">hostname</span><br><span class="hljs-string">10</span>.<span class="hljs-string">16</span>.<span class="hljs-string">2</span>.<span class="hljs-string">58</span> <span class="hljs-string">rhel9</span><br></code></pre></td></tr></table></figure><h4 id="检查系统环境"><a href="#检查系统环境" class="headerlink" title="检查系统环境"></a>检查系统环境</h4><figure class="highlight perl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs perl"><span class="hljs-comment">#硬件</span><br>dmidecode |<span class="hljs-keyword">grep</span> Name<br><span class="hljs-comment">#内存</span><br>dmidecode|<span class="hljs-keyword">grep</span> -A5 <span class="hljs-string">"Memory Device"</span>|<span class="hljs-keyword">grep</span> Size|<span class="hljs-keyword">grep</span> -v No |<span class="hljs-keyword">grep</span> -v Range<br><span class="hljs-comment">#swap</span><br><span class="hljs-keyword">grep</span> SwapTotal /proc/meminfo | awk <span class="hljs-string">'{print $2}'</span><br><span class="hljs-comment">#内存大小</span><br>free -m<br><span class="hljs-comment">#磁盘空间</span><br>df -hT<br></code></pre></td></tr></table></figure><h4 id="关闭防火墙-amp-SElinux"><a href="#关闭防火墙-amp-SElinux" class="headerlink" title="关闭防火墙&SElinux"></a>关闭防火墙&SElinux</h4><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs awk">systemctl stop firewalld.service <span class="hljs-comment">#停止firewall</span><br>systemctl disable firewalld.service <span class="hljs-comment">#禁止firewall开机启动</span><br>cp <span class="hljs-regexp">/etc/</span>selinux<span class="hljs-regexp">/config /</span>etc<span class="hljs-regexp">/selinux/</span>config_`date +<span class="hljs-string">"%Y%m%d_%H%M%S"</span>`&& sed -i <span class="hljs-string">'s/SELINUX\=enforcing/SELINUX\=disabled/g'</span> <span class="hljs-regexp">/etc/</span>selinux/config<br>cat <span class="hljs-regexp">/etc/</span>selinux/config<br><span class="hljs-comment">#不重启</span><br>setenforce <span class="hljs-number">0</span><br>getenforce<br>sestatus<br></code></pre></td></tr></table></figure><h4 id="关闭透明大页"><a href="#关闭透明大页" class="headerlink" title="关闭透明大页"></a>关闭透明大页</h4><p>关闭透明大页（即 <code>Transparent Huge Pages</code>，缩写为 THP）。数据库的内存访问模式往往是稀疏的而非连续的。当高阶内存碎片化比较严重时，分配 THP 页面会出现较高的延迟。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs awk">cat <span class="hljs-regexp">/sys/</span>kernel<span class="hljs-regexp">/mm/</span>transparent_hugepage/enabled<br><span class="hljs-comment"># 执行 grubby 命令查看默认内核版本。</span><br><span class="hljs-comment"># grubby --default-kernel</span><br><span class="hljs-regexp">/boot/</span>vmlinuz-<span class="hljs-number">5.14</span>.<span class="hljs-number">0</span>-<span class="hljs-number">70.22</span>.<span class="hljs-number">1</span>.el9_0.x86_64<br><span class="hljs-comment"># 执行 grubby --update-kernel 命令修改内核配置</span><br>grubby --args=<span class="hljs-string">"transparent_hugepage=never"</span> --update-kernel <span class="hljs-regexp">/boot/</span>vmlinuz-<span class="hljs-number">5.14</span>.<span class="hljs-number">0</span>-<span class="hljs-number">70.22</span>.<span class="hljs-number">1</span>.el9_0.x86_64<br>执行 grubby --info 命令查看修改后的默认内核配置。--info 后需要使用实际的默认内核版本。<br>grubby --info <span class="hljs-regexp">/boot/</span>vmlinuz-<span class="hljs-number">5.14</span>.<span class="hljs-number">0</span>-<span class="hljs-number">70.22</span>.<span class="hljs-number">1</span>.el9_0.x86_64<br>echo never > <span class="hljs-regexp">/sys/</span>kernel<span class="hljs-regexp">/mm/</span>transparent_hugepage/enabled<br>echo never > <span class="hljs-regexp">/sys/</span>kernel<span class="hljs-regexp">/mm/</span>transparent_hugepage/defrag<br><span class="hljs-comment">#立即生效</span><br>cat <span class="hljs-regexp">/sys/</span>kernel<span class="hljs-regexp">/mm/</span>transparent_hugepage/enabled<br>cat <span class="hljs-regexp">/proc/</span>cmdline<br></code></pre></td></tr></table></figure><h4 id="关闭swap"><a href="#关闭swap" class="headerlink" title="关闭swap"></a>关闭swap</h4><p>DB 运行需要有足够的内存,如果内存充足，不建议使用 swap 作为内存不足的缓冲，因为这会降低性能,建议永久关闭系统 swap。</p><figure class="highlight maxima"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs maxima">echo <span class="hljs-string">"vm.swappiness = 0"</span>>> /etc/sysctl.conf<br>swapoff -a && swapon -a<br>sysctl -p<br>vim /etc/fstab<br># 注释加载swap分区的那行记录<br>#UUID=4f863b5f-<span class="hljs-number">20b3</span>-4a99-a680-ddf84a3602a4 swap swap defaults <span class="hljs-number">0</span> <span class="hljs-number">0</span><br></code></pre></td></tr></table></figure><h4 id="时钟同步"><a href="#时钟同步" class="headerlink" title="时钟同步"></a>时钟同步</h4><figure class="highlight nsis"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs nsis"><span class="hljs-params">system</span>ctl status chronyd.service<br>chronyc tracking<br><br><span class="hljs-comment">#结果是这个就正常</span><br>Leap status : <span class="hljs-params">Normal</span><br></code></pre></td></tr></table></figure><p>如果要使 NTP 服务尽快开始同步，执行以下命令。可以将 pool.ntp.org 替换为你的 NTP 服务器：</p><figure class="highlight sas"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs sas">sudo systemctl <span class="hljs-keyword">stop</span> ntpd.service <span class="hljs-variable">&&</span> \<br>sudo ntpdate pool.ntp.org <span class="hljs-variable">&&</span> \<br>sudo systemctl start ntpd.service<br></code></pre></td></tr></table></figure><p>如果要在 CentOS 7 系统上手动安装 NTP 服务，可执行以下命令：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs routeros">sudo yum install<span class="hljs-built_in"> ntp </span>ntpdate && \<br>sudo systemctl start ntpd.service && \<br>sudo systemctl <span class="hljs-built_in">enable</span> ntpd.service<br></code></pre></td></tr></table></figure><h4 id="磁盘、I-O调度设置"><a href="#磁盘、I-O调度设置" class="headerlink" title="磁盘、I/O调度设置"></a>磁盘、I/O调度设置</h4><p>将存储介质的 I/O 调度器设置为 noop。对于高速 SSD 存储介质，内核的 I/O 调度操作会导致性能损失。将调度器设置为 noop 后，内核不做任何操作，直接将 I/O 请求下发给硬件，以获取更好的性能。同时，noop 调度器也有较好的普适性。</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs gradle"># cat <span class="hljs-regexp">/sys/</span>block<span class="hljs-regexp">/sda/</span>queue/scheduler<br>[mq-deadline] kyber bfq none<br></code></pre></td></tr></table></figure><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><code class="hljs gradle"><span class="hljs-number">1</span>、查看CentOS6 CentOS7下IO支持的调度算法<br>#CentOS <span class="hljs-number">6</span>.x<br>#dmesg | <span class="hljs-keyword">grep</span> -i scheduler<br>io scheduler noop registered<br>io scheduler anticipatory registered<br>io scheduler deadline registered<br>io scheduler cfq registered (<span class="hljs-keyword">default</span>)<br>#CentOS <span class="hljs-number">7</span>.x<br>#dmesg | <span class="hljs-keyword">grep</span> -i scheduler<br>[ <span class="hljs-number">0.739263</span>] io scheduler noop registered<br>[ <span class="hljs-number">0.739267</span>] io scheduler deadline registered (<span class="hljs-keyword">default</span>)<br>[ <span class="hljs-number">0.739315</span>] io scheduler cfq registered<br>#rhel9<br>[ <span class="hljs-number">0.069059</span>] rcu: RCU calculated value of scheduler-enlistment delay is <span class="hljs-number">100</span> jiffies.<br>[ <span class="hljs-number">0.445431</span>] io scheduler mq-deadline registered<br>[ <span class="hljs-number">0.446503</span>] io scheduler kyber registered<br>[ <span class="hljs-number">0.447362</span>] io scheduler bfq registered<br><br><span class="hljs-number">2</span>、查看设备当前的 I/O 调度器<br>#cat <span class="hljs-regexp">/sys/</span>block<span class="hljs-comment">//queue/scheduler</span><br>假设磁盘名称是 <span class="hljs-regexp">/dev/</span>sda<br>#cat <span class="hljs-regexp">/sys/</span>block<span class="hljs-regexp">/sda/</span>queue/scheduler<br>noop [deadline] cfq<br><span class="hljs-number">3</span>、临时生效的方法<br>#cat <span class="hljs-regexp">/sys/</span>block<span class="hljs-regexp">/sda/</span>queue/scheduler<br>noop [deadline] cfq<br>#echo bfq><span class="hljs-regexp">/sys/</span>block<span class="hljs-regexp">/sda/</span>queue/scheduler<br>#cat <span class="hljs-regexp">/sys/</span>block<span class="hljs-regexp">/sda/</span>queue/scheduler<br>noop deadline [cfq]<br><span class="hljs-number">4</span>、永久生效的方法<br>CentOS <span class="hljs-number">7</span>.x<br>#grubby --update-kernel=ALL --args=<span class="hljs-string">"elevator=deadline"</span><br>#reboot<br>#cat <span class="hljs-regexp">/sys/</span>block<span class="hljs-regexp">/sda/</span>queue/scheduler<br>noop [deadline] cfq<br>或者使用vi编辑器修改配置文件，添加elevator= cfq<br>#vim <span class="hljs-regexp">/etc/</span><span class="hljs-keyword">default</span>/grub<br>GRUB_CMDLINE_LINUX=<span class="hljs-string">"crashkernel=auto rhgb quiet elevator=noop numa=off"</span><br>然后保存文件，重新编译配置文件<br>BIOS-Based： grub2-mkconfig -o <span class="hljs-regexp">/boot/g</span>rub2/grub.cfg<br>UEFI-Based： grub2-mkconfig -o <span class="hljs-regexp">/boot/</span>efi<span class="hljs-regexp">/EFI/</span>centos/grub.cfg<br></code></pre></td></tr></table></figure><h4 id="CPU-频率模式设置"><a href="#CPU-频率模式设置" class="headerlink" title="CPU 频率模式设置"></a>CPU 频率模式设置</h4><p>为调整 CPU 频率的 <code>cpufreq</code> 模块选用 <code>performance</code> 模式。将 CPU 频率固定在其支持的最高运行频率上，不进行动态调节，可获取最佳的性能。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs pgsql">cpupower frequency-<span class="hljs-keyword">info</span> <span class="hljs-comment">--policy</span><br>analyzing CPU <span class="hljs-number">0</span>:<br> Unable <span class="hljs-keyword">to</span> determine <span class="hljs-keyword">current</span> <span class="hljs-keyword">policy</span><br></code></pre></td></tr></table></figure><p><code>The governor "powersave"</code> 表示 cpufreq 的节能策略使用 <code>powersave</code>，需要调整为 <code>performance</code> 策略。如果是虚拟机或者云主机，则不需要调整，命令输出通常为 <code>Unable to determine current policy</code>。此为虚拟机不需修改，如物理机</p><p>创建 CPU 节能策略配置服务。</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs routeros">cat >> /etc/systemd/system/cpupower.service << EOF<br>[Unit]<br><span class="hljs-attribute">Description</span>=CPU performance<br>[Service]<br><span class="hljs-attribute">Type</span>=oneshot<br><span class="hljs-attribute">ExecStart</span>=/usr/bin/cpupower frequency-<span class="hljs-built_in">set</span> --governor performance<br>[Install]<br><span class="hljs-attribute">WantedBy</span>=multi-user.target<br>EOF<br></code></pre></td></tr></table></figure><p>应用 CPU 节能策略配置服务。</p><figure class="highlight nsis"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs nsis"><span class="hljs-params">system</span>ctl daemon-reload<br><span class="hljs-params">system</span>ctl enable cpupower.service<br><span class="hljs-params">system</span>ctl start cpupower.service<br></code></pre></td></tr></table></figure><h3 id="系统参数调整"><a href="#系统参数调整" class="headerlink" title="系统参数调整"></a>系统参数调整</h3><p><code>file-max</code> 中指定了系统范围内所有进程可打开的文件句柄的数量限制，在 MySQL 中很容易收到<code>”Too many open files in system”</code>这样的错误消息, 就应该增加这个值，这里由于测试环境设置为 100 万足够。<br><code>net.core.somaxconn</code> 是 Linux 中的一个 <code>kernel</code> 参数，表示 <code>socket</code> 监听 (listen) 的 <code>backlog</code> 上限。什么是 <code>backlog</code> 呢？<code>backlog</code> 就是 <code>socket</code> 的监听队列，当一个请求 (request) 尚未被处理或建立时，他会进入 <code>backlog</code>。而 <code>socket server</code> 可以一次性处理 <code>backlog</code> 中的所有请求，处理后的请求不再位于监听队列中。当 server 处理请求较慢，以至于监听队列被填满后，新来的请求会被拒绝。<br><code>net.ipv4.tcp_tw_recycle</code> 表示关闭启用 <code>TIME-WAIT</code> 状态 <code>sockets</code> 的快速回收，这个选项不推荐启用。在 <code>NAT(Network Address Translation)</code> 网络下，会导致大量的 TCP 连接建立错误。此值默认为 0，也是关闭状态，在 MySQL 的配置中有人也会开启此配置，这里在强调下。<br><code>net.ipv4.tcp_syncookies</code> 表示关闭 <code>SYN Cookies</code>。默认为 0，表示关闭；如果开启时，当出现 SYN 等待队列溢出时，启用cookies 来处理，可防范少量 SYN 攻击，和 tcp_tw_recycle 一样，在 NAT 网卡模式下不建议开启此参数。<br><code>vm.overcommit_memory</code> 文件指定了内核针对内存分配的策略，其值可以是 0、1、2。<br>0:(默认)表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。0 即是启发式的 <code>overcommitting handle</code>,会尽量减少 <code>swap</code> 的使用,root 可以分配比一般用户略多的内存。<br>1:表示内核允许分配所有的物理内存，而不管当前的内存状态如何，允许超过 <code>CommitLimit</code>，直至内存用完为止。<br>2:表示不允许超过 <code>CommitLimit</code> 值。<br><code>limit.conf</code> 用于限制用户可以使用的最大文件数、最大线程、最大内存使用量，<code>soft</code> 是一个告警值，<code>hard</code> 则是一定意义上的阈值，一旦超过 <code>hard</code> ，系统就会报错</p><figure class="highlight arcade"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs arcade">echo <span class="hljs-string">"fs.file-max = 1000000"</span>>> <span class="hljs-regexp">/etc/</span>sysctl.conf<br>echo <span class="hljs-string">"net.core.somaxconn = 32768"</span>>> <span class="hljs-regexp">/etc/</span>sysctl.conf<br>echo <span class="hljs-string">"net.ipv4.tcp_tw_recycle = 0"</span>>> <span class="hljs-regexp">/etc/</span>sysctl.conf<br>echo <span class="hljs-string">"net.ipv4.tcp_syncookies = 0"</span>>> <span class="hljs-regexp">/etc/</span>sysctl.conf<br>echo <span class="hljs-string">"vm.overcommit_memory = 1"</span>>> <span class="hljs-regexp">/etc/</span>sysctl.conf<br><br>sysctl -p<br></code></pre></td></tr></table></figure><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs pgsql">cat <span class="hljs-symbol"><< EOF >></span>/etc/<span class="hljs-keyword">security</span>/limits.conf<br>db soft nofile <span class="hljs-number">1000000</span><br>db hard nofile <span class="hljs-number">1000000</span><br>db soft stack <span class="hljs-number">32768</span><br>db hard stack <span class="hljs-number">32768</span><br>EOF<br></code></pre></td></tr></table></figure><h3 id="安装前卸载"><a href="#安装前卸载" class="headerlink" title="安装前卸载"></a>安装前卸载</h3><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs vim">rpm -<span class="hljs-keyword">qa</span> | <span class="hljs-keyword">grep</span> -i mariadb<br><span class="hljs-keyword">c</span><br><br>yum <span class="hljs-built_in">remove</span> mariadb-libs<br></code></pre></td></tr></table></figure><p>#删除用户/用户组</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs gradle">more <span class="hljs-regexp">/etc/</span>passwd | <span class="hljs-keyword">grep</span> mysql<br>more <span class="hljs-regexp">/etc/</span>shadow | <span class="hljs-keyword">grep</span> mysql<br>more <span class="hljs-regexp">/etc/g</span>roup | <span class="hljs-keyword">grep</span> mysql<br><br>groupdel mysql<br>userdel mysql<br></code></pre></td></tr></table></figure><h3 id="列出安装包明细"><a href="#列出安装包明细" class="headerlink" title="列出安装包明细"></a>列出安装包明细</h3><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">mysql</span>-commercial-backup-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 备份<br><span class="hljs-attribute">mysql</span>-commercial-backup-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-client-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 客户端<br><span class="hljs-attribute">mysql</span>-commercial-client-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-client-plugins-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 插件<br><span class="hljs-attribute">mysql</span>-commercial-client-plugins-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-common-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 公共文件<br><span class="hljs-attribute">mysql</span>-commercial-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-devel-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 开发库<br><span class="hljs-attribute">mysql</span>-commercial-icu-data-files-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm ICU<br><span class="hljs-attribute">mysql</span>-commercial-libs-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 共享库<br><span class="hljs-attribute">mysql</span>-commercial-libs-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-server-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 服务端<br><span class="hljs-attribute">mysql</span>-commercial-server-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-server-debug-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 服务端调试组件<br><span class="hljs-attribute">mysql</span>-commercial-server-debug-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-commercial-test-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 测试组件<br><span class="hljs-attribute">mysql</span>-commercial-test-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">mysql</span>-router-commercial-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm MySQL路由<br><span class="hljs-attribute">mysql</span>-router-commercial-debuginfo-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm 调试组件<br><span class="hljs-attribute">README</span>.txt 更新说明<br></code></pre></td></tr></table></figure><h3 id="安装依赖包"><a href="#安装依赖包" class="headerlink" title="安装依赖包"></a>安装依赖包</h3><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs cmake">dnf <span class="hljs-keyword">install</span> net-tools -y<br>dnf <span class="hljs-keyword">install</span> perl-Module-Build.noarch<br></code></pre></td></tr></table></figure><h3 id="安装rpm包（必要安装包）"><a href="#安装rpm包（必要安装包）" class="headerlink" title="安装rpm包（必要安装包）"></a>安装rpm包（必要安装包）</h3><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">rpm</span> -ivh mysql-commercial-client-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm \<br><span class="hljs-attribute">mysql</span>-commercial-client-plugins-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm \<br><span class="hljs-attribute">mysql</span>-commercial-common-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm \<br><span class="hljs-attribute">mysql</span>-commercial-icu-data-files-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm \<br><span class="hljs-attribute">mysql</span>-commercial-libs-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm \<br><span class="hljs-attribute">mysql</span>-commercial-server-<span class="hljs-number">8</span>.<span class="hljs-number">0</span>.<span class="hljs-number">30</span>-<span class="hljs-number">1</span>.<span class="hljs-number">1</span>.el9.x86_64.rpm<br></code></pre></td></tr></table></figure><h3 id="验证MySQL是否安装成功"><a href="#验证MySQL是否安装成功" class="headerlink" title="验证MySQL是否安装成功"></a>验证MySQL是否安装成功</h3><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs gradle">mysqladmin --version<br>cat <span class="hljs-regexp">/etc/</span>passwd|<span class="hljs-keyword">grep</span> mysql<br>cat <span class="hljs-regexp">/etc/g</span>roup|<span class="hljs-keyword">grep</span> mysql<br>usermod mysql -m -g mysql -s <span class="hljs-regexp">/bin/</span>bash -d <span class="hljs-regexp">/home/my</span>sql<br><br>rpm -qa | <span class="hljs-keyword">grep</span> -i mysql<br></code></pre></td></tr></table></figure><h3 id="服务状态"><a href="#服务状态" class="headerlink" title="服务状态"></a>服务状态</h3><figure class="highlight nsis"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs nsis"><span class="hljs-params">system</span>ctl status mysqld<br><span class="hljs-params">system</span>ctl start mysqld<br><span class="hljs-params">system</span>ctl stop mysqld<br><span class="hljs-params">system</span>ctl restart mysqld<br><br>设置开机自启动：<br><span class="hljs-params">system</span>ctl enable mysqld<br>chkconfig --list | grep mysql<br><span class="hljs-params">system</span>ctl list-unit-files<br></code></pre></td></tr></table></figure><h3 id="初始化"><a href="#初始化" class="headerlink" title="初始化"></a>初始化</h3><h4 id="因MySQL-8-0在Linux下库表名默认区分大小写，且只能在初始化时指定不能修改，现直接修改初始化配置文件避免重新初始化"><a href="#因MySQL-8-0在Linux下库表名默认区分大小写，且只能在初始化时指定不能修改，现直接修改初始化配置文件避免重新初始化" class="headerlink" title="因MySQL 8.0在Linux下库表名默认区分大小写，且只能在初始化时指定不能修改，现直接修改初始化配置文件避免重新初始化"></a>因MySQL 8.0在Linux下库表名默认区分大小写，且只能在初始化时指定不能修改，现直接修改初始化配置文件避免重新初始化</h4><p>mysqld –initialize –user=mysql –lower-case-table-names=1 –datadir=/opt/mysql/data/</p><p>#查看初始密码<br>cat /var/log/mysqld.log | grep password<br>#报错<br>##文件打不开<br>cp /usr/share/mysql-8.0/english/errmsg.sys /usr/share/mysql-8.0/errmsg.sys<br>##socket文件打不开<br>mkdir -p /var/lib/mysql<br>chown -R mysql:mysql /var/lib/mysql</p><h3 id="创建用户"><a href="#创建用户" class="headerlink" title="创建用户"></a>创建用户</h3><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-comment">## 先设置密码，例如：Mysql@123，注意密码为高安保等级（例如大小写和特殊字符的组合），不然无法使用其他操作</span><br>ALTER<span class="hljs-built_in"> USER </span><span class="hljs-string">'root'</span>@<span class="hljs-string">'localhost'</span> IDENTIFIED BY <span class="hljs-string">'你的密码'</span>;<br><span class="hljs-comment">## 看当前所有数据库</span><br>show databases;<br><span class="hljs-comment">## 进入mysql库</span><br>use mysql;<br><span class="hljs-comment">## 查看当前默认规则：</span><br>show variables like <span class="hljs-string">'validate_password%'</span>;<br><span class="hljs-comment">## 修改校验密码策略等级:LOW，默认为：MEDIUM</span><br><span class="hljs-built_in">set</span> global validate_password.<span class="hljs-attribute">policy</span>=LOW;<br><span class="hljs-comment">## 设置密码长度至少为 6，默认为8；</span><br><span class="hljs-built_in">set</span> global validate_password.<span class="hljs-attribute">length</span>=6;<br><span class="hljs-comment">## 立即生效</span><br>flush privileges;<br><span class="hljs-comment">## 查看用户信息</span><br>select host, user, authentication_string, plugin <span class="hljs-keyword">from</span><span class="hljs-built_in"> user;</span><br><span class="hljs-built_in"></span><span class="hljs-comment"># 更新root信息</span><br>update<span class="hljs-built_in"> user </span><span class="hljs-built_in">set</span> <span class="hljs-attribute">host</span>=<span class="hljs-string">'%'</span> where <span class="hljs-attribute">user</span>=<span class="hljs-string">'root'</span>;<br><span class="hljs-comment"># 授权root用户可以远程登陆</span><br>GRANT ALL ON *.* <span class="hljs-keyword">TO</span> <span class="hljs-string">'root'</span>@<span class="hljs-string">'%'</span>;<br><span class="hljs-comment">## 远程连接设置</span><br>ALTER<span class="hljs-built_in"> USER </span><span class="hljs-string">'root'</span>@<span class="hljs-string">'%'</span> IDENTIFIED WITH mysql_native_password BY <span class="hljs-string">'你的密码'</span>; <br><span class="hljs-comment">## 立即生效</span><br>flush privileges;<br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>数据库</category>
<category>MySQL</category>
</categories>
<tags>
<tag>MySQL</tag>
<tag>自建数据库</tag>
</tags>
</entry>
<entry>
<title>ES学习资料</title>
<link href="/2022/07/18/ES%E5%AD%A6%E4%B9%A0%E8%B5%84%E6%96%99/"/>
<url>/2022/07/18/ES%E5%AD%A6%E4%B9%A0%E8%B5%84%E6%96%99/</url>
<content type="html"><![CDATA[<h2 id="关系型DB与ES元数据区别"><a href="#关系型DB与ES元数据区别" class="headerlink" title="关系型DB与ES元数据区别"></a>关系型DB与ES元数据区别</h2><p>Relational DB -> Databases -> Tables -> Rows -> Columns</p><p>Elasticsearch -> Indices -> Types -> Documents -> Fields</p><h2 id="模板"><a href="#模板" class="headerlink" title="模板"></a>模板</h2><p><strong>动态模板参数：</strong></p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-comment">//匹配索引名称,*为通配符,可以配置多个, 对于有固定后缀以及按照时间创建的索引特别方便</span><br> <span class="hljs-attr">"index_patterns"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"index_name_pattern*"</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//template匹配优先级,值越小 优先级越高</span><br> <span class="hljs-attr">"order"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">0</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//索引版本号,仅用于外部管理,可不加</span><br> <span class="hljs-attr">"version"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1001</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//索引设置信息 如副本、分片、分词器配置等</span><br> <span class="hljs-attr">"settings"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> ...<br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//动态别名配置</span><br> <span class="hljs-attr">"aliases"</span><span class="hljs-punctuation">:</span><span class="hljs-punctuation">{</span><br> <span class="hljs-comment">//所有匹配索引全部添加对应别名</span><br> <span class="hljs-attr">"alias1"</span> <span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//根据条件过滤,符合条件添加别名</span><br> <span class="hljs-attr">"alias2"</span> <span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"filter"</span> <span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"term"</span> <span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><span class="hljs-attr">"user"</span> <span class="hljs-punctuation">:</span> <span class="hljs-string">"kimchy"</span> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"routing"</span> <span class="hljs-punctuation">:</span> <span class="hljs-string">"kimchy"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//给每个索引添加 索引名称-alias 别名</span><br> <span class="hljs-attr">"{index}-alias"</span> <span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><span class="hljs-punctuation">}</span> <br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//索引配置内容</span><br> <span class="hljs-attr">"mappings"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-comment">//索引type类型,7版本(包括7版本)之后取消了type, 7版本默认内置_doc</span><br> <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-comment">//内置属性设置 默认下划线开头 如_source, _all等</span><br> <span class="hljs-attr">"_source"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"enabled"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">false</span></span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//日期格式化方式</span><br> <span class="hljs-attr">"dynamic_date_formats"</span><span class="hljs-punctuation">:</span><span class="hljs-punctuation">[</span><br> <span class="hljs-string">"yyyy-MM-dd HH:mm:ss"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"yyyy-MM-dd"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//动态字段属性配置 与创建索引时指定mapping类似, 同样可以使用*通配</span><br> <span class="hljs-attr">"dynamic_templates"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">{</span><br> <span class="hljs-comment">//所有以Time结尾的字段都匹配为日期, 注意同一个字段只能有一种date_format格式</span><br> <span class="hljs-attr">"time_field"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"mapping"</span><span class="hljs-punctuation">:</span><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span><span class="hljs-string">"date"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"match"</span><span class="hljs-punctuation">:</span><span class="hljs-string">"*Time"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> ...<br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-comment">//指定字段匹配</span><br> <span class="hljs-attr">"properties"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"host_name"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"keyword"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> ...<br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br><br></code></pre></td></tr></table></figure><p><strong>动态模板2</strong></p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><code class="hljs json">PUT _template/fruits_template<br><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"index_patterns"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"shop*"</span><span class="hljs-punctuation">,</span> <span class="hljs-string">"bar*"</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span> <span class="hljs-comment">// 可以通过"shop*"和"bar*"来适配, template字段已过期</span><br> <span class="hljs-attr">"order"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">0</span><span class="hljs-punctuation">,</span> <span class="hljs-comment">// 模板的权重, 多个模板的时候优先匹配用, 值越大, 权重越高</span><br> <span class="hljs-attr">"settings"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"number_of_shards"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1</span> <span class="hljs-comment">// 分片数量, 可以定义其他配置项</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"aliases"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"alias_1"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><span class="hljs-punctuation">}</span> <span class="hljs-comment">// 索引对应的别名</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"mappings"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-comment">// ES 6.0开始只支持一种type, 名称为“_doc”</span><br> <span class="hljs-attr">"_doc"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"_source"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-comment">// 是否保存字段的原始值</span><br> <span class="hljs-attr">"enabled"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">false</span></span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"properties"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-comment">// 字段的映射</span><br> <span class="hljs-attr">"@timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-comment">// 具体的字段映射</span><br> <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"date"</span><span class="hljs-punctuation">,</span> <br> <span class="hljs-attr">"format"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"yyyy-MM-dd HH:mm:ss"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"@version"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"doc_values"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">true</span></span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"index"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"false"</span><span class="hljs-punctuation">,</span> <span class="hljs-comment">// 设置为false, 不索引</span><br> <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"text"</span> <span class="hljs-comment">// text类型</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"logLevel"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"long"</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p><strong>模板参数官方文档：</strong></p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.1/index-modules.html">https://www.elastic.co/guide/en/elasticsearch/reference/7.1/index-modules.html</a></p><p><strong>多表关联模板设计</strong></p>]]></content>
<categories>
<category>数据库</category>
<category>ElasticSearch</category>
</categories>
<tags>
<tag>ElasticSearch</tag>
<tag>搜索引擎</tag>
</tags>
</entry>
<entry>
<title>MySQL压测工具-super-smack</title>
<link href="/2022/06/27/MySQL%E5%8E%8B%E6%B5%8B%E5%B7%A5%E5%85%B7-super-smack/"/>
<url>/2022/06/27/MySQL%E5%8E%8B%E6%B5%8B%E5%B7%A5%E5%85%B7-super-smack/</url>
<content type="html"><![CDATA[<p>介绍这个主要是测试MySQL多线程指定SQL语句执行，测试其QPS。</p><p>安装：</p><p>仓库地址：<a href="https://github.com/winebarrel/super-smack">https://github.com/winebarrel/super-smack</a></p><p>编译安装（依赖包都太老了，基本安装不成功）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs SHELL">./configure --prefix=/usr/local/super-smack--with-mysql \<br> --with-mysql-lib=/var/lib/mysql \<br> --with-mysql-include=/usr/include/mysql \<br> --with-mysql<br> <br>make -j 4 && make install<br></code></pre></td></tr></table></figure><p>CentOS可直接现在rpm安装包</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs SHELL">rmp -ivh super-smack-1.3.3-1.el7.x86_64.rpm<br></code></pre></td></tr></table></figure><p>安装好后，运行 <code>which super-smack</code> 可以查到具体安装位置<code>/usr/bin/super-smack</code></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">/usr/bin/super-smack #压测可执行文件<br>/usr/bin/gen-data #随机生成压测条件数据<br>/usr/share/smacks #存放压测脚本<br>/usr/share/smacks/select-key.smack #根据随机键压测查询<br>/usr/share/smacks/update-select.smack #根据随机键压测更新、查询<br></code></pre></td></tr></table></figure><p>自动生成随机查询条件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs shell">gen-data -n 10 -f %48-64s%n,%32-64s,%n,%d > /usr/share/smacks/locationcenter.dat<br>-n 生成随机数行数<br>-f 格式<br>s 表示字符串<br>n 表示序列<br>d 表示数字<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看字典文件</span><br>cat /usr/share/smacks/locationcenter.dat <br></code></pre></td></tr></table></figure><p>执行压测语句</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">super-smack -dmysql /usr/share/smacks/select-key-userlocation.smack -D /usr/share/smacks/ 5 200<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>压测工具</category>
</categories>
<tags>
<tag>super-smack</tag>
<tag>数据库压测</tag>
</tags>
</entry>
<entry>
<title>MySQl压测工具-sysbench</title>
<link href="/2022/06/27/MySQl%E5%8E%8B%E6%B5%8B%E5%B7%A5%E5%85%B7-sysbench/"/>
<url>/2022/06/27/MySQl%E5%8E%8B%E6%B5%8B%E5%B7%A5%E5%85%B7-sysbench/</url>
<content type="html"><![CDATA[<p>sysBench是一个模块化工具，用于测试各种不同系统参数下的OLAP(MySQL/PorsgreSQL/Oracle)数据库负载情况。</p><p>仓库地址：<a href="https://github.com/akopytov/sysbench">https://github.com/akopytov/sysbench</a></p><p>（1）CPU性能<br>（2）磁盘IO性能<br>（3）内存分配及传输速度<br>（4）POSIX线程性能<br>（5）mutex调度性能<br>（6）数据库性能(OLTP基准测试) </p><p><strong>系统安装</strong></p><p>编译、yum都可以，网上很多教程。</p><p><strong>指令参数</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench [general-options]... --test=<test-name> [test-options]... command <br><br>--threads: 线程数，若设置为2，则sysbench会启动2个线程，同时分别进行素数的计算，默认值为1。<br>--time: 运行时长，单位秒。若设置为5，则sysbench会在5秒内循环进行素数计算，每完成一轮就是一个event，默认值为10。<br>--events: event上限次数，若设置为100，则表示当完成100次event后，即使时间富余也停止运行。默认值为0，则表示不限event次数。<br>--forced-shutdown=STRING：超时强制中断，默认为off。 --thread-stack-size=SIZE：线程栈大小，默认64K<br>--thread-init-timeout=N：线程初始化等待时间，默认为30秒<br>--rate=N：平均事务率，0表示不限制，默认为0。 <br>--report-interval=N：测试进度报告输出的时间间隔，0表示关闭，默认为0。<br>--report-checkpoints=[LIST,...]：转储完全统计信息并在指定时间点复位所有计数器，参数是逗号分隔值的列表，表示从必须执行报告检查点的测试开始所经过的时间（以秒为单位）。 默认情况下，报告检查点处于关闭状态[off]。<br>--debug[=on|off]：打印调试信息，默认为off<br>--validate[=on|off]：尽可能执行验证检查，默认为off。 <br>--help[=on|off]：打印help信息并退出，默认为off <br>--version[=on|off]：打印版本信息并退出，默认为off <br>--config-file=FILENAME：命令行选项文件<br>--luajit-cmd=STRING：执行LuaJIT控制命令<br></code></pre></td></tr></table></figure><p>commond:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell">prepare：测试前准备工作，例如为fileis测试在磁盘上创建必要的文件，为OLTP测试准备测试数据 <br>run：执行完整的测试，必须指定–-test选项 <br>cleanup：测试结束后删除数据<br>help：帮助<br></code></pre></td></tr></table></figure><p>**CPU性能测试 **</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench --test=cpu help<br>cpu options:<br> --cpu-max-prime=N upper limit for primes generator [10000]<br></code></pre></td></tr></table></figure><p>–cpu-max-prime：素数生成数量的上限，若设置为3，则表示2、3、5（要计算1-5共5次），默认值为10000。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench cpu --cpu-max-prime=20000 --threads=8 --time=30 run<br></code></pre></td></tr></table></figure><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206270957063.png" alt="image-20220627095739910"></p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs log">Prime numbers limit: 20000，每个线程产生的素数上限均为2000<br>events per second: 721.63，所有线程每秒完成了721.63次event<br>total time: 30.0056s，共耗时30秒<br>total number of events: 21654，30秒内所有线程一共完成了21654次event<br>min: 2.70，完成1次event的最少耗时2.70秒<br>avg: 11.08，所有event的平均耗时11.08毫秒<br>max: 129.76，完成1次event的最多耗时129.76毫秒<br>95th percentile: 30.26， 95%次event在30.26秒毫秒内完成<br>sum: 239822.50，每个线程耗时30秒，8个线程叠加耗时就是240秒<br>events (avg/stddev): 2706.7500/362.10 // 平均每个线程完成2706次event，标准差为362<br>execution time (avg/stddev): 29.9778/0.01 // 每个线程平均耗时29秒，标准差为0<br></code></pre></td></tr></table></figure><p><strong>内存负载测试</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench --test=memory help<br>memory options:<br> --memory-block-size=SIZE 测试block块大小 [默认1K]<br> --memory-total-size=SIZE 文件总传输量 [默认100G]<br> --memory-scope=STRING memory access scope {global,local} [global]<br> --memory-hugetlb[=on|off] allocate memory from HugeTLB pool [off]<br> --memory-oper=STRING type of memory operations {read, write, none} [write]<br> --memory-access-mode=STRING memory access mode {seq,rnd} [seq]<br></code></pre></td></tr></table></figure><p>开启12个线程，内存块大小为8K，顺序读写</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench memory --threads=12 --memory-block-size=8K --memory-total-size=100G --memory-access-mode=seq run<br>Running the test with following options:<br>Number of threads: 12<br>Initializing random number generator from current time<br>Running memory speed test with the following options:<br> block size: 8KiB<br> total size: 102400MiB<br> operation: write<br> scope: global<br>Initializing worker threads...<br>Threads started!<br>Total operations: 13107192 (1684079.88 per second)<br>102399.94 MiB transferred (13156.87 MiB/sec)<br>General statistics:<br> total time: 7.7816s<br> total number of events: 13107192<br>Latency (ms):<br> min: 0.00<br> avg: 0.01<br> max: 139.09<br> 95th percentile: 0.00<br> sum: 68748.61<br>Threads fairness:<br> events (avg/stddev): 1092266.0000/0.00<br> execution time (avg/stddev): 5.7291/1.08<br><br>sysbench memory --threads=12 --memory-block-size=8K --memory-total-size=100G --memory-access-mode=rnd run<br>Running the test with following options:<br>Number of threads: 12<br>Initializing random number generator from current time<br>Running memory speed test with the following options:<br> block size: 8KiB<br> total size: 102400MiB<br> operation: write<br> scope: global<br>Initializing worker threads...<br>Threads started!<br>Total operations: 3099810 (309929.69 per second)<br>24217.27 MiB transferred (2421.33 MiB/sec)<br>General statistics:<br> total time: 10.0002s<br> total number of events: 3099810<br>Latency (ms):<br> min: 0.00<br> avg: 0.04<br> max: 95.04<br> 95th percentile: 0.01<br> sum: 116396.55<br>Threads fairness:<br> events (avg/stddev): 258317.5000/22378.41<br> execution time (avg/stddev): 9.6997/0.09 <br></code></pre></td></tr></table></figure><p><strong>磁盘IO性能基准测试</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench --test=fileio help<br>fileio options:<br> --file-num=N：生成测试文件的数量，默认为128。<br> --file-block-size=N：测试时所使用文件块的大小，如果想磁盘针对innodb存储引擎进行测试，可以将其设置为16384（innodb存储引擎页的大小），默认为16384。<br> --file-total-size=SIZE：创建测试文件的总大小，默认为2G大小。<br> --file-test-mode=STRING：文件测试模式，seqwr(顺序写)、seqrewr(顺序读写)、seqrd(顺序读)、rndrd(随机读)、rndwr(随机写)、rndrw(随机读写)。<br> --file-io-mode=STRING：文件操作模式，sync（同步）、async（异步）、fastmmap（快速mmap）、slowmmap（慢速mmap），默认为sync。<br> --file-async-backlog=N：对应每个线程队列的异步操作数，默认为128。<br> --file-extra-flags=STRING：打开文件时的选项，与API相关的参数。<br> --file-fsync-freq=N：执行fsync函数的频率。fsync主要是同步磁盘文件，0代表不使用fsync函数，默认值为100。<br> --file-fsync-all=[on|off]：每执行完一次写操作，就执行一次fsync。默认为off。<br> --file-fsync-end=[on|off]：测试结束时执行fsync函数，默认为on。<br> --file-fsync-mode=STRING：文件同步函数选择，和API相关的参数，由于多个操作系统对于fdatasync支持不同，因此不建议使用fdatasync，默认为fsync。<br> --file-merged-requests=N：大多情况下，合并可能的IO的请求数，默认为0。<br> --file-rw-ratio=N：测试时的读写比例，默认时为1.5，即可3：2。<br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">数据准备</span><br>sysbench fileio --file-num=16 --file-total-size=2G prepare<br><span class="hljs-meta prompt_">#</span><span class="language-bash">测试</span><br>sysbench fileio --file-total-size=2G --file-test-mode=rndrd --time=180 --threads=16 --file-num=16 --file-extra-flags=direct --file-fsync-freq=0 --file-block-size=16384 run<br><span class="hljs-meta prompt_">#</span><span class="language-bash">测试数据清理</span><br>sysbench fileio --file-num=16 --file-total-size=2G cleanup<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><code class="hljs log">Running the test with following options:<br>Number of threads: 16<br>Initializing random number generator from current time<br>Extra file open flags: directio<br>16 files, 128MiB each<br>2GiB total file size<br>Block size 16KiB<br>Number of IO requests: 0<br>Read/Write ratio for combined random IO test: 1.50<br>Calling fsync() at the end of test, Enabled.<br>Using synchronous I/O mode<br>Doing random read test<br>Initializing worker threads...<br>Threads started!<br>File operations:<br> reads/s: 2291.53<br> writes/s: 0.00<br> fsyncs/s: 0.00<br>Throughput:<br> read, MiB/s: 35.81<br> written, MiB/s: 0.00<br>General statistics:<br> total time: 180.0079s<br> total number of events: 412498<br>Latency (ms):<br> min: 0.10<br> avg: 6.98<br> max: 49.85<br> 95th percentile: 9.91<br> sum: 2879915.11<br>Threads fairness:<br> events (avg/stddev): 25781.1250/46.71<br> execution time (avg/stddev): 179.9947/0.00<br></code></pre></td></tr></table></figure><p><strong>多线程调度基准测试</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench --test=mutex help <br>mutex options:<br> --mutex-num=N total size of mutex array [4096]<br> --mutex-locks=N number of mutex locks to do per thread [50000]<br> --mutex-loops=N number of empty loops to do outside mutex lock [10000]<br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench mutex --threads=8 --mutex-num=2000 --mutex-locks=10000 --mutex-loops=5000 run<br>Running the test with following options:<br>Number of threads: 8<br>Initializing random number generator from current time<br>Initializing worker threads...<br>Threads started!<br>General statistics:<br> total time: 0.1319s<br> total number of events: 8<br>Latency (ms):<br> min: 33.03<br> avg: 98.79<br> max: 128.91<br> 95th percentile: 127.81<br> sum: 790.31<br>Threads fairness:<br> events (avg/stddev): 1.0000/0.00<br> execution time (avg/stddev): 0.0988/0.03<br></code></pre></td></tr></table></figure><p><strong>POST多线程基准测试</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench --test=threads help<br>threads options:<br> --threads=N 开启线程数量<br> --thread-yields=N number of yields to do per request [1000]<br> --thread-locks=N number of locks per thread [8]<br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs shell">sysbench threads --threads=8 --thread-yields=200 --thread-locks=2 run<br>Running the test with following options:<br>Number of threads: 8<br>Initializing random number generator from current time<br>Initializing worker threads...<br>Threads started!<br>General statistics:<br> total time: 10.0018s<br> total number of events: 30114<br>Latency (ms):<br> min: 0.16<br> avg: 2.66<br> max: 239.52<br> 95th percentile: 7.56<br> sum: 79992.02<br>Threads fairness:<br> events (avg/stddev): 3764.2500/200.20<br> execution time (avg/stddev): 9.9990/0.00<br></code></pre></td></tr></table></figure><p><strong><font color=red>OLTP基准测试</font></strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">参数解析</span><br><span class="hljs-meta prompt_">#</span><span class="language-bash">连接信息</span><br>--mysql-host：MySQL服务器主机名，默认localhost；如果在本机上使用localhost报错，提示无法连接MySQL服务器，改成本机的IP地址应该就可以了。<br>--mysql-port：MySQL服务器端口，默认3306<br>--mysql-user：用户名<br>--mysql-password：密码<br><span class="hljs-meta prompt_">#</span><span class="language-bash">=执行参数</span><br>--oltp-test-mode：执行模式，包括simple、nontrx和complex，默认是complex。simple模式下只测试简单的查询；nontrx不仅测试查询，还测试插入更新等，但是不使用事务；complex模式下测试最全面，会测试增删改查，而且会使用事务。可以根据自己的需要选择测试模式。<br>--oltp-tables-count：测试的表数量，根据实际情况选择<br>--oltp-table-size：测试的表的大小，根据实际情况选择<br>--threads：客户端的并发连接数<br>--time：测试执行的时间，单位是秒，该值不要太短，可以选择120<br></code></pre></td></tr></table></figure><p>Lua脚本用途</p><table><thead><tr><th align="center"><strong>脚本名称</strong></th><th align="center"><strong>脚本作用</strong></th></tr></thead><tbody><tr><td align="center">bulk_insert</td><td align="center">批量插入性能</td></tr><tr><td align="center">oltp_common</td><td align="center">初始化脚本，表结构初始化，数据填充方式等</td></tr><tr><td align="center">oltp_delete</td><td align="center">事务性删除性能</td></tr><tr><td align="center">oltp_insert</td><td align="center">事务性插入性能</td></tr><tr><td align="center">oltp_point_select</td><td align="center">点查性能</td></tr><tr><td align="center">oltp_read_only</td><td align="center">读性能</td></tr><tr><td align="center">oltp_read_write</td><td align="center">读写性能</td></tr><tr><td align="center">oltp_update_index</td><td align="center">有索引情况下更新性能</td></tr><tr><td align="center">oltp_update_non_index</td><td align="center">无索引情况下更新性能</td></tr><tr><td align="center">oltp_write_only</td><td align="center">写性能</td></tr><tr><td align="center">select_random_points</td><td align="center">随机点查</td></tr><tr><td align="center">select_random_ranges</td><td align="center">随机范围查询</td></tr></tbody></table><p><strong>Lua脚本解析</strong><br><code>oltp_common.lua</code></p><p>大部分的脚本都会底层调用<code>oltp_common.lua</code>脚本里的函数<code>stmt_defs</code>的变量</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs sql">#point_selects:主键等值查询<br><span class="hljs-keyword">SELECT</span> c <span class="hljs-keyword">FROM</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">WHERE</span> id<span class="hljs-operator">=</span>?<br>#simple_ranges:简单范围查询(主键<span class="hljs-keyword">range</span>)<br><span class="hljs-keyword">SELECT</span> c <span class="hljs-keyword">FROM</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">WHERE</span> id <span class="hljs-keyword">BETWEEN</span> ? <span class="hljs-keyword">AND</span> ?<br>#sum_ranges:范围求和(简单范围查询的基础上做sum)<br><span class="hljs-keyword">SELECT</span> <span class="hljs-built_in">SUM</span>(k) <span class="hljs-keyword">FROM</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">WHERE</span> id <span class="hljs-keyword">BETWEEN</span> ? <span class="hljs-keyword">AND</span> ?<br>#order_ranges:简单范围查询(主键<span class="hljs-keyword">range</span>的基础上做排序)<br><span class="hljs-keyword">SELECT</span> c <span class="hljs-keyword">FROM</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">WHERE</span> id <span class="hljs-keyword">BETWEEN</span> ? <span class="hljs-keyword">AND</span> ? <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> c<br>#distinct_ranges:去重范围查询(主键<span class="hljs-keyword">range</span> <span class="hljs-operator">+</span> 排序 <span class="hljs-operator">+</span> 去重)<br><span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">DISTINCT</span> c <span class="hljs-keyword">FROM</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">WHERE</span> id <span class="hljs-keyword">BETWEEN</span> ? <span class="hljs-keyword">AND</span> ? <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> c<br>#index_updates:索引更新操作<br><span class="hljs-keyword">UPDATE</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">SET</span> k<span class="hljs-operator">=</span>k<span class="hljs-operator">+</span><span class="hljs-number">1</span> <span class="hljs-keyword">WHERE</span> id<span class="hljs-operator">=</span>?<br>#non_index_updates:非索引更新<br><span class="hljs-keyword">UPDATE</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">SET</span> c<span class="hljs-operator">=</span>? <span class="hljs-keyword">WHERE</span> id<span class="hljs-operator">=</span>?<br>#deletes:删除操作<br><span class="hljs-keyword">DELETE</span> <span class="hljs-keyword">FROM</span> sbtest<span class="hljs-operator">%</span>u <span class="hljs-keyword">WHERE</span> id<span class="hljs-operator">=</span>?<br>#inserts:插入操作<br><span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span> sbtest<span class="hljs-operator">%</span>u (id, k, c, pad) <span class="hljs-keyword">VALUES</span> (?, ?, ?, ?)<br></code></pre></td></tr></table></figure><p><code>oltp_common.lua</code><br>这个脚本相当于一个公共的函数库</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 初始化</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">init</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 创建表结构，获取c,pad字段随机字符串，插入数据，建立二级索引</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">create_table</span><span class="hljs-params">(drv, con, table_num)</span></span> <br> <span class="hljs-comment">-- 创建表结构，表结构如下</span><br> <span class="hljs-comment">-- CREATE TABLE `sbtest1` (</span><br> <span class="hljs-comment">-- `id` int(11) NOT NULL AUTO_INCREMENT, # 主键</span><br> <span class="hljs-comment">-- `k` int(11) NOT NULL DEFAULT '0', # 二级索引</span><br> <span class="hljs-comment">-- `c` char(120) NOT NULL DEFAULT '',</span><br> <span class="hljs-comment">-- `pad` char(60) NOT NULL DEFAULT '',</span><br> <span class="hljs-comment">-- PRIMARY KEY (`id`),</span><br> <span class="hljs-comment">-- KEY `k_1` (`k`)</span><br> <span class="hljs-comment">-- ) ENGINE=InnoDB AUTO_INCREMENT=1000001 DEFAULT CHARSET=latin1</span><br> <span class="hljs-comment">-- get_c_value() 获取字段c的值</span><br> <span class="hljs-comment">-- get_pad_value() 获取字段pad的值</span><br> <span class="hljs-comment">-- INSERT 语句插入到相关表，直到达到指定的table-size</span><br> <span class="hljs-comment">-- 建立二级索引k</span><br><span class="hljs-comment">-- 获取字段c的值，返回一个随机字符串</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">get_c_value</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 获取字段pad的值，返回一个随机字符串 </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">get_pad_value</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 初始化线程 </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">thread_init</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 关闭线程连接 </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">thread_done</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- Close prepared statements</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">close_statements</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 如果我们已经重新连接了，那么重新准备语句， </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">sysbench.hooks.before_restart_event</span><span class="hljs-params">(errdesc)</span></span><br> <span class="hljs-comment">-- close_statements() //Close prepared statements</span><br> <span class="hljs-comment">-- prepare_statements() // This function is a 'callback' defined by individual benchmark scripts</span><br><span class="hljs-comment">-- 构造语句 </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_for_each_table</span><span class="hljs-params">(key)</span></span> <br><span class="hljs-comment">-- 对应prepare命令 </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">cmd_prepare</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 对应prewarm命令 </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">cmd_prewarm</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 清理数据,把相关表都DROP掉</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">cleanup</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 启动事务 BEGIN</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_begin</span><span class="hljs-params">()</span></span> //构造 BEGIN 语句<br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">begin</span><span class="hljs-params">()</span></span> //执行 BEGIN 语句<br><span class="hljs-comment">-- 提交事务 COMMIT</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_commit</span><span class="hljs-params">()</span></span> //构造 BEGIN 语句<br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">commit</span><span class="hljs-params">()</span></span> //执行 COMMIT 语句<br><span class="hljs-comment">-- 主键等值查询 SELECT c FROM sbtest? WHERE id = ? function prepare_point_selects() // 调用 prepare_for_each_table() 函数生成ponit_selects语句 function execute_point_selects() // 执行 prepare_point_selects() 生成的语句</span><br><span class="hljs-comment">-- 主键范围查询 SELECT c FROM sbtest? WHERE id BETWEEN ? AND ?</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_simple_ranges</span><span class="hljs-params">()</span></span> // 调用 prepare_for_each_table() 函数生成simple_ranges语句 <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">execute_simple_ranges</span><span class="hljs-params">()</span></span> // 执行 prepare_point_selects() 生成的语句 <br><span class="hljs-comment">-- 主键范围查询 + 求和 SELECT SUM(k) FROM sbtest? WHERE id BETWEEN ? AND ?</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_sum_ranges</span><span class="hljs-params">()</span></span> // 调用 prepare_for_each_table() 函数生成sum_ranges语句 <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">execute_sum_ranges</span><span class="hljs-params">()</span></span> // 执行 prepare_point_selects() 生成的语句 <br><span class="hljs-comment">-- 主键范围查询 + 排序 SELECT c FROM sbtest%u WHERE id BETWEEN ? AND ? ORDER BY c function prepare_order_ranges()</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">execute_order_ranges</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 主键范围查询 + 排序 + 去重 SELECT DISTINCT c FROM sbtest%u WHERE id BETWEEN ? AND ? ORDER BY c</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_distinct_ranges</span><span class="hljs-params">()</span></span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">execute_distinct_ranges</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 二级索引上更新(k是二级索引) UPDATE sbtest%u SET k=k+1 WHERE id=? function prepare_index_updates() // 调用 prepare_for_each_table() 函数生成index_updates语句 function execute_index_updates() // 执行 prepare_point_selects() 生成的语句</span><br><span class="hljs-comment">-- 无索引的更新 UPDATE sbtest%u SET c=? WHERE id=? function prepare_non_index_updates() </span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">execute_non_index_updates</span><span class="hljs-params">()</span></span> <br><span class="hljs-comment">-- 删除 DELETE FROM sbtest%u WHERE id=?</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">prepare_delete_inserts</span><span class="hljs-params">()</span></span> <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">execute_delete_inserts</span><span class="hljs-params">()</span></span> // 执行 prepare_for_each_table 生成的语句<br></code></pre></td></tr></table></figure><p>读写测试的语句格式如下（定义在<code>stmt_defs</code>变量中）</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">--point_selects:主键等值查询</span><br>SELECT c FROM sbtest%u WHERE id=?<br><span class="hljs-comment">-- simple_ranges:简单范围查询(主键range)</span><br>SELECT c FROM sbtest%u WHERE id BETWEEN ? AND ?<br><span class="hljs-comment">-- sum_ranges:范围求和(简单范围查询的基础上做sum)</span><br>SELECT SUM(k) FROM sbtest%u WHERE id BETWEEN ? AND ?<br><span class="hljs-comment">-- order_ranges:简单范围查询(主键range的基础上做排序)</span><br>SELECT c FROM sbtest%u WHERE id BETWEEN ? AND ? ORDER BY c<br><span class="hljs-comment">-- distinct_ranges:去重范围查询(主键range + 排序 + 去重)</span><br>SELECT DISTINCT c FROM sbtest%u WHERE id BETWEEN ? AND ? ORDER BY c<br><span class="hljs-comment">-- index_updates:索引更新操作</span><br>UPDATE sbtest%u SET k=k+<span class="hljs-number">1</span> WHERE id=?<br><span class="hljs-comment">-- non_index_updates:非索引更新</span><br>UPDATE sbtest%u SET c=? WHERE id=?<br><span class="hljs-comment">-- deletes:删除操作</span><br>DELETE FROM sbtest%u WHERE id=?<br><span class="hljs-comment">-- inserts:插入操作</span><br>INSERT INTO sbtest%u (id, k, c, pad) VALUES (?, ?, ?, ?)<br></code></pre></td></tr></table></figure><p>bulk_insert.lua<br>插入的lua脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 初始化线程</span><br><span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">thread_init</span><span class="hljs-params">()</span></span><br><span class="hljs-comment">-- 准备表结构</span><br>prepare()<br> <span class="hljs-comment">-- CREATE TABLE IF NOT EXISTS sbtest%d (</span><br> <span class="hljs-comment">-- id INTEGER NOT NULL,</span><br> <span class="hljs-comment">-- k INTEGER DEFAULT '0' NOT NULL,</span><br> <span class="hljs-comment">-- PRIMARY KEY (id))]], i)) </span><br><span class="hljs-comment">-- 批量插入</span><br>event()<br><span class="hljs-comment">-- 关闭线程</span><br>thread_done(thread_9d)<br><span class="hljs-comment">-- 清理数据，DROP掉相关的表</span><br>cleanup()<br></code></pre></td></tr></table></figure><p>oltp_delete.lua<br>DELETE语句的lua脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造DELETE语句</span><br>prepare_statements()<br> <span class="hljs-comment">-- prepare_for_each_table("deletes") 构造DELETE语句</span><br><span class="hljs-comment">-- 执行DELETE语句</span><br>event()<br></code></pre></td></tr></table></figure><p>oltp_insert.lua<br>INSERT语句的Lua脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造DELETE语句</span><br>prepare_statements()<br><span class="hljs-comment">-- 执行DELETE语句</span><br>event()<br></code></pre></td></tr></table></figure><p>oltp_point_select.lua<br>点查语句的Lua脚本，主键查询<code>SELECT c FROM sbtest? WHERE id = ?</code></p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本 </span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造点查语句</span><br>prepare_statements()<br><span class="hljs-comment">-- 执行语句</span><br>event()<br></code></pre></td></tr></table></figure><p>oltp_read_only.lua<br>只读语句的Lua脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造语句</span><br>prepare_statements()<br> <span class="hljs-comment">-- prepare_point_selects() // 构造点查语句</span><br> <span class="hljs-comment">-- prepare_begin() // 构造 BEGIN 语句</span><br> <span class="hljs-comment">-- prepare_commit() // 构造 COMMIT 语句</span><br> <span class="hljs-comment">-- prepare_simple_ranges() // 构造简单范围查询语句</span><br> <span class="hljs-comment">-- prepare_sum_ranges() // 构造求和语句</span><br> <span class="hljs-comment">-- prepare_order_ranges() // 构造排序语句</span><br> <span class="hljs-comment">-- prepare_distinct_ranges() // 构造排序去重语句</span><br><span class="hljs-comment">-- 执行语句</span><br>event() <br> <span class="hljs-comment">-- execute_point_selects() // 执行点查语句</span><br> <span class="hljs-comment">-- execute_simple_ranges() // 执行简单范围查询语句</span><br> <span class="hljs-comment">-- execute_sum_ranges() // 执行求和语句</span><br> <span class="hljs-comment">-- execute_order_ranges() // 执行排序语句</span><br> <span class="hljs-comment">-- execute_distinct_ranges() //排序去重语句</span><br></code></pre></td></tr></table></figure><p>oltp_read_write.lua<br>读写的Lua脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造语句</span><br>prepare_statements()<br> <span class="hljs-comment">-- 读操作</span><br> <span class="hljs-comment">-- prepare_point_selects() // 构造点查语句</span><br> <span class="hljs-comment">-- prepare_begin() // 构造 BEGIN 语句</span><br> <span class="hljs-comment">-- prepare_commit() // 构造 COMMIT 语句</span><br> <span class="hljs-comment">-- prepare_simple_ranges() // 构造简单范围查询语句</span><br> <span class="hljs-comment">-- prepare_sum_ranges() // 构造求和语句</span><br> <span class="hljs-comment">-- prepare_order_ranges() // 构造排序语句</span><br> <span class="hljs-comment">-- prepare_distinct_ranges() // 构造排序去重语句</span><br> <span class="hljs-comment">-- 写操作</span><br> <span class="hljs-comment">-- prepare_index_updates() // 构造更新索引操作语句</span><br> <span class="hljs-comment">-- prepare_non_index_updates() // 构造更新无索引字段操作语句</span><br> <span class="hljs-comment">-- prepare_delete_inserts() // 构造删除+更新操作语句</span><br><span class="hljs-comment">-- 执行语句</span><br>event() <br> <span class="hljs-comment">-- 读操作</span><br> <span class="hljs-comment">-- execute_point_selects() // 执行点查语句</span><br> <span class="hljs-comment">-- execute_simple_ranges() // 执行简单范围查询语句</span><br> <span class="hljs-comment">-- execute_sum_ranges() // 执行求和语句</span><br> <span class="hljs-comment">-- execute_order_ranges() // 执行排序语句</span><br> <span class="hljs-comment">-- execute_distinct_ranges() //排序去重语句</span><br> <span class="hljs-comment">-- 写操作</span><br> <span class="hljs-comment">-- execute_index_updates() // 执行更新索引操作语句</span><br> <span class="hljs-comment">-- execute_non_index_updates() // 执行更新无索引字段操作语句</span><br> <span class="hljs-comment">-- execute_delete_inserts() // 执行删除+更新操作语句</span><br></code></pre></td></tr></table></figure><p>oltp_update_index.lua<br>更新索引字段操作的Lua语句</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造语句</span><br>prepare_statements()<br> <span class="hljs-comment">-- prepare_index_updates() </span><br><span class="hljs-comment">-- 执行语句</span><br>event() <br> <span class="hljs-comment">-- execute_index_updates(con)</span><br></code></pre></td></tr></table></figure><p>oltp_update_non_index.lua<br>更新无索引字段操作语句</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造语句</span><br>prepare_statements()<br> <span class="hljs-comment">-- prepare_non_index_updates()</span><br><span class="hljs-comment">-- 执行语句</span><br>event() <br> <span class="hljs-comment">-- execute_non_index_updates()</span><br></code></pre></td></tr></table></figure><p>oltp_write_only.lua<br>纯写语句的Lua脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 构造语句</span><br>prepare_statements()<br> <span class="hljs-comment">-- prepare_index_updates()</span><br> <span class="hljs-comment">-- prepare_non_index_updates()</span><br> <span class="hljs-comment">-- prepare_delete_inserts()</span><br><span class="hljs-comment">-- 执行语句</span><br>event() <br> <span class="hljs-comment">-- execute_index_updates()</span><br> <span class="hljs-comment">-- execute_non_index_updates()</span><br> <span class="hljs-comment">-- execute_delete_inserts()</span><br></code></pre></td></tr></table></figure><p>select_random_points.lua<br>随机点查脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 准备阶段</span><br>prepare() <br><span class="hljs-comment">-- 线程初始化</span><br>thread_init() <br><span class="hljs-comment">-- 关闭线程</span><br>thread_done() <br><span class="hljs-comment">-- 执行相关SQL</span><br>event()<br></code></pre></td></tr></table></figure><p>select_random_ranges.lua</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 准备阶段</span><br>prepare() <br><span class="hljs-comment">-- 线程初始化</span><br>thread_init() <br><span class="hljs-comment">-- 关闭线程</span><br>thread_done() <br><span class="hljs-comment">-- 执行相关SQL</span><br>event()<br></code></pre></td></tr></table></figure><p>oints.lua<br>随机点查脚本</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 准备阶段</span><br>prepare() <br><span class="hljs-comment">-- 线程初始化</span><br>thread_init() <br><span class="hljs-comment">-- 关闭线程</span><br>thread_done() <br><span class="hljs-comment">-- 执行相关SQL</span><br>event()<br></code></pre></td></tr></table></figure><p>select_random_ranges.lua</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs lua"><span class="hljs-comment">-- 加载oltp_common脚本</span><br><span class="hljs-built_in">require</span>(<span class="hljs-string">"oltp_common"</span>)<br><span class="hljs-comment">-- 准备阶段</span><br>prepare() <br><span class="hljs-comment">-- 线程初始化</span><br>thread_init() <br><span class="hljs-comment">-- 关闭线程</span><br>thread_done() <br><span class="hljs-comment">-- 执行相关SQL</span><br>event()<br></code></pre></td></tr></table></figure><p>测试例子：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">准备测试脚本</span><br>mysqladmin -hxxx -uxxx -pxxx -P3306 create sbtest;<br><br>sysbench --mysql-host=xxx \<br> --mysql-port=3306 \<br> --mysql-user=xxx \<br> --mysql-password=xxx \<br> --mysql-db=sbtest \<br> /usr/share/sysbench/oltp_common.lua \<br> --tables=10 \<br> --table_size=100000 \<br> prepare<br><span class="hljs-meta prompt_">#</span><span class="language-bash">压测脚本</span><br>sysbench --db-driver=mysql --time=30 --threads=5 --report-interval=1 \<br> --mysql-host=xxx \<br> --mysql-port=3306 \<br> --mysql-user=xxx \<br> --mysql-password=xxx \<br> --mysql-db=sbtest --tables=10 --table_size=1000000 \<br> /usr/share/sysbench/oltp_point_select.lua \<br> --db-ps-mode=disable run<br><span class="hljs-meta prompt_">#</span><span class="language-bash">清理测试数据</span><br>sysbench --db-driver=mysql --time=30 --threads=5 --report-interval=1 \<br> --mysql-host=xxx \<br> --mysql-port=3306 \<br> --mysql-user=xxx \<br> --mysql-password=xxx \<br> --mysql-db=sbtest --tables=10 --table_size=1000000 \<br> oltp_read_only --db-ps-mode=disable cleanup<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>压测工具</category>
</categories>
<tags>
<tag>数据库压测</tag>
<tag>sysbench</tag>
</tags>
</entry>
<entry>
<title>ES初学</title>
<link href="/2022/06/18/ES%E5%88%9D%E5%AD%A6/"/>
<url>/2022/06/18/ES%E5%88%9D%E5%AD%A6/</url>
<content type="html"><![CDATA[<p>因阿里云es有多多限制，自建下es，使用docker安装</p><p>查询镜像</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">搜索镜像</span><br>docker search elasticsearch<br>docker search kibana<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看所有历史版本</span><br>curl https://registry.hub.docker.com/v1/repositories/elasticsearch/tags| tr -d '[\[\]" ]' | tr '}' '\n'| awk -F: -v image='elasticsearch' '{if(NR!=NF && $3 != ""){printf("%s:%s\n",image,$3)}}'<br>curl https://registry.hub.docker.com/v1/repositories/elastic/kibana/tags| tr -d '[\[\]" ]' | tr '}' '\n'| awk -F: -v image='kibana' '{if(NR!=NF && $3 != ""){printf("%s:%s\n",image,$3)}}'<br></code></pre></td></tr></table></figure><p>拉取镜像</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker pull elasticsearch:7.10.1<br>docker pull kibana:7.10.1<br></code></pre></td></tr></table></figure><p>创建数据文件夹</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /opt/elasticsearch/config<br>mkdir -p /opt/elasticsearch/data<br><br>echo "http.host: 0.0.0.0" >/opt/elasticsearch/config/elasticsearch.yml<br>chmod -R 777 /opt/elasticsearch/<br></code></pre></td></tr></table></figure><p>启动容器</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">首次启动</span><br>docker run --name elasticsearch -p 9200:9200 -p 9300:9300 -d elasticsearch:7.10.1<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看容器中配置文件路径</span><br>find / -name elasticsearch.yml<br><span class="hljs-meta prompt_">#</span><span class="language-bash">删除容器</span><br>docker container ls -all<br>docker rm {$containerid}<br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">ES</span><br>docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \<br>-e "discovery.type=single-node" \<br>-e ES_JAVA_OPTS="-Xms64m -Xmx512m" \<br>-v /opt/elasticsearch/config/elasticsearch.yml:/var/lib/docker/overlay2/cb02e3fbbb80dfb252f1e7aaa1b35307f1424cbe21baf11bf36c7d895c572ff3/diff/usr/share/elasticsearch/config/elasticsearch.yml \<br>-v /opt/elasticsearch/data:/var/lib/docker/overlay2/cb02e3fbbb80dfb252f1e7aaa1b35307f1424cbe21baf11bf36c7d895c572ff3/diff/usr/share/elasticsearch/data \<br>-v /opt/elasticsearch/plugins:/var/lib/docker/overlay2/cb02e3fbbb80dfb252f1e7aaa1b35307f1424cbe21baf11bf36c7d895c572ff3/diff/usr/share/elasticsearch/plugins \<br>-d elasticsearch:7.10.1 <br><span class="hljs-meta prompt_"></span><br><span class="hljs-meta prompt_">#</span><span class="language-bash">设置开机启动elasticsearch</span><br>docker update elasticsearch --restart=always<br><span class="hljs-meta prompt_"></span><br><span class="hljs-meta prompt_">#</span><span class="language-bash">Kibana</span><br>docker run --name kibana -e ELASTICSEARCH_HOSTS=http://10.0.16.15:9200 -p 5601:5601 \<br>-e ES_JAVA_OPTS="-Xms32m -Xmx128m" \<br>-d kibana:7.10.1<br><span class="hljs-meta prompt_"></span><br><span class="hljs-meta prompt_">#</span><span class="language-bash">设置开机启动kibana</span><br>docker update kibana --restart=always<br></code></pre></td></tr></table></figure><p>查看容器运行状态</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}\t{{.Status}}"<br></code></pre></td></tr></table></figure><p>开启xpack</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">进入容器</span><br>docker attach elasticsearch<br><span class="hljs-meta prompt_">#</span><span class="language-bash">进不去的话换这个</span><br>docker exec -it elasticsearch /bin/bash<br><span class="hljs-meta prompt_">#</span><span class="language-bash">修改配置文件</span><br>cd config<br>vi elasticsearch.yml<br><span class="hljs-meta prompt_">#</span><span class="language-bash">添加保存</span><br>xpack.security.enabled: true<br><span class="hljs-meta prompt_">#</span><span class="language-bash">重启elasticsearch容器</span><br>docker restart elasticsearch<br><span class="hljs-meta prompt_">#</span><span class="language-bash">重新进入</span><br>docker exec -it elasticsearch /bin/bash<br><span class="hljs-meta prompt_">#</span><span class="language-bash">初始化密码</span><br>cd bin<br>./elasticsearch-setup-passwords interactive<br>890425<br><span class="hljs-meta prompt_">#</span><span class="language-bash">此处需要出入n个密码</span><br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">修改kibana配置文件</span><br><span class="hljs-meta prompt_">#</span><span class="language-bash">进入容器</span><br>docker exec -it kibana /bin/bash<br>cd config<br>vi kibana.yml<br></code></pre></td></tr></table></figure><p>新增连接es的地址、账号密码，添加汉化配置</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206221344241.png"></p><p>重启</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker restart kibana<br></code></pre></td></tr></table></figure><p>安装插件方法</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">下载</span><br>wget https://d3g5vo6xdbdb9a.cloudfront.net/downloads/elasticsearch-plugins/opendistro-sql/opendistro_sql-1.13.2.0.zip<br><span class="hljs-meta prompt_">#</span><span class="language-bash">复制</span><br>docker cp opendistro_sql-1.13.2.0.zip elasticsearch:/usr/share/elasticsearch/plugins<br><span class="hljs-meta prompt_">#</span><span class="language-bash">进入容器目录</span><br>docker exec -it elasticsearch /bin/bash<br>cd bin<br><span class="hljs-meta prompt_">#</span><span class="language-bash">执行</span><br>./elasticsearch-plugin install file:///usr/share/elasticsearch/plugins/opendistro_sql-1.13.2.0.zip<br></code></pre></td></tr></table></figure><p>客户端（类似、替代工具等）</p><p>es-head 最好用</p><p>下载地址：</p><p><a href="https://raw.githubusercontent.com/mobz/elasticsearch-head/master/crx/es-head.crx">https://raw.githubusercontent.com/mobz/elasticsearch-head/master/crx/es-head.crx</a></p><p>dbeave、dbvisualizer等都需要破解白金版，还不能通过验证</p><p>与关系型数据库对应关系</p><table><thead><tr><th>MySQL</th><th>ElasticSearch</th></tr></thead><tbody><tr><td>Table</td><td>Index</td></tr><tr><td>Table</td><td>Type</td></tr><tr><td>Row</td><td>Document</td></tr><tr><td>Column</td><td>Field</td></tr><tr><td>Schema</td><td>Mapping</td></tr><tr><td>Index</td><td>Everything is indexed</td></tr><tr><td>SQL</td><td>Query DSL</td></tr><tr><td>select * from …</td><td>GET http://…</td></tr><tr><td>update table set …</td><td>POST http://…</td></tr><tr><td>group by、avg、sum</td><td>Aggregations</td></tr><tr><td>去重 distinct</td><td>cardinality</td></tr><tr><td>数据迁移</td><td>reindex</td></tr></tbody></table><p>Mapping 映射</p><p>模板、动态模板</p><p>Mapping参数</p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html#mapping-params">https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html#mapping-params</a></p><p>Field字段类型</p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#mapping-types">https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#mapping-types</a></p>]]></content>
<categories>
<category>数据库</category>
<category>ElasticSearch</category>
</categories>
<tags>
<tag>ElasticSearch</tag>
<tag>搜索引擎</tag>
</tags>
</entry>
<entry>
<title>PolarDB-X应用测试</title>
<link href="/2022/06/10/PolarDB-X%E5%BA%94%E7%94%A8%E6%B5%8B%E8%AF%95/"/>
<url>/2022/06/10/PolarDB-X%E5%BA%94%E7%94%A8%E6%B5%8B%E8%AF%95/</url>
<content type="html"><![CDATA[<h1 id="PolarDB-X应用"><a href="#PolarDB-X应用" class="headerlink" title="PolarDB-X应用"></a>PolarDB-X应用</h1><h2 id="搭建实时分析系统"><a href="#搭建实时分析系统" class="headerlink" title="搭建实时分析系统"></a>搭建实时分析系统</h2><p>本步骤将指导您如何使用PolarDB-X+Canal+ClickHouse搭建实时分析系统。</p><ol><li>部署Canal。</li></ol><p>Canal是一款流行的MySQL Binlog增量订阅工具，详情请参见<a href="https://github.com/alibaba/canal">Canal说明文档</a>。Canal提供了Docker镜像，详情请参见<a href="https://github.com/alibaba/canal/wiki/Docker-QuickStart">Canal Docker镜像文档</a>。</p><p>(1) 执行如下命令，下载脚本。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">wget https:<span class="hljs-regexp">//</span>raw.githubusercontent.com<span class="hljs-regexp">/alibaba/</span>canal<span class="hljs-regexp">/master/</span>docker/run.sh<br></code></pre></td></tr></table></figure><p>(2) 执行如下命令，构建一个destination name为test的队列。</p><p><strong>注意 ：</strong>您需要将none_loopback_host_ip修改为云产品资源列表中的ECS的弹性IP，请勿使用localhost或127.0.0.1。</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs routeros">sh run.sh -e canal.auto.<span class="hljs-attribute">scan</span>=<span class="hljs-literal">false</span> \<br>-e canal.<span class="hljs-attribute">destinations</span>=test \<br>-e canal.instance.master.<span class="hljs-attribute">address</span>=none_loopback_host_ip:8527 \<br>-e canal.instance.<span class="hljs-attribute">dbUsername</span>=polardbx_root \<br>-e canal.instance.<span class="hljs-attribute">dbPassword</span>=123456 \<br>-e canal.instance.<span class="hljs-attribute">connectionCharset</span>=UTF-8 \<br>-e canal.instance.tsdb.<span class="hljs-attribute">enable</span>=<span class="hljs-literal">true</span> \<br>-e canal.instance.<span class="hljs-attribute">gtidon</span>=<span class="hljs-literal">false</span><br></code></pre></td></tr></table></figure><ol><li>部署ClickHouse。</li></ol><p>ClickHouse是一款分析系统，详情请参见<a href="https://clickhouse.com/">ClickHouse官方文档</a>。ClickHouse提供了Docker镜像，详情请参见<a href="https://hub.docker.com/r/clickhouse/clickhouse-server">ClickHouseDocker镜像文档</a>。</p><p>执行如下命令，部署ClickHouse。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> run -d --name some-clickhouse-server --ulimit nofile=<span class="hljs-number">262144</span>:<span class="hljs-number">262144</span> -p <span class="hljs-number">8123</span>:<span class="hljs-number">8123</span> yandex/clickhouse-server<br></code></pre></td></tr></table></figure><ol><li>在PolarDB-X和ClickHouse中创建测试库和表。</li></ol><p>(1) 执行如下命令，登录PolarDB-X数据库。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">mysql</span> -h<span class="hljs-number">127.0.0.1</span> -P8527 -upolardbx_root -p123456<br></code></pre></td></tr></table></figure><p>(2) 执行如下SQL语句，创建数据库testdb。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">DATABASE</span> testdb;<br></code></pre></td></tr></table></figure><p>(3) 执行如下SQL语句，使用数据库testdb。</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">USE testdb<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p>(4) 执行如下SQL语句，创建test表。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> test(<br>id <span class="hljs-type">INT</span>(<span class="hljs-number">11</span>) AUTO_INCREMENT <span class="hljs-keyword">PRIMARY KEY</span>,<br><span class="hljs-type">name</span> <span class="hljs-type">CHAR</span>(<span class="hljs-number">20</span>) <span class="hljs-keyword">not</span> <span class="hljs-keyword">null</span> );<br></code></pre></td></tr></table></figure><p>(5) 输入exit退出数据库。</p><p><img src="https://ucc.alicdn.com/pic/developer-ecology/83b8b5c857e0400481d57e4d0b0f0ef9.png" alt="img"></p><p>(6) 执行如下命令，登录ClickHouse数据库。</p><figure class="highlight axapta"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs axapta">docker run -it --rm --link some-clickhouse-<span class="hljs-keyword">server</span>:clickhouse-<span class="hljs-keyword">server</span> yandex/clickhouse-<span class="hljs-keyword">client</span> --host clickhouse-<span class="hljs-keyword">server</span><br></code></pre></td></tr></table></figure><p>(7) 执行如下SQL语句，创建数据库testdb。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">DATABASE</span> testdb;<br></code></pre></td></tr></table></figure><p>(8) 执行如下SQL语句，使用数据库testdb。</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">USE testdb<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p>(9) 执行如下SQL语句，创建test表。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">Create</span> <span class="hljs-keyword">Table</span> test(id <span class="hljs-type">INT</span>(<span class="hljs-number">32</span>),<span class="hljs-type">name</span> <span class="hljs-type">CHAR</span>(<span class="hljs-number">20</span>)) Engine = MergeTree() <span class="hljs-keyword">Order</span> <span class="hljs-keyword">By</span> id;<br></code></pre></td></tr></table></figure><p>(10) 输入exit退出数据库。</p><ol><li>运行Canal Client消费并投递增量变更。</li></ol><p>经过以上步骤，您已经准备好了PolarDB-X、Canal Server和ClickHouse三个容器，并且在源端（PolarDB-X）和目标端（ClickHouse）建好了测试用的数据库和表。接下来我们通过Canal Client消费Canal Server获取的增量数据，并将源端DML中的Insert事件投递到ClickHouse中。</p><p>(1) 执行如下命令，使用yum安装JDK 1.8。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">yum</span> -y install java-<span class="hljs-number">1</span>.<span class="hljs-number">8</span>.<span class="hljs-number">0</span>-openjdk*<br></code></pre></td></tr></table></figure><p>(2) 执行如下命令，下载polardb-x-to-clickhouse-canal-client.jar投递代码文件。</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs vim">wget https://labfileapp.oss-<span class="hljs-keyword">cn</span>-hangzhou.aliyuncs.<span class="hljs-keyword">com</span>/polardb-<span class="hljs-keyword">x</span>-<span class="hljs-keyword">to</span>-clickhouse-canal-client.jar<br></code></pre></td></tr></table></figure><p>(3) 执行如下命令，运行polardb-x-to-clickhouse-canal-client.jar代码文件。</p><figure class="highlight llvm"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs llvm">java -jar polardb-<span class="hljs-keyword">x</span>-<span class="hljs-keyword">to</span>-clickhouse-canal-client.jar<br></code></pre></td></tr></table></figure><p><strong>注意：</strong>请勿中断投递代码文件，否则会造成投递失败。</p><p>(4) 投递链路已成功打通，接下来您可以在源端（PolarDB-X）执行INSERT语句，并观察目标端（ClickHouse）中的数据变化。</p><p>在实验页面，单击右上角的<img src="https://ucc.alicdn.com/pic/developer-ecology/fb9a00a2cbff4bf59597914e7413505b.png" alt="img"> 图标，创建新的终端窗口。</p><p><img src="https://ucc.alicdn.com/pic/developer-ecology/e9d5f36fdfee40318fc6ea64b84841e0.png" alt="img"></p><p>(5) 在新的终端窗口中，执行如下命令，登录PolarDB-X数据库。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">mysql</span> -h<span class="hljs-number">127.0.0.1</span> -P8527 -upolardbx_root -p123456<br></code></pre></td></tr></table></figure><p>(6) 执行如下SQL语句，使用数据库testdb。</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">USE testdb<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p>(7) 执行如下SQL语句，插入一条数据。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span> test(<span class="hljs-type">name</span>) <span class="hljs-keyword">values</span>("polardb-x"), ("is"), ("awsome");<br></code></pre></td></tr></table></figure><p>(8) 输入exit退出数据库。</p><p>(9) 执行如下命令，登录ClickHouse数据库。</p><figure class="highlight axapta"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs axapta">docker run -it --rm --link some-clickhouse-<span class="hljs-keyword">server</span>:clickhouse-<span class="hljs-keyword">server</span> yandex/clickhouse-<span class="hljs-keyword">client</span> --host clickhouse-<span class="hljs-keyword">server</span><br></code></pre></td></tr></table></figure><p>(10) 执行如下SQL语句，使用数据库testdb。</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">USE testdb<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p>(11) 执行如下SQL语句，查询test表</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">SELECT</span> * <span class="hljs-keyword">FROM</span> test;<br></code></pre></td></tr></table></figure><p>返回结果如下，您可以看到目标端（ClickHouse）接收到投递过来的数据。</p><p><img src="https://ucc.alicdn.com/pic/developer-ecology/4aed414de5954f2da0849c14764f1e0f.png" alt="img"></p>]]></content>
<categories>
<category>数据库</category>
<category>PolarDB-X</category>
</categories>
<tags>
<tag>PolarDB-X</tag>
<tag>分布式数据库</tag>
</tags>
</entry>
<entry>
<title>PolarDB-X部署</title>
<link href="/2022/06/09/PolarDB-X%E9%83%A8%E7%BD%B2/"/>
<url>/2022/06/09/PolarDB-X%E9%83%A8%E7%BD%B2/</url>
<content type="html"><![CDATA[<p>本步骤将指导您如何安装并启动依赖Docker。</p><p>执行如下命令，安装Docker。</p><figure class="highlight livecodeserver"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs livecodeserver">curl -fsSL <span class="hljs-keyword">https</span>://<span class="hljs-built_in">get</span>.docker.com | bash -s docker <span class="hljs-comment">--mirror Aliyun</span><br></code></pre></td></tr></table></figure><p>执行如下命令，启动Docker。</p><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs crmsh">systemctl <span class="hljs-literal">start</span> docker<br></code></pre></td></tr></table></figure><p>本步骤将指导您如何使用Docker镜像安装PolarDB-X。</p><p>执行如下命令，拉取PolarDB-X容器镜像。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">docker pull polardbx/polardb-x<br></code></pre></td></tr></table></figure><ol><li>执行如下命令，运行PolarDB-X容器。</li></ol><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">docker</span> run -d --name some-polardb-x -p <span class="hljs-number">8527</span>:<span class="hljs-number">8527</span> polardbx/polardb-x<br></code></pre></td></tr></table></figure><p>本步骤将指导您如何登录PolarDB-X并体验PolarDB-X的分布式特性。</p><p>PolarDB-X支持通过MySQL Client命令行、第三方客户端以及符合MySQL交互协议的第三方程序代码进行连接。本实验场景主要介绍如何通过MySQL Client命令行连接到PolarDB-X数据库。</p><p>执行如下命令，安装MySQL。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">yum <span class="hljs-keyword">install</span> mysql -y<br></code></pre></td></tr></table></figure><p>执行如下命令，查看MySQL版本号。</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">mysql -V</span><br></code></pre></td></tr></table></figure><p>返回结果如下，表示您已成功安装MySQL。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636919.png" alt="img"></p><p>执行如下命令，登录PolarDB-X数据库。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">mysql</span> -h<span class="hljs-number">127.0.0.1</span> -P8527 -upolardbx_root -p123456<br></code></pre></td></tr></table></figure><p><strong>说明</strong>：</p><p>本实验场景中的PolarDB-X数据库用户名和密码已预设，请您使用下方命令登录即可。</p><p>如遇到mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0报错，重新执行登录命令即可。</p><p>执行如下SQL语句，检查GMS。</p><figure class="highlight axapta"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs axapta"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> information_schema.schemata;<br></code></pre></td></tr></table></figure><p>体验PolarDB-X数据库的分布式特性。</p><p>执行如下SQL语句，创建数据库。</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">create database polarx_example mode</span>=<span class="hljs-string">'auto'</span>; <br></code></pre></td></tr></table></figure><p>执行如下SQL语句，使用polarx_example数据库。</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">use polarx_example<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p>执行如下SQL语句，创建数据表。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">create</span> <span class="hljs-keyword">table</span> example (<br> `id` <span class="hljs-type">bigint</span>(<span class="hljs-number">11</span>) auto_increment <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span>,<br> `<span class="hljs-type">name</span>` <span class="hljs-type">varchar</span>(<span class="hljs-number">255</span>) <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span>,<br> `score` <span class="hljs-type">bigint</span>(<span class="hljs-number">11</span>) <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span>,<br> <span class="hljs-keyword">primary key</span> (`id`)<br>) engine=InnoDB <span class="hljs-keyword">default</span> charset=utf8 <br><span class="hljs-keyword">partition by hash</span>(id) <br>partitions <span class="hljs-number">8</span>;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，向example数据表中插入数据。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">insert</span> <span class="hljs-keyword">into</span> example <span class="hljs-keyword">values</span>(<span class="hljs-literal">null</span>,<span class="hljs-string">'lily'</span>,<span class="hljs-number">375</span>),(<span class="hljs-literal">null</span>,<span class="hljs-string">'lisa'</span>,<span class="hljs-number">400</span>),(<span class="hljs-literal">null</span>,<span class="hljs-string">'ljh'</span>,<span class="hljs-number">500</span>);<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，查询example表所有数据。</p><figure class="highlight axapta"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs axapta"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> example;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，查看example数据表的分区。</p><figure class="highlight maxima"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs maxima"><span class="hljs-built_in">show</span> topology from <span class="hljs-built_in">example</span>;<br></code></pre></td></tr></table></figure><p>返回结果如下，您可以看到example数据表分布在8个分区。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636508.png" alt="img"></p><p>执行如下SQL语句，检查CDC。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">show</span> master status;<br><span class="hljs-keyword">show</span> binlog events <span class="hljs-keyword">in</span> <span class="hljs-string">'binlog.000001'</span> <span class="hljs-keyword">from</span> <span class="hljs-number">4</span>;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，检查DN和CN。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">show</span> <span class="hljs-keyword">storage</span>; <br><span class="hljs-keyword">show</span> mpp;<br></code></pre></td></tr></table></figure><p>输入exit退出数据库。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636746.png" alt="img"></p><p>执行如下命令，停止PolarDB-X容器。</p><figure class="highlight ada"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ada">docker stop <span class="hljs-keyword">some</span>-polardb-x<br></code></pre></td></tr></table></figure><p>本步骤将指导您如何使用PXD工具一键安装PolarDB-X，详细文档参考<a href="https://doc.polardbx.com/quickstart/topics/quickstart-pxd-cluster.html">使用PXD工具一键安装PolarDB-X</a>。</p><p>安装 PXD</p><p>PXD是PolarDB-X的部署工具，除了支持在本地一键快速拉起测试环境外，也支持在Linux集群中通过指定的拓扑的方式部署PolarDB-X分布式数据库。</p><p><strong>说明</strong>：本实验场景使用virtual environment安装PXD工具。</p><p>执行如下命令，创建并激活虚拟场景。</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs gradle">python3 -m venv venv<br><span class="hljs-keyword">source</span> venv<span class="hljs-regexp">/bin/</span>activate<br></code></pre></td></tr></table></figure><p>执行如下命令，升级pip。</p><figure class="highlight ada"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ada">pip install <span class="hljs-comment">--upgrade pip</span><br></code></pre></td></tr></table></figure><p>执行如下命令，安装PXD。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> pxd<br></code></pre></td></tr></table></figure><p>部署PolarDB-X。</p><p><strong>说明</strong>：部署PolarDB-X有两种方式，请您选择其中一种方式即可。</p><ul><li>方式一：执行如下命令，创建一个PolarDB-X数据库，其中GMS、CN、DN和CDC节点个数为1。</li></ul><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">pxd tryout</span><br></code></pre></td></tr></table></figure><ul><li>方式二：执行如下命令，创建一个PolarDB-X数据库，指定CN、DN和CDC节点个数为1以及版本为latest。</li></ul><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">pxd</span> tryout -cn_replica <span class="hljs-number">1</span> -cn_version latest -dn_replica <span class="hljs-number">1</span> -dn_version latest -cdc_replica <span class="hljs-number">1</span> -cdc_version latest<br></code></pre></td></tr></table></figure><p>返回结果如下，表示您已成功部署PolarDB-X数据库，您可以看到输出的连接信息，通过MySQL Client即可连接。</p><p><strong>注意</strong>：PolarDB-X管理员账号的密码随机生成，仅出现这一次，请注意保存。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636752.png" alt="img"></p><p>执行如下命令，登录PolarDB-X数据库。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">mysql</span> -h<span class="hljs-number">127.0.0.1</span> -P8527 -upolardbx_root -p123456<br></code></pre></td></tr></table></figure><p>体验PolarDB-X。</p><p>执行如下SQL语句，检查GMS。</p><figure class="highlight axapta"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs axapta"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> information_schema.schemata;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，创建数据库。</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">create database polarx_example mode</span>=<span class="hljs-string">'auto'</span>; <br></code></pre></td></tr></table></figure><p>执行如下SQL语句，使用polarx_example数据库。sd</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">use polarx_example<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p>执行如下SQL语句，创建数据表。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">create</span> <span class="hljs-keyword">table</span> example (<br> `id` <span class="hljs-type">bigint</span>(<span class="hljs-number">11</span>) auto_increment <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span>,<br> `<span class="hljs-type">name</span>` <span class="hljs-type">varchar</span>(<span class="hljs-number">255</span>) <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span>,<br> `score` <span class="hljs-type">bigint</span>(<span class="hljs-number">11</span>) <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span>,<br> <span class="hljs-keyword">primary key</span> (`id`)<br>) engine=InnoDB <span class="hljs-keyword">default</span> charset=utf8 <br><span class="hljs-keyword">partition by hash</span>(id) <br>partitions <span class="hljs-number">8</span>;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，向example数据表中插入数据。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">insert</span> <span class="hljs-keyword">into</span> example <span class="hljs-keyword">values</span>(<span class="hljs-literal">null</span>,<span class="hljs-string">'lily'</span>,<span class="hljs-number">375</span>),(<span class="hljs-literal">null</span>,<span class="hljs-string">'lisa'</span>,<span class="hljs-number">400</span>),(<span class="hljs-literal">null</span>,<span class="hljs-string">'ljh'</span>,<span class="hljs-number">500</span>);<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，查询example表所有数据。</p><figure class="highlight axapta"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs axapta"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> example;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，查看example数据表的分区。</p><figure class="highlight maxima"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs maxima"><span class="hljs-built_in">show</span> topology from <span class="hljs-built_in">example</span>;<br></code></pre></td></tr></table></figure><p>返回结果如下，您可以看到example数据表分布在8个分区。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636496.png" alt="img"></p><p>执行如下SQL语句，检查CDC。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">show</span> master status;<br><span class="hljs-keyword">show</span> binlog events <span class="hljs-keyword">in</span> <span class="hljs-string">'binlog.000001'</span> <span class="hljs-keyword">from</span> <span class="hljs-number">4</span>;<br></code></pre></td></tr></table></figure><p>执行如下SQL语句，检查DN和CN。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">show</span> <span class="hljs-keyword">storage</span>; <br><span class="hljs-keyword">show</span> mpp;<br></code></pre></td></tr></table></figure><p>输入exit退出数据库。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636486.png" alt="img"></p><p>查看PolarDB-X状态。</p><p>执行如下命令，查看当前环境的PolarDB-X状态。</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">pxd list</span><br></code></pre></td></tr></table></figure><p>返回如下结果，您可查看到PolarDB-X状态。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636576.png" alt="img"></p><p>清理PolarDB-X。</p><p>执行如下命令，清理本地环境所有的PolarDB-X</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">pxd cleanup</span><br></code></pre></td></tr></table></figure><p>本步骤将指导您如何创建一个简单的Kubernetes集群并部署PolarDB-X Operator ，使用Operator部署一个完整的PolarDB-X集群，详细文档请参考<a href="https://doc.polardbx.com/quickstart/topics/quickstart-k8s.html">通过</a>。</p><p>安装kubectl。</p><p>执行如下命令，下载kubectl文件。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl<br></code></pre></td></tr></table></figure><p>执行如下命令，赋予可执行权限。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">chmod +x ./kubectl<br></code></pre></td></tr></table></figure><p>执行如下命令，移动到系统目录。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">mv ./kubectl /usr/local/bin/kubectl<br></code></pre></td></tr></table></figure><p>安装minikube。</p><p>执行如下命令，下载并安装minikube。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64<br>sudo install minikube-linux-amd64 /usr/local/bin/minikube<br></code></pre></td></tr></table></figure><p>安装Helm3。</p><p>执行如下命令，下载Helm3。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">wget https://labfileapp.oss-cn-hangzhou.aliyuncs.com/helm-v3.9.0-linux-amd64.tar.gz<br></code></pre></td></tr></table></figure><p>执行如下命令，解压Helm3。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">tar -zxvf helm-v3.9.0-linux-amd64.tar.gz<br></code></pre></td></tr></table></figure><p>执行如下命令，移动到系统目录。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">mv linux-amd64/helm /usr/local/bin/helm<br></code></pre></td></tr></table></figure><p>使用minikube创建Kubernetes集群。</p><p>minikube是由社区维护的用于快速创建Kubernetes测试集群的工具，适合测试和学习Kubernetes。使用minikube创建的Kubernetes集群可以运行在容器或是虚拟机中，本实验场景以CentOS 8.5上创建Kubernetes为例。</p><p><strong>说明</strong>：如您使用其他操作系统部署minikube，例如macOS或Windows，部分步骤可能略有不同。</p><p>执行如下命令，新建账号galaxykube，并将galaxykube加入docker组中。minikube要求使用非root账号进行部署，所有您需要新建一个账号。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">useradd -ms /bin/bash galaxykube <br>usermod -aG docker galaxykube <br></code></pre></td></tr></table></figure><p>执行如下命令，切换到账号galaxykube。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">su galaxykube<br></code></pre></td></tr></table></figure><p>执行如下命令，进入到home/galaxykube目录。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">cd<br></code></pre></td></tr></table></figure><p>执行如下命令，启动一个minikube。</p><p><strong>说明</strong>：这里我们使用了阿里云的minikube镜像源以及USTC提供的docker镜像源来加速镜像的拉取。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">minikube start --cpus 4 --memory 7168 --image-mirror-country cn --registry-mirror=https://docker.mirrors.ustc.edu.cn<br></code></pre></td></tr></table></figure><p>返回结果如下，表示minikube已经正常运行，minikube将自动设置kubectl的配置文件。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011636646.png" alt="img"></p><p>执行如下命令，使用kubectl查看集群信息。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">minikube kubectl -- cluster-info<br></code></pre></td></tr></table></figure><p>返回如下结果，您可以查看到集群相关信息.</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011637433.png" alt="img"></p><p>部署 PolarDB-X Operator。</p><p>执行如下命令，创建一个名为polardbx-operator-system的命名空间。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl create namespace polardbx-operator-system<br></code></pre></td></tr></table></figure><p>执行如下命令，安装PolarDB-X Operator。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">helm install --namespace polardbx-operator-system polardbx-operator https://github.com/ApsaraDB/galaxykube/releases/download/v1.1.0/polardbx-operator-1.1.0.tgz<br></code></pre></td></tr></table></figure><p>执行如下命令，查看PolarDB-X Operator组件的运行情况。</p><figure class="highlight arduino"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs arduino">kubectl get pods --<span class="hljs-keyword">namespace</span> polardbx-<span class="hljs-keyword">operator</span>-system<br></code></pre></td></tr></table></figure><p>返回结果如下，请您耐心等待2分钟，等待所有组件都进入Running状态，表示PolarDB-X Operator已经安装完成。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011637508.png" alt="img"></p><p>部署 PolarDB-X 集群。</p><p>执行如下命令，部署一个PolarDB-X集群，它包含1个GMS 节点、1个 CN节点、1个DN节点和1 个CDC节点。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">echo "apiVersion: polardbx.aliyun.com/v1<br>kind: PolarDBXCluster<br>metadata:<br> name: quick-start<br> annotations:<br> polardbx/topology-mode-guide: quick-start" | kubectl apply -f -<br></code></pre></td></tr></table></figure><p>执行如下命令，查看创建状态。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">kubectl get polardbxcluster -w<br></code></pre></td></tr></table></figure><p>返回结果如下，请您耐心等待大约七分钟，当PHASE显示为Running时，表示PolarDB-X集群已经部署完成。现在您可以开始连接并体验PolarDB-X分布式数据库了。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011637419.png" alt="img"></p><p>本步骤将指导您如何使用源码编译安装PolarDB-X，详细文档请详见<a href="https://doc.polardbx.com/quickstart/topics/quickstart-development.html">源码编译安装PolarDB-X</a>。</p><p>执行如下命令，安装git。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">yum -y <span class="hljs-keyword">install</span> git<br></code></pre></td></tr></table></figure><p>执行如下命令，下载源码。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">git clone https:<span class="hljs-regexp">//gi</span>thub.com<span class="hljs-regexp">/ApsaraDB/</span>PolarDB-X.git<br></code></pre></td></tr></table></figure><p>执行如下命令，进入PolarDB-X目录。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">cd</span> PolarDB-X<br></code></pre></td></tr></table></figure><p>执行如下命令，编译。</p><p><strong>说明</strong>：make将下载所有源码、工具和库，并构建和安装PolarDB-X。源码将下载到./build目录下，编译好的二进制文件将安装到./build/run目录下。您可以运行make clean来移除安装，并尝试再次构建PolarDB-X。您也可以运行make cleanAll来删除./build下的所有东西。这个过程可能需要半个多小时，取决于您的网络带宽。</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">make</span><br></code></pre></td></tr></table></figure><p>执行如下命令，运行PolarDB-X。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">.<span class="hljs-regexp">/build/</span>run<span class="hljs-regexp">/bin/</span>polardb-x.sh start<br></code></pre></td></tr></table></figure><p>执行如下命令，停止PolarDB-X。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">.<span class="hljs-regexp">/build/</span>run<span class="hljs-regexp">/bin/</span>polardb-x.sh stop<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>数据库</category>
<category>PolarDB-X</category>
</categories>
<tags>
<tag>PolarDB-X</tag>
<tag>分布式数据库</tag>
</tags>
</entry>
<entry>
<title>otter使用介绍</title>
<link href="/2022/06/09/otter%E4%BD%BF%E7%94%A8%E4%BB%8B%E7%BB%8D/"/>
<url>/2022/06/09/otter%E4%BD%BF%E7%94%A8%E4%BB%8B%E7%BB%8D/</url>
<content type="html"><![CDATA[<h3 id="otter"><a href="#otter" class="headerlink" title="otter"></a>otter</h3><h4 id="产品介绍"><a href="#产品介绍" class="headerlink" title="产品介绍"></a>产品介绍</h4><p>开源仓地址：<a href="https://github.com/alibaba/otter">https://github.com/alibaba/otter</a></p><p>基于数据库增量日志解析，准实时同步到本机房或异地机房的mysql/oracle数据库. 一个分布式数据库同步系统</p><p>官放介绍(入门)ppt： <a href="https://cloud.189.cn/t/Uzq6reEZ3A3m%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Aduq6%EF%BC%89">https://cloud.189.cn/t/Uzq6reEZ3A3m（访问码：duq6）</a></p><p>官放介绍(进阶)ppt：<a href="https://cloud.189.cn/t/RBraMnFz2Abi%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9A0g2l%EF%BC%89">https://cloud.189.cn/t/RBraMnFz2Abi（访问码：0g2l）</a></p><p>整体架构</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206080931156.jpeg" alt="img"></p><p>原理描述：</p><ol><li><p>基于Canal开源产品，获取数据库增量日志数据。 什么是Canal, 请<a href="https://github.com/alibaba/canal">点击</a></p></li><li><p>典型管理系统架构，manager(web管理)+node(工作节点)</p></li></ol><p> a. manager运行时推送同步配置到node节点</p><p> b. node节点将同步状态反馈到manager上</p><ol start="3"><li>基于zookeeper，解决分布式状态调度的，允许多node节点之间协同工作.</li></ol><h4 id="使用场景"><a href="#使用场景" class="headerlink" title="使用场景"></a>使用场景</h4><p>1.多实例数据源、小数据量实时同步（避免重复购买DTS实时同步）；</p><p>2.仅实验mysql-mysql同步，其他同步未测试。</p><h4 id="部署方案"><a href="#部署方案" class="headerlink" title="部署方案"></a>部署方案</h4><p>建议部署：</p><p>otter服务1台机器</p><p>zookeeper，最好单独部署</p><p>node单独部署，manager端可部署一个node， 承载计算公式：</p><p>数据库连接均与正式环境数据库在一个VPC下，使用内网链接</p><table><thead><tr><th>端口用途描述</th><th>默认端口号</th><th>添加方式</th></tr></thead><tbody><tr><td>Web管理端</td><td>8080</td><td>部署机外网出</td></tr><tr><td>zookeeper</td><td>2181</td><td>部署机外网出</td></tr></tbody></table><h4 id="使用指南"><a href="#使用指南" class="headerlink" title="使用指南"></a>使用指南</h4><p>官方开源仓wiki：<a href="https://github.com/alibaba/otter/wiki">https://github.com/alibaba/otter/wiki</a></p><p>介绍很详细，在这里就不cpoy过来了，这边举例几个实际使用场景，及大数据量同步场景稳定性测试结果；</p><p>前提准备：</p><p>登录系统后，按照命名规范新建条目（不然后面越加越乱）</p><p>Zookeeper注册，集群名称命名规则：otter_zookeeper_1、otter_zookeeper_2</p><p>Node注册，集群名称命名规则：otter_node_1、otter_node_2</p><p>数据源添加，命名规则：项目群名称_实例命名@账号名称</p><p>数据库表添加，按实际库表名称展示</p><p>cannal添加，命名规则：otter_canal_实例/项目群代码缩写@业务描述，同样的数据可做多份canal，即同源多分发需建多个canal</p><p>channal添加，命名规则：项目名称_业务描述，新增表同步后加同名channal并增加下划线区分（）等同于阿里DTS的同步任务</p><p>注意：<font color=red>目标表支持ADB，使用mysql驱动，但是ADB端需手动建表（按照数仓开发规范，需审核使用方法正确性）</font></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206081040253.png" alt="image-20220608104050022"></p><p>Pipeline添加，命名规范：单向传输即可，同一channal不能对应多个Pipeline</p><h4 id="新增表同步实验："><a href="#新增表同步实验：" class="headerlink" title="新增表同步实验："></a>新增表同步实验：</h4><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs sql">#创建临时表<br><span class="hljs-keyword">DROP</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">EXISTS</span> tb_aaa;<br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> tb_aaa(<br>Id <span class="hljs-type">bigint</span>(<span class="hljs-number">20</span>) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT,<br><span class="hljs-keyword">Value</span> <span class="hljs-type">int</span>,<br><span class="hljs-keyword">PRIMARY</span> KEY (Id)<br>);<br><br><span class="hljs-keyword">DROP</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">EXISTS</span> tb_bbb;<br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> tb_bbb(<br>Id <span class="hljs-type">bigint</span>(<span class="hljs-number">20</span>) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT,<br><span class="hljs-keyword">Value</span> <span class="hljs-type">int</span>,<br><span class="hljs-keyword">PRIMARY</span> KEY (Id)<br>);<br></code></pre></td></tr></table></figure><h5 id="新建数据源"><a href="#新建数据源" class="headerlink" title="新建数据源"></a>新建数据源</h5><p>表名称支持正则表达式，或者直接填写.*添加全部表</p><p>注意：只读数据源需使用 _bl账号，写数据源需使用有写权限账号，最好是user_dts。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071628517.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071627118.png"></p><h5 id="建立源表目标表"><a href="#建立源表目标表" class="headerlink" title="建立源表目标表"></a>建立源表目标表</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071615817.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071617843.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071618904.png"></p><h5 id="建立中途添加源表"><a href="#建立中途添加源表" class="headerlink" title="建立中途添加源表"></a>建立中途添加源表</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071619020.png"></p><h5 id="新建channel"><a href="#新建channel" class="headerlink" title="新建channel"></a>新建channel</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071707923.png"></p><p>有需要指定时间点和binlog位置点为起点时，选中位点自定义设置，填写</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">show</span> master status;<br><span class="hljs-keyword">select</span> unix_timestamp();<br>#得出最终填写参数<br>{"journalName":"mysql-bin.003047","position":<span class="hljs-number">60293232</span>,"timestamp":<span class="hljs-number">1654594309</span>};<br></code></pre></td></tr></table></figure><p>canal具体参数含义见官方wiki文档。</p><h5 id="建立同步channel"><a href="#建立同步channel" class="headerlink" title="建立同步channel"></a>建立同步channel</h5><p>直接同步管理-左下角，添加</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071709319.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071710240.png"></p><p>具体参数可见官方wiki文档。</p><h5 id="点击Channel名字建立Pipeline"><a href="#点击Channel名字建立Pipeline" class="headerlink" title="点击Channel名字建立Pipeline"></a>点击Channel名字建立Pipeline</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071711227.png"></p><p>高级设置里参数解释详见官方wiki，注意需开启 支持ddl同步选项</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071712612.png"></p><p>注意：每个canal只能提供给一个Pipeline。</p><h5 id="点击Pipeline名称添加同步映射关系表"><a href="#点击Pipeline名称添加同步映射关系表" class="headerlink" title="点击Pipeline名称添加同步映射关系表"></a>点击Pipeline名称添加同步映射关系表</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071714889.png"></p><p>下一步，可编辑字段对应关系及修改字段名称</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071715439.png"></p><h5 id="回到Channel管理界面，点击启动"><a href="#回到Channel管理界面，点击启动" class="headerlink" title="回到Channel管理界面，点击启动"></a>回到Channel管理界面，点击启动</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071733170.png"></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs sql"><br>#建立同步任务后写入数据<br><span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span> tb_aaa(<span class="hljs-keyword">Value</span>)<br><span class="hljs-keyword">SELECT</span> <span class="hljs-number">1</span><br><span class="hljs-keyword">UNION</span><br><span class="hljs-keyword">SELECT</span> <span class="hljs-number">2</span>;<br><br>#查看目标表<br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <br><span class="hljs-keyword">FROM</span> tb_bbb;<br></code></pre></td></tr></table></figure><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071733590.png"></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs sql">#源表新增字段<br><span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">TABLE</span> tb_aaa<br><span class="hljs-keyword">ADD</span> <span class="hljs-keyword">COLUMN</span> AddColumn <span class="hljs-type">int</span> <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> <span class="hljs-keyword">DEFAULT</span> <span class="hljs-number">1024</span> AFTER `<span class="hljs-keyword">Value</span>`<br>;<br><br>#查看目标表<br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span><br><span class="hljs-keyword">FROM</span> tb_bbb;<br></code></pre></td></tr></table></figure><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071735058.png"></p><h5 id="新加表同步，暂停channel，添加表映射（这里变更下目标表的表名和字段名）"><a href="#新加表同步，暂停channel，添加表映射（这里变更下目标表的表名和字段名）" class="headerlink" title="新加表同步，暂停channel，添加表映射（这里变更下目标表的表名和字段名）"></a>新加表同步，暂停channel，添加表映射（这里变更下目标表的表名和字段名）</h5><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">DROP</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">EXISTS</span> tb_ccc;<br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> tb_ccc(<br>Id <span class="hljs-type">bigint</span>(<span class="hljs-number">20</span>) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT,<br>ValueC <span class="hljs-type">int</span>,<br><span class="hljs-keyword">PRIMARY</span> KEY (Id)<br>);<br><br><span class="hljs-keyword">DROP</span> <span class="hljs-keyword">TABLE</span> IF <span class="hljs-keyword">EXISTS</span> tb_ddd;<br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> tb_ddd(<br>Id <span class="hljs-type">bigint</span>(<span class="hljs-number">20</span>) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT,<br>ValueD <span class="hljs-type">int</span>,<br><span class="hljs-keyword">PRIMARY</span> KEY (Id)<br>);<br></code></pre></td></tr></table></figure><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071741234.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071740263.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071741950.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071746191.png"></p><h5 id="写入源表数据"><a href="#写入源表数据" class="headerlink" title="写入源表数据"></a>写入源表数据</h5><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span> tb_aaa(<span class="hljs-keyword">Value</span>)<br><span class="hljs-keyword">SELECT</span> <span class="hljs-number">3</span><br><span class="hljs-keyword">UNION</span><br><span class="hljs-keyword">SELECT</span> <span class="hljs-number">4</span>;<br></code></pre></td></tr></table></figure><h5 id="回到Channel管理，开启同步"><a href="#回到Channel管理，开启同步" class="headerlink" title="回到Channel管理，开启同步"></a>回到Channel管理，开启同步</h5><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071750752.png"></p><p>可以看到刚才开启同步之前写入的数据已经在同步记录里了<br><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071751064.png"></p><p>也就是说canal支持暂停同步，添加表后再开启同步</p><p><font color=red>暂停期间不支持DDL，不能修改表结构，不然重启任务后会报错，暂停期间需要占用很大ottermanager部署机器的内存空间，需短时间内完成表添加</font></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs sql">#查看目标表<br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span><br><span class="hljs-keyword">FROM</span> tb_bbb<br><span class="hljs-keyword">WHERE</span> <span class="hljs-keyword">Value</span> <span class="hljs-keyword">IN</span> (<span class="hljs-number">3</span>,<span class="hljs-number">4</span>);<br></code></pre></td></tr></table></figure><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071753703.png"></p><h5 id="验证新加的表数据是否能够完成同步任务"><a href="#验证新加的表数据是否能够完成同步任务" class="headerlink" title="验证新加的表数据是否能够完成同步任务"></a>验证新加的表数据是否能够完成同步任务</h5><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs sql">#查看目标新增表<br><span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span> tb_ccc(ValueC)<br><span class="hljs-keyword">SELECT</span> <span class="hljs-number">1</span><br><span class="hljs-keyword">UNION</span><br><span class="hljs-keyword">SELECT</span> <span class="hljs-number">2</span>;<br><br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span><br><span class="hljs-keyword">FROM</span> tb_ddd;<br></code></pre></td></tr></table></figure><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206080926045.png"></p><p><font color=red>字段名称对应关系变更一定要放到下面的方框内</font></p><p>3.大数据量变更稳定性测试</p><p>案例1：备案中心设备状态表</p><p>变更数据量：</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071049049.png"></p><p>平均延迟时间</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071046778.png"></p><p>历史吞吐量</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206071050819.png"></p><p>同步稳定运行半个月</p><p>案例2：客户中心配置信息实时同步，稳定运行</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206081008418.png" alt="image-20220608100822238"></p><p>4.预警配置</p><p>系统管理菜单下，点击系统参数设置</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206080947691.png"></p><p>添加预警组后，回到Pipeline管理界面添加监控规则</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206080948261.png"></p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206080951047.png" alt="image-20220608095144721"></p><p>5.实时数据初始化方案</p><p>方式1：<br>otter本身支持自动以数据同步(自由门)，增量同步需用行记录模式，</p><p>主要原理：</p><p>a. 基于otter系统表retl_buffer，插入特定的数据，包含需要同步的表名，pk信息。</p><p>b. otter系统感知后会根据表名和pk提取对应的数据(整行记录)，和正常的增量同步一起同步到目标库。</p><p>目前otter系统感知的自 由 门数据方式为：</p><ul><li>日志记录. (插入表数据的每次变更，需要开启binlog，otter获取binlog数据，提取同步的表名，pk信息，然后回表查询整行记录)</li><li></li></ul><p>方式2：</p><p>使用DATAX配合，先开otter启实时同步稳定后，再开启部署好的DATAX离线同步，待离线完成后，恢复otter同步。（保障otter组件canal记录的时间点在DATAX开启时间点之前即可，还要保障日志都能够读到(未释放)）</p><p><font color=red>目标表必需有主键，DATAX writer使用replace方式写入</font></p><h4 id="FAQ"><a href="#FAQ" class="headerlink" title="FAQ"></a>FAQ</h4><p>官方列举出的常见问题：<a href="https://github.com/alibaba/otter/wiki/Faq">https://github.com/alibaba/otter/wiki/Faq</a></p><p>网络资料：</p><p>Otter的限制</p><p>1 otter只支持ROW模式的数据同步，其他两种模式不支持<br>2 源库只支持mysql,目标库支持mysql和oracle<br>3 同步的表必须要有主键，无主键表update会是一个全表扫描，效率比较差),就是全字段匹配,如果出现重复记录的话,同步会导致数据错乱<br>4 支持部分ddl同步 (支持create table / drop table / alter table / truncate table / rename table / create index / drop index，其他类型的暂不支持，比如grant,create user,trigger等等)，同时ddl语句不支持幂等性操作，所以出现重复同步时，会导致同步挂起，可通过配置高级参数:跳过ddl异常，来解决这个问题<br>5 otter ddl同步不支持异构处理，库名、表名都要求一致<br>6 不支持带外键的记录同步. (数据载入算法会打散事务，进行并行处理，会导致外键约束无法满足)<br>7 数据库上trigger配置慎重. (比如源库，有一张A表配置了trigger，将A表上的变化记录到B表中，而B表也需要同步。如果目标库也有这trigger，在同步时会插入一次A表，2次B表，因为A表的同步插入也会触发trigger插入一次B表，所以有2次B表同步.)<br>8 2个manager,可以部署2个manager,manager之间本身没有通信,而是通过zk和数据库,配置node的时候需要知道manager,可以指定其中一个即可,数据会反应到数据库和zk中,但是当这个manager挂了,新配置的node信息就不能反馈到另外一个manager上,所以最佳实践是指定所有的manage</p><p>常见故障处理</p><p>1 异构环境下（源与目标库名表名有一个不相同），主库执行DDL导致同步挂起，此时可跳过DDL同步进行恢复，将源库执行的DDL SQL手动执行到目标库。<br>2 网络中断导致同步挂起，此时可重启服务器对应node节点进行恢复。<br>3 其他问题，可通过修改canal post位点或者重新同步进行解决。<br>4 Channel挂起或暂停已超源实例MySQLbinlog清理时间，可初始化其同步进度重新从某时间点开始继续同步，但需要做一次离线同步。</p><p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202207011651905.png"></p>]]></content>
<categories>
<category>ETL工具</category>
<category>实时同步</category>
</categories>
<tags>
<tag>ETL</tag>
<tag>otter</tag>
<tag>canal</tag>
</tags>
</entry>
<entry>
<title>TDEngine应用测试</title>
<link href="/2022/06/02/TDEngine%E5%BA%94%E7%94%A8%E6%B5%8B%E8%AF%95/"/>
<url>/2022/06/02/TDEngine%E5%BA%94%E7%94%A8%E6%B5%8B%E8%AF%95/</url>
<content type="html"><![CDATA[<h2 id="应用1-壹好车服定位数据"><a href="#应用1-壹好车服定位数据" class="headerlink" title="应用1 壹好车服定位数据"></a>应用1 壹好车服定位数据</h2><h4 id="新增专用账号"><a href="#新增专用账号" class="headerlink" title="新增专用账号"></a>新增专用账号</h4><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">USER</span> root PASS <span class="hljs-string">'Lunz2017'</span>;<br><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">USER</span> user_service PASS <span class="hljs-string">'Lunz2017'</span>;<br><span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">USER</span> user_service PRIVILEGE write;<br></code></pre></td></tr></table></figure><h4 id="建库"><a href="#建库" class="headerlink" title="建库"></a>建库</h4><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> DATABASE IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> locationcenter <span class="hljs-keyword">UPDATE</span> <span class="hljs-number">2</span>;<br></code></pre></td></tr></table></figure><p>参数解析</p><blockquote><h5 id="INFO"><a href="#INFO" class="headerlink" title="INFO"></a>INFO</h5><ol><li><p>KEEP 是该数据库的数据保留多长天数，缺省是 3650 天(10 年)，数据库会自动删除超过时限的数据；</p></li><li><p>UPDATE 标志数据库支持更新相同时间戳数据；（从 2.1.7.0 版本开始此参数支持设为 2，表示允许部分列更新，也即更新数据行时未被设置的列会保留原值。）（从 2.0.8.0 版本开始支持此参数。注意此参数不能通过<code>ALTER DATABASE</code>指令进行修改。）</p><ol><li>UPDATE 设为 0 时，表示不允许更新数据，后发送的相同时间戳的数据会被直接丢弃；</li><li>UPDATE 设为 1 时，表示更新全部列数据，即如果更新一个数据行，其中某些列没有提供取值，那么这些列会被设为 NULL；</li><li>UPDATE 设为 2 时，表示支持更新部分列数据，即如果更新一个数据行，其中某些列没有提供取值，那么这些列会保持原有数据行中的对应值；</li><li>更多关于 UPDATE 参数的用法，请参考<a href="https://docs.taosdata.com/train-faq/faq">FAQ</a>。</li></ol></li><li><p>数据库名最大长度为 33；</p></li><li><p>一条 SQL 语句的最大长度为 65480 个字符；</p></li><li><p>创建数据库时可用的参数有：</p><ul><li>cache: <a href="https://docs.taosdata.com/reference/config/#cache">详细说明</a></li><li>blocks: <a href="https://docs.taosdata.com/reference/config/#blocks">详细说明</a></li><li>days: <a href="https://docs.taosdata.com/reference/config/#days">详细说明</a></li><li>keep: <a href="https://docs.taosdata.com/reference/config/#keep">详细说明</a></li><li>minRows: <a href="https://docs.taosdata.com/reference/config/#minrows">详细说明</a></li><li>maxRows: <a href="https://docs.taosdata.com/reference/config/#maxrows">详细说明</a></li><li>wal: <a href="https://docs.taosdata.com/reference/config/#wallevel">详细说明</a></li><li>fsync: <a href="https://docs.taosdata.com/reference/config/#fsync">详细说明</a></li><li>update: <a href="https://docs.taosdata.com/reference/config/#update">详细说明</a></li><li>cacheLast: <a href="https://docs.taosdata.com/reference/config/#cachelast">详细说明</a></li><li>replica: <a href="https://docs.taosdata.com/reference/config/#replica">详细说明</a></li><li>quorum: <a href="https://docs.taosdata.com/reference/config/#quorum">详细说明</a></li><li>maxVgroupsPerDb: <a href="https://docs.taosdata.com/reference/config/#maxvgroupsperdb">详细说明</a></li><li>comp: <a href="https://docs.taosdata.com/reference/config/#comp">详细说明</a></li><li>precision: <a href="https://docs.taosdata.com/reference/config/#precision">详细说明</a></li></ul></li><li><p>请注意上面列出的所有参数都可以配置在配置文件 <code>taosd.cfg</code> 中作为创建数据库时使用的默认配置， <code>create database</code> 的参数中明确指定的会覆盖配置文件中的设置。</p></li></ol></blockquote><p>显示当前参数</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SHOW</span> VARIABLES;<br></code></pre></td></tr></table></figure><p>设置压缩值</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> DATABASE locationcenter COMP <span class="hljs-number">2</span>;<br></code></pre></td></tr></table></figure><blockquote><p>COMP 参数是指修改数据库文件压缩标志位，缺省值为 2，取值范围为 [0, 2]。0 表示不压缩，1 表示一阶段压缩，2 表示两阶段压缩。</p></blockquote><p>设置副本数</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> DATABASE locationcenter REPLICA <span class="hljs-number">2</span>;<br></code></pre></td></tr></table></figure><blockquote><p>REPLICA 参数是指修改数据库副本数，取值范围 [1, 3]。在集群中使用，副本数必须小于或等于 DNODE 的数目。</p></blockquote><p>设置保留天数参数</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> DATABASE locationcenter KEEP <span class="hljs-number">365000</span>;<br></code></pre></td></tr></table></figure><blockquote><p>KEEP 参数是指修改数据文件保存的天数，缺省值为 3650，取值范围 [days, 365000]，必须大于或等于 days 参数值。查看参数时有的话默认就行</p></blockquote><p>设置块大小</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> DATABASE locationcenter BLOCKS <span class="hljs-number">100</span>;<br></code></pre></td></tr></table></figure><blockquote><p>BLOCKS 参数是每个 VNODE (TSDB) 中有多少 cache 大小的内存块，因此一个 VNODE 的用的内存大小粗略为（cache * blocks）。取值范围 [3, 1000]。</p></blockquote><p>设置确认数</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> DATABASE locationcenter QUORUM <span class="hljs-number">2</span>;<br></code></pre></td></tr></table></figure><blockquote><p>QUORUM 参数是指数据写入成功所需要的确认数，取值范围 [1, 2]。对于异步复制，quorum 设为 1，具有 master 角色的虚拟节点自己确认即可。对于同步复制，quorum 设为 2。原则上，Quorum >= 1 并且 Quorum <= replica(副本数)，这个参数在启动一个同步模块实例时需要提供。</p></blockquote><p>设置缓存</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">ALTER</span> DATABASE locationcenter CACHELAST <span class="hljs-number">3</span>;<br></code></pre></td></tr></table></figure><blockquote><p>CACHELAST 参数控制是否在内存中缓存子表的最近数据。缺省值为 0，取值范围 [0, 1, 2, 3]。其中 0 表示不缓存，1 表示缓存子表最近一行数据，2 表示缓存子表每一列的最近的非 NULL 值，3 表示同时打开缓存最近行和列功能。（从 2.0.11.0 版本开始支持参数值 [0, 1]，从 2.1.2.0 版本开始支持参数值 [0, 1, 2, 3]。）<br>说明：缓存最近行，将显著改善 LAST_ROW 函数的性能表现；缓存每列的最近非 NULL 值，将显著改善无特殊影响（WHERE、ORDER BY、GROUP BY、INTERVAL）下的 LAST 函数的性能表现。</p></blockquote><p>显示所有数据库及设置参数</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SHOW</span> DATABASES;<br><span class="hljs-keyword">SHOW</span> <span class="hljs-keyword">CREATE</span> DATABASE locationcenter\G;<br></code></pre></td></tr></table></figure><p>服务器参数优化参考 <a href="https://docs.taosdata.com/reference/config/">https://docs.taosdata.com/reference/config/</a></p><p>taos服务和taosadapter参数配置，dnode添加删除，mnode数量配置，vnode表数量多少，表在vnode分布，vnode在dnode上的分布调整。</p><p>参考文章：<a href="https://zhuanlan.zhihu.com/p/380778972">https://zhuanlan.zhihu.com/p/380778972</a></p><p>taosadapter生产环境参数，需要修改日志记录级别，修改日志记录位置（放数据盘）。</p><h3 id="建表"><a href="#建表" class="headerlink" title="建表"></a>建表</h3><p>实际场景解释：创建超级表，为模板表，根据实际业务，壹号车服app使用人员3000-4000人，数据写入时根据人员登录Code分不同子表存储数据，这样取数据时会每个人只从一个单子表里取值，且数据库参数CACHELAST设置为3后，每个人最新一次定位直接从缓存获取，大大加快查询速度。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs sql">USE `locationcenter`;<br><span class="hljs-keyword">DROP</span> STABLE IF <span class="hljs-keyword">EXISTS</span> `userlocation`;<br><span class="hljs-keyword">CREATE</span> STABLE `userlocation` <br>(`locationtime` <span class="hljs-type">TIMESTAMP</span>,<br> `loginname` <span class="hljs-type">NCHAR</span>(<span class="hljs-number">50</span>),<br> `lat` <span class="hljs-keyword">DOUBLE</span>,<br> `lng` <span class="hljs-keyword">DOUBLE</span>,<br> `speed` <span class="hljs-keyword">DOUBLE</span>,<br> `address` <span class="hljs-type">NCHAR</span>(<span class="hljs-number">200</span>)) <br> TAGS (`username` <span class="hljs-type">NCHAR</span>(<span class="hljs-number">50</span>),<br> `deviceimei` <span class="hljs-type">NCHAR</span>(<span class="hljs-number">50</span>),<br> `devicemodel` <span class="hljs-type">NCHAR</span>(<span class="hljs-number">100</span>),<br> `devicetype` BOOL);<br></code></pre></td></tr></table></figure><blockquote><p>说明</p><ol><li>表的第一个字段必须是 TIMESTAMP，并且系统自动将其设为主键；</li><li>表名最大长度为 192；</li><li>表的每行长度不能超过 48KB;（注意：每个 BINARY/NCHAR 类型的列还会额外占用 2 个字节的存储位置）；</li><li>子表名只能由字母、数字和下划线组成，且不能以数字开头，不区分大小写；</li><li>使用数据类型 binary 或 nchar，需指定其最长的字节数，如 binary(20)，表示 20 字节；</li><li>为了兼容支持更多形式的表名，TDengine 引入新的转义符 “<code>"，可以让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查。但是同样具有长度限制要求。使用转义字符以后，不再对转义字符中的内容进行大小写统一。 例如：</code>aBc<code>和</code>abc` 是不同的表名，但是 abc 和 aBc 是相同的表名。 需要注意的是转义字符中的内容必须是可打印字符。 上述的操作逻辑和约束要求与 MySQL 数据的操作一致。 从 2.3.0.0 版本开始支持这种方式；</li><li>数据迁移注意保障列长足够，表的第一个字段可以使用UNIX_TIMESTAMP格式写入，根据库的时间经度自动转换；</li><li>TAGS 列的数据类型不能是 timestamp 类型；（从 2.1.3.0 版本开始，TAGS 列中支持使用 timestamp 类型，但需注意在 TAGS 中的 timestamp 列写入数据时需要提供给定值，而暂不支持四则运算，例如 <code>NOW + 10s</code> 这类表达式）</li><li>TAGS 列名不能与其他列名相同；</li><li>TAGS 列名不能为预留关键字（参见：<a href="https://docs.taosdata.com/taos-sql/keywords/">参数限制与保留关键字</a> 章节）；</li><li>TAGS 最多允许 128 个，至少 1 个，总长度不超过 16 KB。</li></ol></blockquote><p>支持无模式表数据写入，兼容 InfluxDB 的 行协议（Line Protocol）、OpenTSDB 的 telnet 行协议、OpenTSDB 的 JSON 格式协议。</p><p>介绍：<a href="https://docs.taosdata.com/reference/schemaless/">https://docs.taosdata.com/reference/schemaless/</a></p><p>如表列不定，随时增加可使用此模式。</p><p>支持的数据</p><table><thead><tr><th>#</th><th><strong>类型</strong></th><th><strong>Bytes</strong></th><th><strong>说明</strong></th></tr></thead><tbody><tr><td>1</td><td>TIMESTAMP</td><td>8</td><td>时间戳。缺省精度毫秒，可支持微秒和纳秒。从格林威治时间 1970-01-01 00:00:00.000 (UTC/GMT) 开始，计时不能早于该时间。（从 2.0.18.0 版本开始，已经去除了这一时间范围限制）（从 2.1.5.0 版本开始支持纳秒精度）</td></tr><tr><td>2</td><td>INT</td><td>4</td><td>整型，范围 [-2^31+1, 2^31-1], -2^31 用作 NULL</td></tr><tr><td>3</td><td>BIGINT</td><td>8</td><td>长整型，范围 [-2^63+1, 2^63-1], -2^63 用作 NULL</td></tr><tr><td>4</td><td>FLOAT</td><td>4</td><td>浮点型，有效位数 6-7，范围 [-3.4E38, 3.4E38]</td></tr><tr><td>5</td><td>DOUBLE</td><td>8</td><td>双精度浮点型，有效位数 15-16，范围 [-1.7E308, 1.7E308]</td></tr><tr><td>6</td><td>BINARY</td><td>自定义</td><td>记录单字节字符串，建议只用于处理 ASCII 可见字符，中文等多字节字符需使用 nchar。理论上，最长可以有 16374 字节。binary 仅支持字符串输入，字符串两端需使用单引号引用。使用时须指定大小，如 binary(20) 定义了最长为 20 个单字节字符的字符串，每个字符占 1 byte 的存储空间，总共固定占用 20 bytes 的空间，此时如果用户字符串超出 20 字节将会报错。对于字符串内的单引号，可以用转义字符反斜线加单引号来表示，即 <code>\’</code>。</td></tr><tr><td>7</td><td>SMALLINT</td><td>2</td><td>短整型， 范围 [-32767, 32767], -32768 用作 NULL</td></tr><tr><td>8</td><td>TINYINT</td><td>1</td><td>单字节整型，范围 [-127, 127], -128 用作 NULL</td></tr><tr><td>9</td><td>BOOL</td><td>1</td><td>布尔型，{true, false}</td></tr><tr><td>10</td><td>NCHAR</td><td>自定义</td><td>记录包含多字节字符在内的字符串，如中文字符。每个 nchar 字符占用 4 bytes 的存储空间。字符串两端使用单引号引用，字符串内的单引号需用转义字符 <code>\’</code>。nchar 使用时须指定字符串大小，类型为 nchar(10) 的列表示此列的字符串最多存储 10 个 nchar 字符，会固定占用 40 bytes 的空间。如果用户字符串长度超出声明长度，将会报错。</td></tr><tr><td>11</td><td>JSON</td><td></td><td>json 数据类型， 只有 tag 可以是 json 格式</td></tr></tbody></table><p>显示建表语句</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SHOW</span> <span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> userlocation\G;<br><span class="hljs-keyword">DESCRIBE</span> userlocation;<br></code></pre></td></tr></table></figure><blockquote><p>注意：查询超级表时对表描述词要改为 STABLE。</p><p>例：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SHOW</span> TABLES;<br><span class="hljs-keyword">SHOW</span> STABLES;<br><span class="hljs-keyword">DROP</span> STABLE XX;<br></code></pre></td></tr></table></figure></blockquote><p>表字段操作同MySQL，表标签相关管理：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs sql">#修改子表标签<br><span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">TABLE</span> userlocation <span class="hljs-keyword">SET</span> TAG loginname<span class="hljs-operator">=</span>loginname;<br>#添加超级表标签<br><span class="hljs-keyword">ALTER</span> STABLE userlocation <span class="hljs-keyword">ADD</span> TAG testtag <span class="hljs-type">NCHAR</span>(<span class="hljs-number">50</span>);<br>#修改超级表标签名<br><span class="hljs-keyword">ALTER</span> STABLE userlocation CHANGE TAG testtag testtag1;<br>#修改超级表列宽<br><span class="hljs-keyword">ALTER</span> STABLE stb_name MODIFY TAG testtag1 <span class="hljs-type">NCHAR</span>(<span class="hljs-number">100</span>);<br>#删除超级表标签<br><span class="hljs-keyword">ALTER</span> STABLE userlocation <span class="hljs-keyword">DROP</span> TAG testtag1;<br></code></pre></td></tr></table></figure><blockquote><p>使用 SELECT 语句可以完成在超级表上的投影及聚合两类查询，在 WHERE 语句中可以对标签及列进行筛选及过滤。</p><p>如果在超级表查询语句中不加 ORDER BY, 返回顺序是先返回一个子表的所有数据，然后再返回下个子表的所有数据，所以返回的数据是无序的。如果增加了 ORDER BY 语句，会严格按 ORDER BY 语句指定的顺序返回的。</p><p>除了更新标签的值的操作是针对子表进行，其他所有的标签操作（添加标签、删除标签等）均只能作用于 STable，不能对单个子表操作。对 STable 添加标签以后，依托于该 STable 建立的所有表将自动增加了一个标签，所有新增标签的默认值都是 NULL。</p></blockquote><p>开源版限制</p><p>1.mnode最多3个；</p><p>2.内存分配机制没有企业版控制的好；</p><p>内存OOM原因：</p><p> 剩余内存小于 vm.min_free_kbytes ；</p><p>程序请求的内存大于剩余内存；</p><p>内存充足但程序占用了特殊的内存地址。</p><h3 id="数据迁移"><a href="#数据迁移" class="headerlink" title="数据迁移"></a>数据迁移</h3><p>这里使用DATAX</p><p>☆ DATAX工具包必需使用官方提供，不然会出各种依赖包问题</p><p>亲测可用工具包下载地址：<a href="https://cloud.189.cn/t/AVbmaqmUJRf2%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Ae1jy%EF%BC%89">https://cloud.189.cn/t/AVbmaqmUJRf2（访问码：e1jy）</a></p><p>安装 datax后，此文默认DATAX_HOME:/opt/datax</p><p>迁移脚本放入 /datax/job文件夹内</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"job"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"content"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"reader"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"mysqlreader"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"parameter"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"username"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"user_service"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"password"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Lunz2017"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"connection"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"querySql"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-string">"SELECT CONCAT(CONCAT(CONCAT('t',LoginName),'_'),DeviceIMEI) AS tbname,LocationTime,Lat,Lng,Speed,Address,LoginName,UserName,DeviceIMEI,DeviceModel,DeviceType FROM tb_userlocation WHERE LocationTime IS NOT NULL AND Lat IS NOT NULL AND Lng IS NOT NULL AND Speed IS NOT NULL AND Address IS NOT NULL AND LoginName IS NOT NULL AND UserName IS NOT NULLAND DeviceIMEI IS NOT NULL AND DeviceModel IS NOT NULL AND DeviceType IS NOT NULL;"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"jdbcUrl"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"jdbc:mysql://pc-bp1zip05gl1b1ga3veo.rwlb.rds.aliyuncs.com:3306/locationcenter"</span><span class="hljs-punctuation">]</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">]</span><br> <span class="hljs-punctuation">}</span> <br><span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"writer"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"tdenginewriter"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"parameter"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"username"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"user_service"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"password"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Lunz2017"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"column"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-string">"tbname"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"locationtime"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"lat"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"lng"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"speed"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"address"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"loginname"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"username"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"deviceimei"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"devicemodel"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"devicetype"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"connection"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"table"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-string">"userlocation"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"jdbcUrl"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"jdbc:TAOS-RS://192.168.200.99:6041/locationcenter"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"jdbcUrl"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"jdbc:TAOS-RS://192.168.200.148:6041/locationcenter"</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"batchSize"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">10000</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"ignoreTagsUnmatched"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">true</span></span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"setting"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"speed"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"channel"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">3</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"byte"</span> <span class="hljs-punctuation">:</span> <span class="hljs-number">5242880</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p>执行迁移</p><p>数据使用DATAX迁移，速度较慢</p><p><strong>优化方式</strong></p><p><em><strong>速度控制：</strong></em></p><p>迁移 json脚本中直接设置</p><table><thead><tr><th>job.setting.speed.channel</th><th>总并发数</th></tr></thead><tbody><tr><td>job.setting.speed.record</td><td>总record限速</td></tr><tr><td>job.setting.speed.byte</td><td>总byte限速</td></tr><tr><td>core.transport.channel.speed.record</td><td>单个channel的record限速，默认值为10000（10000条/s）</td></tr><tr><td>core.transport.channel.speed.byte</td><td>单个channel的byte限速，默认值1024*1024（1M/s）</td></tr></tbody></table><p><strong>注意事项：</strong></p><ol><li>若配置了总record限速，则必须配置单个channel的record限速</li><li>若配置了总byte限速，则必须配置单个channe的byte限速</li><li>若配置了总record限速和总byte限速，channel并发数参数就会失效。因为配置了总record限速和总byte限速之后，实际channel并发数是通过计算得到的：</li></ol><p><strong>计算公式为:</strong></p><p>min(总byte限速/单个channle的byte限速，总record限速/单个channel的record限速)</p><p>配置示例：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"core"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"transport"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"channel"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"speed"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"byte"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1048576</span> <span class="hljs-comment">//单个channel byte限速1M/s</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"job"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"setting"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"speed"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"byte"</span> <span class="hljs-punctuation">:</span> <span class="hljs-number">5242880</span> <span class="hljs-comment">//总byte限速5M/s</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> ...<br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p><em><strong>内存调整</strong></em></p><p>当提升DataX Job内Channel并发数时，内存的占用会显著增加，因为DataX作为数据交换通道，在内存中会缓存较多的数据。例如Channel中会有一个Buffer，作为临时的数据交换的缓冲区，而在部分Reader和Writer的中，也会存在一些Buffer，为了防止OOM等错误，需调大JVM的堆内存。</p><p> 调整JVM xms xmx参数的两种方式：一种是直接更改datax.py脚本；另一种是在启动的时候，加上对应的参数，如下：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">python /newdisk/soft/datax/bin/datax.py --jvm="-Xms6G -Xmx6G" /newdisk/soft/datax/job/mysql2td_location.json<br></code></pre></td></tr></table></figure><ul><li><p>-Xms8G：运行的最小分配内存，如果可用内存不足5G，这个命令将不能被执行。</p></li><li><p>-Xmx8G：运行时最大占用内存大小。</p></li></ul><p>最终优化后json任务文件</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"job"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"content"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"reader"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"mysqlreader"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"parameter"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"username"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"user_service"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"password"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Lunz2017"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"splitPk"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Id"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"column"</span><span class="hljs-punctuation">:</span><span class="hljs-punctuation">[</span><br> <span class="hljs-string">"td_tablebane"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"LocationTime"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"Lat"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"Lng"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"Speed"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"Address"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"LoginName"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"UserName"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"DeviceIMEI"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"DeviceModel"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"DeviceType"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"connection"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"table"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"tb_userlocation_td3"</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"jdbcUrl"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"jdbc:mysql://pc-bp1zip05gl1b1ga3veo.rwlb.rds.aliyuncs.com:3306/locationcenter"</span><span class="hljs-punctuation">]</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">]</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"writer"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"tdenginewriter"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"parameter"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"username"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"user_service"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"password"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Lunz2017"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"column"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-string">"tbname"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"locationtime"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"lat"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"lng"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"speed"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"address"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"loginname"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"username"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"deviceimei"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"devicemodel"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-string">"devicetype"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"connection"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"table"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-string">"userlocation"</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"jdbcUrl"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"jdbc:TAOS-RS://192.168.200.99:6041/locationcenter"</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"batchSize"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">10000</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"ignoreTagsUnmatched"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">true</span></span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"setting"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"speed"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"byte"</span> <span class="hljs-punctuation">:</span> <span class="hljs-number">15728640</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br><span class="hljs-attr">"core"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"transport"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"channel"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"speed"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"byte"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">5242880</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p><em><strong>执行脚本</strong></em></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">新建日志文档，记得定时清理，比较大</span><br>touch /opt/datax/log/mysql2td_location.log<br><span class="hljs-meta prompt_">#</span><span class="language-bash">后台执行迁移脚本，因数据量较大时间较久，为防止session断掉，后台运行</span><br>nohup python /opt/datax/bin/datax.py --jvm="-Xms5G -Xmx5G" /opt/datax/job/mysql2td_location.json >> /opt/datax/log/mysql2td_location.log 2>&1 &<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看日志</span><br>tail -200f /opt/datax/log/mysql2td_location.log<br><span class="hljs-meta prompt_">#</span><span class="language-bash">日志太大了，不要日志</span><br>nohup python /newdisk/soft/datax/bin/datax.py --jvm="-Xms6G -Xmx6G" /newdisk/soft/datax/job/mysql2td_location.json >/dev/null 2>log &<br><span class="hljs-meta prompt_">#</span><span class="language-bash">因为是后台运行 执行完毕需要杀掉进程</span><br>ps -ef|grep datax<br>kill pid<br></code></pre></td></tr></table></figure><h3 id="应用数据压测对比"><a href="#应用数据压测对比" class="headerlink" title="应用数据压测对比"></a>应用数据压测对比</h3><blockquote><p>测试数据：1.3亿条</p><p>存储格式：超级表加单表</p><p>服务器：双节点4C8G</p></blockquote><p>查询单人3天内所有定位点并按定位时间排序：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span><br><span class="hljs-keyword">FROM</span> userlocation <br><span class="hljs-keyword">WHERE</span> locationtime <span class="hljs-operator">></span> now<span class="hljs-number">-750</span>d <br><span class="hljs-keyword">AND</span> loginname<span class="hljs-operator">=</span><span class="hljs-string">'ZR18080096'</span><br><span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> locationtime <br>LIMIT <span class="hljs-number">10</span>;<br></code></pre></td></tr></table></figure><blockquote><p>平均用时：Query OK, 10 row(s) in set (0.029506s)</p></blockquote><p>查询每个人最新一次定位</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">LAST</span>(<span class="hljs-operator">*</span>) <br><span class="hljs-keyword">FROM</span> userlocation <br><span class="hljs-keyword">WHERE</span> loginname<span class="hljs-operator">=</span><span class="hljs-string">'ZR18080096'</span>;<br></code></pre></td></tr></table></figure><blockquote><p>平均用时：Query OK, 1 row(s) in set (0.004578s)</p></blockquote><p>使用<code>taosBenchmark</code>测试TD性能</p><p><font color=red>压测使用超级表时，经常会在WHERE条件里的需要筛选的字段，放在字段里不要放TAG内</font></p><ul><li>时序数据：存放于 vnode 里，由 data、head 和 last 三个文件组成，数据量大，查询量取决于应用场景。容许乱序写入，但暂时不支持删除操作，并且仅在 update 参数设置为 1 时允许更新操作。通过采用一个采集点一张表的模型，一个时间段的数据是连续存储，对单张表的写入是简单的追加操作，一次读，可以读到多条记录，这样保证对单个采集点的插入和查询操作，性能达到最优。</li><li>标签数据：存放于 vnode 里的 meta 文件，支持增删改查四个标准操作。数据量不大，有 N 张表，就有 N 条记录，因此可以全内存存储。如果标签过滤操作很多，查询将十分频繁，因此 TDengine 支持多核多线程并发查询。只要计算资源足够，即使有数千万张表，过滤结果能毫秒级返回。</li><li>元数据：存放于 mnode 里，包含系统节点、用户、DB、Table Schema 等信息，支持增删改查四个标准操作。这部分数据的量不大，可以全内存保存，而且由于客户端有缓存，查询量也不大。因此目前的设计虽是集中式存储管理，但不会构成性能瓶颈。</li></ul><p>工具介绍: <a href="https://www.taosdata.com/docs/cn/v2.0/tools/taosbenchmark#cli">https://www.taosdata.com/docs/cn/v2.0/tools/taosbenchmark#cli</a></p><p>检测脚本</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">touch /newdisk/soft/taosbenchmark/locationsearch.json<br></code></pre></td></tr></table></figure><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"filetype"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"query"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"cfgdir"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"/etc/taos"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"host"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"192.168.200.99"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"port"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">6030</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"user"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"user_service"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"password"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Lunz2017"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"confirm_parameter_prompt"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"no"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"databases"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"locationcenter"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"query_times"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">200</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"query_mode"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"taosc"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"super_table_query"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"stblname"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"userlocation"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"query_interval"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">0</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"threads"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">5</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"sqls"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"sql"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"SELECT * FROM xxxx WHERE locationtime > now-750d AND loginname='ZR18080096' ORDER BY locationtime LIMIT 1000;"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"sql"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"SELECT LAST(*) FROM xxxx WHERE loginname='ZR18080096';"</span><br> <span class="hljs-punctuation">}</span> <br> <span class="hljs-punctuation">]</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p>执行指令</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">taosdemo -f /newdisk/soft/taosbenchmark/locationsearch.json -o /newdisk/soft/taosbenchmark/locationsearch.txt<br></code></pre></td></tr></table></figure><p>第一次测试WHERE 筛选条件里不带loginname筛选</p><p>检测结果：</p><details> <summary>展开查看测试结果</summary> <pre><code class="hljs"> [06/23 11:25:29.826294] INFO: /newdisk/soft/taosbenchmark/locationsearch.json{ "filetype": "query", "cfgdir": "/etc/taos", "host": "192.168.200.99", "port": 6030, "user": "user_service", "password": "Lunz2017", "confirm_parameter_prompt": "no", "databases": "locationcenter", "query_times": 20, "query_mode": "taosc", "super_table_query": { "stblname": "userlocation", "query_interval": 0, "threads": 5, "sqls": [{ "sql": "SELECT * FROM xxxx WHERE locationtime > now-750d ORDER BY locationtime LIMIT 1000;" }, { "sql": "SELECT LAST(*) FROM xxxx;" }] }}[06/23 11:25:29.942103] INFO: taos client version: 2.6.0.0[06/23 11:25:29.985904] INFO: userlocation's childTblCount: 2204[06/23 11:25:47.395394] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.3860s[06/23 11:25:47.661997] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:17.6520s[06/23 11:25:49.801203] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:19.7930s[06/23 11:25:51.781808] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.7750s[06/23 11:25:52.382326] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:22.3750s[06/23 11:26:00.011229] INFO: thread[4] has currently completed queries: 1531, QPS: 51.030[06/23 11:26:00.014133] INFO: thread[3] has currently completed queries: 1534, QPS: 51.125[06/23 11:26:00.024295] INFO: thread[2] has currently completed queries: 1369, QPS: 45.609[06/23 11:26:00.024452] INFO: thread[1] has currently completed queries: 1201, QPS: 40.011[06/23 11:26:00.059128] INFO: thread[0] has currently completed queries: 1241, QPS: 41.294[06/23 11:26:04.529191] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.1340s[06/23 11:26:04.737206] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:17.0750s[06/23 11:26:08.044346] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.2430s[06/23 11:26:13.127916] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.3460s[06/23 11:26:13.680380] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:21.2980s[06/23 11:26:21.806175] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.2770s[06/23 11:26:22.789503] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:18.0520s[06/23 11:26:26.866865] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.8220s[06/23 11:26:30.013590] INFO: thread[4] has currently completed queries: 3067, QPS: 51.113[06/23 11:26:30.017715] INFO: thread[3] has currently completed queries: 3023, QPS: 50.377[06/23 11:26:30.068283] INFO: thread[0] has currently completed queries: 2470, QPS: 41.124[06/23 11:26:30.072142] INFO: thread[1] has currently completed queries: 2441, QPS: 40.639[06/23 11:26:30.081825] INFO: thread[2] has currently completed queries: 2781, QPS: 46.294[06/23 11:26:33.868119] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:20.7410s[06/23 11:26:34.794845] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:21.1140s[06/23 11:26:38.007910] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.2010s[06/23 11:26:39.027625] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.2380s[06/23 11:26:44.349937] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:17.4830s[06/23 11:26:53.690002] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:19.8220s[06/23 11:26:54.179681] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.1720s[06/23 11:26:54.751370] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:19.9570s[06/23 11:26:55.450360] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.4230s[06/23 11:27:00.032487] INFO: thread[4] has currently completed queries: 4689, QPS: 52.087[06/23 11:27:00.036375] INFO: thread[3] has currently completed queries: 4625, QPS: 51.373[06/23 11:27:00.081331] INFO: thread[1] has currently completed queries: 3736, QPS: 41.477[06/23 11:27:00.093663] INFO: thread[0] has currently completed queries: 3794, QPS: 42.115[06/23 11:27:00.111722] INFO: thread[2] has currently completed queries: 4303, QPS: 47.756[06/23 11:27:02.482377] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.1320s[06/23 11:27:12.567807] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:18.3880s[06/23 11:27:15.218884] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:19.7680s[06/23 11:27:16.970065] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:23.2800s[06/23 11:27:18.378608] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:23.6270s[06/23 11:27:23.262542] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:20.7800s[06/23 11:27:30.036647] INFO: thread[4] has currently completed queries: 6052, QPS: 50.422[06/23 11:27:30.040622] INFO: thread[3] has currently completed queries: 5937, QPS: 49.462[06/23 11:27:30.096812] INFO: thread[0] has currently completed queries: 4880, QPS: 40.636[06/23 11:27:30.111611] INFO: thread[1] has currently completed queries: 4829, QPS: 40.207[06/23 11:27:30.121808] INFO: thread[2] has currently completed queries: 5556, QPS: 46.256[06/23 11:27:31.958135] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:19.3910s[06/23 11:27:34.558165] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:19.3400s[06/23 11:27:39.456386] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:22.4860s[06/23 11:27:40.932419] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:22.5540s[06/23 11:27:41.732982] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.4700s[06/23 11:27:48.320896] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.3620s[06/23 11:27:50.724743] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.1660s[06/23 11:27:58.327522] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:16.5940s[06/23 11:28:00.021201] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:20.5650s[06/23 11:28:00.053498] INFO: thread[4] has currently completed queries: 7668, QPS: 51.105[06/23 11:28:00.069814] INFO: thread[3] has currently completed queries: 7541, QPS: 50.253[06/23 11:28:00.107871] INFO: thread[0] has currently completed queries: 6178, QPS: 41.159[06/23 11:28:00.117358] INFO: thread[1] has currently completed queries: 6139, QPS: 40.897[06/23 11:28:00.129142] INFO: thread[2] has currently completed queries: 7133, QPS: 47.515[06/23 11:28:00.883925] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:19.9510s[06/23 11:28:04.528633] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.2080s[06/23 11:28:07.278929] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.5540s[06/23 11:28:16.284883] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:17.9570s[06/23 11:28:20.968207] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:20.9470s[06/23 11:28:21.440716] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.9120s[06/23 11:28:21.903253] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:21.0200s[06/23 11:28:23.760206] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.4820s[06/23 11:28:30.081073] INFO: thread[3] has currently completed queries: 9175, QPS: 50.952[06/23 11:28:30.085854] INFO: thread[4] has currently completed queries: 9275, QPS: 51.506[06/23 11:28:30.127840] INFO: thread[1] has currently completed queries: 7423, QPS: 41.211[06/23 11:28:30.133642] INFO: thread[2] has currently completed queries: 8664, QPS: 48.100[06/23 11:28:30.171622] INFO: thread[0] has currently completed queries: 7459, QPS: 41.401[06/23 11:28:33.132875] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:16.8480s[06/23 11:28:38.366290] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.9260s[06/23 11:28:40.760953] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:17.0000s[06/23 11:28:42.121350] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.1530s[06/23 11:28:43.137537] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:21.2340s[06/23 11:28:51.169437] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.0370s[06/23 11:28:55.081240] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.7150s[06/23 11:28:57.246965] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.4850s[06/23 11:29:00.082296] INFO: thread[3] has currently completed queries: 10722, QPS: 51.039[06/23 11:29:00.120325] INFO: thread[4] has currently completed queries: 10817, QPS: 51.482[06/23 11:29:00.149787] INFO: thread[1] has currently completed queries: 8681, QPS: 41.310[06/23 11:29:00.164720] INFO: thread[2] has currently completed queries: 10131, QPS: 48.207[06/23 11:29:00.183676] INFO: thread[0] has currently completed queries: 8708, QPS: 41.432[06/23 11:29:03.182366] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.0610s[06/23 11:29:03.668377] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:20.5310s[06/23 11:29:09.831966] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.6620s[06/23 11:29:13.002151] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.9210s[06/23 11:29:15.903543] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:18.6560s[06/23 11:29:24.725181] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.5430s[06/23 11:29:25.210472] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:21.5420s[06/23 11:29:27.849301] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.0170s[06/23 11:29:30.002702] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.0000s[06/23 11:29:30.085149] INFO: thread[3] has currently completed queries: 12182, QPS: 50.742[06/23 11:29:30.150275] INFO: thread[4] has currently completed queries: 12325, QPS: 51.324[06/23 11:29:30.157143] INFO: thread[1] has currently completed queries: 9895, QPS: 41.203[06/23 11:29:30.199164] INFO: thread[2] has currently completed queries: 11555, QPS: 48.108[06/23 11:29:30.208238] INFO: thread[0] has currently completed queries: 9915, QPS: 41.278[06/23 11:29:33.398296] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:17.4950s[06/23 11:29:45.909740] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.0600s[06/23 11:29:46.418577] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.6930s[06/23 11:29:46.684370] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:21.4740s[06/23 11:29:46.867443] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:16.8650s[06/23 11:29:49.834367] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.4360s[06/23 11:30:00.094313] INFO: thread[3] has currently completed queries: 13804, QPS: 51.110[06/23 11:30:00.168550] INFO: thread[4] has currently completed queries: 13965, QPS: 51.692[06/23 11:30:00.178332] INFO: thread[1] has currently completed queries: 11213, QPS: 41.503[06/23 11:30:00.204815] INFO: thread[2] has currently completed queries: 13112, QPS: 48.528[06/23 11:30:00.211187] INFO: thread[0] has currently completed queries: 11214, QPS: 41.502[06/23 11:30:02.210162] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:15.3430s[06/23 11:30:02.611121] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:16.7020s[06/23 11:30:05.967470] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.1330s[06/23 11:30:06.086602] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:19.6680s[06/23 11:30:06.175547] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:19.4910s[06/23 11:30:19.380477] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.1700s[06/23 11:30:20.895797] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:18.2840s[06/23 11:30:23.153220] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:17.1860s[06/23 11:30:27.048434] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:20.8730s[06/23 11:30:27.383756] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:21.2970s[06/23 11:30:30.124032] INFO: thread[3] has currently completed queries: 15317, QPS: 51.037[06/23 11:30:30.184927] INFO: thread[4] has currently completed queries: 15484, QPS: 51.583[06/23 11:30:30.190163] INFO: thread[1] has currently completed queries: 12463, QPS: 41.518[06/23 11:30:30.207185] INFO: thread[2] has currently completed queries: 14500, QPS: 48.301[06/23 11:30:30.222273] INFO: thread[0] has currently completed queries: 12457, QPS: 41.493[06/23 11:30:38.852287] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:19.4720s[06/23 11:30:41.832871] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:20.9370s[06/23 11:30:43.885670] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:20.7320s[06/23 11:30:52.244190] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:25.1960s[06/23 11:30:52.909152] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:25.5260s[06/23 11:30:59.190679] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:20.3380s[06/23 11:31:00.148485] INFO: thread[3] has currently completed queries: 16563, QPS: 50.170[06/23 11:31:00.193099] INFO: thread[4] has currently completed queries: 16768, QPS: 50.784[06/23 11:31:00.196303] INFO: thread[1] has currently completed queries: 13525, QPS: 40.961[06/23 11:31:00.208392] INFO: thread[2] has currently completed queries: 15740, QPS: 47.668[06/23 11:31:00.227029] INFO: thread[0] has currently completed queries: 13500, QPS: 40.882[06/23 11:31:03.109165] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:21.2770s[06/23 11:31:04.170431] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:20.2850s[06/23 11:31:14.628583] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:22.3840s[06/23 11:31:15.104783] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:22.1950s[06/23 11:31:16.755738] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:17.5650s[06/23 11:31:20.150011] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:17.0410s[06/23 11:31:20.423057] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:16.2530s[06/23 11:31:30.207574] INFO: thread[1] has currently completed queries: 14947, QPS: 41.496[06/23 11:31:30.231212] INFO: thread[2] has currently completed queries: 17467, QPS: 48.489[06/23 11:31:30.239761] INFO: thread[0] has currently completed queries: 14937, QPS: 41.465[06/23 11:31:30.896999] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.2680s[06/23 11:31:31.153452] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:16.0490s[06/23 11:31:32.208711] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:12.0580s[06/23 11:31:44.384336] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:13.4870s[06/23 11:31:44.598594] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:13.4450s[06/23 11:31:57.261829] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:12.8770s[06/23 11:31:57.762974] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:13.1640s[06/23 11:32:00.217199] INFO: thread[1] has currently completed queries: 16931, QPS: 43.389[06/23 11:32:00.255773] INFO: thread[0] has currently completed queries: 16907, QPS: 43.324[06/23 11:32:11.843653] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:14.5820s[06/23 11:32:12.100391] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:14.3370s[06/23 11:32:12.101175] INFO: Spend 402.0970 second completed total queries: 88160, the QPS of all threads: 219.251 </code></pre></details><p>QPS大概是295</p><p>第二次测试为上面全部查询条件，带WHERE筛选</p><details> <summary>展开查看测试结果</summary> <pre><code class="hljs"> [06/23 15:56:16.741911] INFO: /newdisk/soft/taosbenchmark/locationsearch.json{ "filetype": "query", "cfgdir": "/etc/taos", "host": "192.168.200.99", "port": 6030, "user": "user_service", "password": "Lunz2017", "confirm_parameter_prompt": "no", "databases": "locationcenter", "query_times": 20, "query_mode": "taosc", "super_table_query": { "stblname": "userlocation", "query_interval": 0, "threads": 5, "sqls": [{ "sql": "SELECT * FROM xxxx WHERE locationtime > now-750d AND loginname='ZR18080096' ORDER BY locationtime LIMIT 1000;" }, { "sql": "SELECT LAST(*) FROM xxxx WHERE loginname='ZR18080096';" }] }}[06/23 15:56:16.882414] INFO: taos client version: 2.6.0.0[06/23 15:56:16.910498] INFO: userlocation's childTblCount: 2204[06/23 15:56:29.610277] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.6770s[06/23 15:56:29.851190] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.9170s[06/23 15:56:30.948195] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:14.0150s[06/23 15:56:33.864315] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.9320s[06/23 15:56:34.081614] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:17.1500s[06/23 15:56:42.131075] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.2800s[06/23 15:56:42.348381] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.7380s[06/23 15:56:45.165683] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:14.2170s[06/23 15:56:46.935699] INFO: thread[0] has currently completed queries: 1582, QPS: 52.726[06/23 15:56:46.941998] INFO: thread[3] has currently completed queries: 2098, QPS: 69.915[06/23 15:56:46.949207] INFO: thread[1] has currently completed queries: 1561, QPS: 52.004[06/23 15:56:46.964167] INFO: thread[4] has currently completed queries: 2083, QPS: 69.364[06/23 15:56:46.965832] INFO: thread[2] has currently completed queries: 1889, QPS: 62.900[06/23 15:56:50.149224] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.2850s[06/23 15:56:50.215609] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:16.1340s[06/23 15:56:54.128171] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:11.7800s[06/23 15:56:54.502407] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.3710s[06/23 15:56:58.316296] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.1510s[06/23 15:57:06.099179] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.9500s[06/23 15:57:06.300253] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:16.0850s[06/23 15:57:06.536024] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.4080s[06/23 15:57:06.668328] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.1660s[06/23 15:57:11.682409] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.3660s[06/23 15:57:16.957269] INFO: thread[3] has currently completed queries: 4237, QPS: 70.588[06/23 15:57:16.969086] INFO: thread[4] has currently completed queries: 4220, QPS: 70.292[06/23 15:57:16.970547] INFO: thread[0] has currently completed queries: 3227, QPS: 53.748[06/23 15:57:16.973480] INFO: thread[1] has currently completed queries: 3215, QPS: 53.547[06/23 15:57:16.985694] INFO: thread[2] has currently completed queries: 3767, QPS: 62.729[06/23 15:57:19.170242] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.6340s[06/23 15:57:19.243252] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.5750s[06/23 15:57:22.138964] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.0390s[06/23 15:57:22.268433] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.9680s[06/23 15:57:25.518519] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.8360s[06/23 15:57:31.449882] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.2790s[06/23 15:57:31.857031] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.6140s[06/23 15:57:37.294125] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.1550s[06/23 15:57:37.499734] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.2310s[06/23 15:57:38.831634] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.3130s[06/23 15:57:44.038288] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.5890s[06/23 15:57:44.764270] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.9070s[06/23 15:57:46.961418] INFO: thread[3] has currently completed queries: 6376, QPS: 70.822[06/23 15:57:46.974507] INFO: thread[0] has currently completed queries: 4943, QPS: 54.896[06/23 15:57:46.976399] INFO: thread[4] has currently completed queries: 6343, QPS: 70.445[06/23 15:57:46.980930] INFO: thread[1] has currently completed queries: 4943, QPS: 54.893[06/23 15:57:46.992588] INFO: thread[2] has currently completed queries: 5799, QPS: 64.391[06/23 15:57:52.125702] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.2940s[06/23 15:57:53.080700] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.7860s[06/23 15:57:53.433868] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.9340s[06/23 15:57:56.327224] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:11.5630s[06/23 15:57:56.615254] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.5770s[06/23 15:58:05.933754] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.8080s[06/23 15:58:09.475644] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:13.1480s[06/23 15:58:09.672404] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:16.2390s[06/23 15:58:09.914506] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.8340s[06/23 15:58:09.956917] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:13.3410s[06/23 15:58:16.966921] INFO: thread[3] has currently completed queries: 8454, QPS: 70.431[06/23 15:58:16.977414] INFO: thread[0] has currently completed queries: 6587, QPS: 54.871[06/23 15:58:16.990589] INFO: thread[1] has currently completed queries: 6565, QPS: 54.682[06/23 15:58:17.000403] INFO: thread[2] has currently completed queries: 7772, QPS: 64.731[06/23 15:58:17.001556] INFO: thread[4] has currently completed queries: 8453, QPS: 70.402[06/23 15:58:19.212969] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.2790s[06/23 15:58:21.579566] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.1040s[06/23 15:58:21.933175] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:11.9760s[06/23 15:58:25.113474] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.4410s[06/23 15:58:25.959218] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.0450s[06/23 15:58:33.001226] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.7880s[06/23 15:58:34.034189] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.4550s[06/23 15:58:34.435996] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.5020s[06/23 15:58:41.647532] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:16.5340s[06/23 15:58:42.139212] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.1800s[06/23 15:58:46.983853] INFO: thread[0] has currently completed queries: 8220, QPS: 54.781[06/23 15:58:46.987185] INFO: thread[3] has currently completed queries: 10563, QPS: 70.395[06/23 15:58:47.005994] INFO: thread[2] has currently completed queries: 9694, QPS: 64.596[06/23 15:58:47.017921] INFO: thread[1] has currently completed queries: 8197, QPS: 54.616[06/23 15:58:47.019911] INFO: thread[4] has currently completed queries: 10543, QPS: 70.247[06/23 15:58:47.106212] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:14.1050s[06/23 15:58:47.251632] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.8150s[06/23 15:58:47.281396] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:13.2470s[06/23 15:58:57.020348] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.3730s[06/23 15:58:57.287225] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.1480s[06/23 15:58:59.345230] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.0640s[06/23 15:58:59.431883] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.1800s[06/23 15:59:00.536989] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.4300s[06/23 15:59:11.005123] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:11.6600s[06/23 15:59:11.162868] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:11.7310s[06/23 15:59:12.037817] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.0170s[06/23 15:59:12.621252] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.3340s[06/23 15:59:13.415012] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:12.8780s[06/23 15:59:16.996454] INFO: thread[0] has currently completed queries: 9981, QPS: 55.430[06/23 15:59:17.022996] INFO: thread[4] has currently completed queries: 12739, QPS: 70.738[06/23 15:59:17.024869] INFO: thread[3] has currently completed queries: 12735, QPS: 70.714[06/23 15:59:17.042181] INFO: thread[1] has currently completed queries: 9931, QPS: 55.139[06/23 15:59:17.079183] INFO: thread[2] has currently completed queries: 11653, QPS: 64.686[06/23 15:59:23.204856] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.1990s[06/23 15:59:23.588289] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.4260s[06/23 15:59:26.897799] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.4820s[06/23 15:59:27.305009] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.2680s[06/23 15:59:27.927708] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.3060s[06/23 15:59:35.304509] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.1000s[06/23 15:59:36.224949] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.6360s[06/23 15:59:40.658812] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.7610s[06/23 15:59:43.117408] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.8120s[06/23 15:59:43.584076] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:15.6570s[06/23 15:59:46.998960] INFO: thread[0] has currently completed queries: 11670, QPS: 55.554[06/23 15:59:47.026291] INFO: thread[4] has currently completed queries: 14896, QPS: 70.902[06/23 15:59:47.035069] INFO: thread[3] has currently completed queries: 14879, QPS: 70.818[06/23 15:59:47.055110] INFO: thread[1] has currently completed queries: 11631, QPS: 55.353[06/23 15:59:47.117361] INFO: thread[2] has currently completed queries: 13581, QPS: 64.615[06/23 15:59:47.813407] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.5090s[06/23 15:59:48.530546] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.3050s[06/23 15:59:54.360848] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.7020s[06/23 15:59:59.613625] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:16.4960s[06/23 16:00:00.209528] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:16.6250s[06/23 16:00:00.379573] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.5660s[06/23 16:00:01.540119] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:13.0100s[06/23 16:00:07.864559] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.5040s[06/23 16:00:12.572445] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:12.1930s[06/23 16:00:13.713394] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:12.1730s[06/23 16:00:15.147517] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:15.5340s[06/23 16:00:15.204324] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:14.9950s[06/23 16:00:17.001404] INFO: thread[0] has currently completed queries: 13332, QPS: 55.534[06/23 16:00:17.031175] INFO: thread[4] has currently completed queries: 17047, QPS: 71.000[06/23 16:00:17.041008] INFO: thread[3] has currently completed queries: 17005, QPS: 70.822[06/23 16:00:17.057005] INFO: thread[1] has currently completed queries: 13331, QPS: 55.517[06/23 16:00:17.129278] INFO: thread[2] has currently completed queries: 15597, QPS: 64.934[06/23 16:00:20.918288] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:13.0540s[06/23 16:00:24.169922] INFO: thread[4] complete all sqls to allocate all sub-tables[1764 - 2203] once queries duration:11.5970s[06/23 16:00:24.723675] INFO: thread[3] complete all sqls to allocate all sub-tables[1323 - 1763] once queries duration:11.0100s[06/23 16:00:29.377737] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:14.1730s[06/23 16:00:29.435308] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:14.2880s[06/23 16:00:31.775970] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:10.8570s[06/23 16:00:41.958418] INFO: thread[2] complete all sqls to allocate all sub-tables[882 - 1322] once queries duration:10.1820s[06/23 16:00:42.163680] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:12.7280s[06/23 16:00:42.360845] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:12.9830s[06/23 16:00:47.006939] INFO: thread[0] has currently completed queries: 15361, QPS: 56.877[06/23 16:00:47.073863] INFO: thread[1] has currently completed queries: 15353, QPS: 56.833[06/23 16:00:53.858526] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:11.6950s[06/23 16:00:54.201840] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:11.8410s[06/23 16:01:06.349354] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:12.1480s[06/23 16:01:06.422182] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:12.5640s[06/23 16:01:17.034805] INFO: thread[0] has currently completed queries: 17509, QPS: 58.343[06/23 16:01:17.090599] INFO: thread[1] has currently completed queries: 17481, QPS: 58.239[06/23 16:01:18.832644] INFO: thread[0] complete all sqls to allocate all sub-tables[0 - 440] once queries duration:12.4100s[06/23 16:01:19.128519] INFO: thread[1] complete all sqls to allocate all sub-tables[441 - 881] once queries duration:12.7790s[06/23 16:01:19.128796] INFO: Spend 302.1980 second completed total queries: 88160, the QPS of all threads: 291.729 </code></pre></details> <p>QPS影响不大，219</p><p>RDS测试使用mysql自带mysqlslap</p><p>并发、SQL语句保持与TD一致</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">FROM</span> tb_userlocation_td3 <span class="hljs-keyword">WHERE</span> locationtime <span class="hljs-operator">></span> DATE_ADD(NOW(),<span class="hljs-type">INTERVAL</span> <span class="hljs-number">-750</span> <span class="hljs-keyword">DAY</span>) <span class="hljs-keyword">AND</span> <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> locationtime LIMIT <span class="hljs-number">1000</span>;<br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">FROM</span> tb_userlocation_td3 <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> locationtime <span class="hljs-keyword">DESC</span> LIMIT <span class="hljs-number">1</span>;<br></code></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">mysqlslap -hpc-bp1zip05gl1b1ga3v.rwlb.rds.aliyuncs.com --create-schema=locationcenter -uuser_service -pLunz2017 --number-of-queries=200 --iterations=5 --concurrency=200 --engine=innodb --query=/opt/userlocationslap.sql --verbose<br></code></pre></td></tr></table></figure><p>第一次压测结果，WHERE 条件中不加loginname筛选条件</p><details> <summary>展开查看测试结果</summary> <pre><code class="hljs">Benchmark Running for engine innodb Average number of seconds to run all queries: 0.163 seconds Minimum number of seconds to run all queries: 0.141 seconds Maximum number of seconds to run all queries: 0.216 seconds Number of clients running queries: 5 Average number of queries per client: 40 </code></pre></details> <p><img src="https://picwong.oss-cn-qingdao.aliyuncs.com/zr.wongcw/202206231714349.png" alt="image-20220623171426893"></p><p>换算为QPS后，大概 235左右</p><p>第二次压测，SQL脚本为：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">FROM</span> tb_userlocation_td3 <span class="hljs-keyword">WHERE</span> locationtime <span class="hljs-operator">></span> DATE_ADD(NOW(),<span class="hljs-type">INTERVAL</span> <span class="hljs-number">-750</span> <span class="hljs-keyword">DAY</span>) <span class="hljs-keyword">AND</span> loginname<span class="hljs-operator">=</span><span class="hljs-string">'ZR18080096'</span> <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> locationtime LIMIT <span class="hljs-number">1000</span>;<br><span class="hljs-keyword">SELECT</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">FROM</span> tb_userlocation_td3 <span class="hljs-keyword">WHERE</span> loginname<span class="hljs-operator">=</span><span class="hljs-string">'ZR18080096'</span> <span class="hljs-keyword">ORDER</span> <span class="hljs-keyword">BY</span> locationtime <span class="hljs-keyword">DESC</span> LIMIT <span class="hljs-number">1</span>;<br></code></pre></td></tr></table></figure><p>这样压测，RDS直接挂了，没结果。</p><p>RDS测试使用sysbench</p><p>工具优点是测试项全面，</p><p>安装工具</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum -y install sysbench<br><span class="hljs-meta prompt_">#</span><span class="language-bash">如果没有此包就安装三方源后再执行</span><br>yum -y install epel-release<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看版本</span><br>sysbench --version<br></code></pre></td></tr></table></figure><p>因工具不能指定查询语句，只能添加测试表测试了</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">准备测试脚本</span><br>mysqladmin -hpc-bp1zip05gl1b1ga3v.rwlb.rds.aliyuncs.com -uuser_service -pLunz2017 -P3306 create sbtest;<br><br>sysbench --mysql-host=pc-bp1zip05gl1b1ga3v.rwlb.rds.aliyuncs.com \<br> --mysql-port=3306 \<br> --mysql-user=user_service \<br> --mysql-password=Lunz2017 \<br> --mysql-db=locationcenter \<br> /usr/share/sysbench/oltp_common.lua \<br> --tables=10 \<br> --table_size=100000 \<br> prepare<br><span class="hljs-meta prompt_">#</span><span class="language-bash">压测脚本</span><br>sysbench --db-driver=mysql --time=30 --threads=5 --report-interval=1 \<br> --mysql-host=pc-bp1zip05gl1b1ga3v.rwlb.rds.aliyuncs.com \<br> --mysql-port=3306 \<br> --mysql-user=user_service \<br> --mysql-password=Lunz2017 \<br> --mysql-db=locationcenter --tables=10 --table_size=1000000 \<br> /usr/share/sysbench/oltp_point_select.lua \<br> --db-ps-mode=disable run<br><span class="hljs-meta prompt_">#</span><span class="language-bash">清理测试数据</span><br>sysbench --db-driver=mysql --time=30 --threads=5 --report-interval=1 \<br> --mysql-host=pc-bp1zip05gl1b1ga3v.rwlb.rds.aliyuncs.com \<br> --mysql-port=3306 \<br> --mysql-user=user_service \<br> --mysql-password=Lunz2017 \<br> --mysql-db=locationcenter --tables=10 --table_size=1000000 \<br> oltp_read_only --db-ps-mode=disable cleanup<br></code></pre></td></tr></table></figure><p>压测结果：</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs log">SQL statistics:<br> queries performed:<br> read: 429542<br> write: 0<br> other: 0<br> total: 429542<br> transactions: 429542 (14316.25 per sec.)<br> queries: 429542 (14316.25 per sec.)<br> ignored errors: 0 (0.00 per sec.)<br> reconnects: 0 (0.00 per sec.)<br>General statistics:<br> total time: 30.0025s<br> total number of events: 429542<br>Latency (ms):<br> min: 0.23<br> avg: 0.35<br> max: 42.47<br> 95th percentile: 0.43<br> sum: 149842.12<br>Threads fairness:<br> events (avg/stddev): 85908.4000/2960.38<br> execution time (avg/stddev): 29.9684/0.00<br></code></pre></td></tr></table></figure><h2 id="应用2-三方监控"><a href="#应用2-三方监控" class="headerlink" title="应用2 三方监控"></a>应用2 三方监控</h2><p>StatsD监控网络，collectd监控服务器硬件，配合garfna前台展示，均使用TDengine存储数据</p><h3 id="安装grafana监控"><a href="#安装grafana监控" class="headerlink" title="安装grafana监控"></a>安装grafana监控</h3><p>安装包下载地址：<a href="https://cloud.189.cn/t/7bUb2eBRfM7f%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9A4bdl%EF%BC%89">https://cloud.189.cn/t/7bUb2eBRfM7f（访问码：4bdl）</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo yum install grafana-8.5.3-1.x86_64.rpm<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看cli版本</span> <br>grafana-cli -V<br><span class="hljs-meta prompt_">#</span><span class="language-bash">安装td插件</span><br>sudo -u grafana grafana-cli \<br> --pluginUrl https://github.com/taosdata/grafanaplugin/releases/download/v3.1.3/tdengine-datasource-3.1.3.zip \<br> plugins install tdengine-datasource<br><span class="hljs-meta prompt_">#</span><span class="language-bash">编辑配置文件</span><br>/etc/grafana/grafana.ini<br>[plugins]<br>allow_loading_unsigned_plugins = tdengine-datasource<br><span class="hljs-meta prompt_">#</span><span class="language-bash">重启服务</span><br>sudo systemctl restart grafana-server<br><span class="hljs-meta prompt_">#</span><span class="language-bash">如果启动失败，就执行下这个</span><br>yum install polkit -y<br></code></pre></td></tr></table></figure><p>配合TDinsight监控数据库<br>配置及监控项解释： <a href="https://docs.taosdata.com/reference/tdinsight/">https://docs.taosdata.com/reference/tdinsight/</a><br>监控脚本下载地址（内网机器可能加载不了脚本，json格式上传）：<a href="https://cloud.189.cn/t/nQrEJzAvmiYj%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Ab3ea%EF%BC%89">https://cloud.189.cn/t/nQrEJzAvmiYj（访问码：b3ea）</a></p><h3 id="StatsD安装"><a href="#StatsD安装" class="headerlink" title="StatsD安装"></a>StatsD安装</h3><p>下载安装包</p><p><a href="https://github.com/statsd/statsd">https://github.com/statsd/statsd</a></p><p>云：<a href="https://cloud.189.cn/t/AFnEzuAfENVb%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9A9els%EF%BC%89">https://cloud.189.cn/t/AFnEzuAfENVb（访问码：9els）</a></p><p>安装nodejs</p><p><a href="https://nodejs.org/en/download">https://nodejs.org/en/download</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">解压</span><br>xz -d node-xxxx.tar.xz<br>tar -xvf node-xxxx.tar<br><span class="hljs-meta prompt_">#</span><span class="language-bash">移动到文件夹</span><br>mv node-xxx /usr/local/node<br><span class="hljs-meta prompt_">#</span><span class="language-bash">建立软连接</span><br>ln -s /usr/local/node/bin/node /usr/bin/node<br>ln -s /usr/local/node/bin/npm /usr/bin/npm<br><span class="hljs-meta prompt_">#</span><span class="language-bash">验证环境变量</span><br>node -v<br>npm -v<br><span class="hljs-meta prompt_">#</span><span class="language-bash">配置安装路径</span><br>cd /usr/local/node/<br>mkdir -p node_global<br>mkdir -p node_cache<br>npm config set prefix "node_global"<br>npm config set cache "node_cache"<br><span class="hljs-meta prompt_">#</span><span class="language-bash">npm慢的话装个cnmp</span><br>npm install cnpm -g --registry=https://registry.npm.taobao.org<br><span class="hljs-meta prompt_">#</span><span class="language-bash">安装依赖</span><br>npm install<br></code></pre></td></tr></table></figure><p>建账号、库</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">USER</span> user_mo PASS <span class="hljs-string">'Lunz2017'</span>;<br><span class="hljs-keyword">ALTER</span> <span class="hljs-keyword">USER</span> user_mo PRIVILEGE write;<br><span class="hljs-keyword">CREATE</span> DATABASE IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> statsd <span class="hljs-keyword">UPDATE</span> <span class="hljs-number">2</span>;<br><span class="hljs-keyword">CREATE</span> DATABASE IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> collectd <span class="hljs-keyword">UPDATE</span> <span class="hljs-number">2</span>;<br><span class="hljs-keyword">CREATE</span> DATABASE IF <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">EXISTS</span> telegraf <span class="hljs-keyword">UPDATE</span> <span class="hljs-number">2</span>;<br></code></pre></td></tr></table></figure><p>编辑TD配置文件</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs shel">vim /etc/taos/taosadapter.toml<br><br>[statsd]<br>enable = true<br>port = 6044<br>db = "statsd"<br>user = "user_mo"<br>password = "Lunz2017"<br>worker = 10<br>gatherInterval = "5s"<br>protocol = "udp"<br>maxTCPConnections = 250<br>tcpKeepAlive = false<br>allowPendingMessages = 50000<br>deleteCounters = true<br>deleteGauges = true<br>deleteSets = true<br>deleteTimings = true<br></code></pre></td></tr></table></figure><p>编辑statd配置文件，假设文件夹位置 /opt/statsd</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">cd /opt/statsd<br>cp exampleConfig.js tdconfig.js<br>vim /opt/statsd/tdconfig.js<br></code></pre></td></tr></table></figure><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs js">{<br><span class="hljs-attr">port</span>: <span class="hljs-number">8125</span><br>, <span class="hljs-attr">backends</span>: [<span class="hljs-string">"./backends/repeater"</span>]<br>, <span class="hljs-attr">repeater</span>: [{ <span class="hljs-attr">host</span>: <span class="hljs-string">'192.168.200.99'</span>, <span class="hljs-attr">port</span>: <span class="hljs-number">6044</span>}]<br>}<br></code></pre></td></tr></table></figure><p>启动</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">node stats.js tdconfig.js &<br></code></pre></td></tr></table></figure><p>测试</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum install nc<br>echo "foo:1|c" | nc -u -w1 192.168.200.99 8125<br></code></pre></td></tr></table></figure><h3 id="collectd-安装"><a href="#collectd-安装" class="headerlink" title="collectd 安装"></a>collectd 安装</h3><p>下载安装包</p><p><a href="https://cloud.189.cn/t/bM3eAzAjQv2y%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9A4mew%EF%BC%89">https://cloud.189.cn/t/bM3eAzAjQv2y（访问码：4mew）</a></p><p>编译安装 略</p><p>配置接收直接采集插件数据</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">vim /opt/collectd/etc/collectd.conf<br><span class="hljs-meta prompt_">#</span><span class="language-bash">td官方文档默认编译安装位置 /etc/collectd/collectd.conf 验证非此文件</span><br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs conf">LoadPlugin network<br><Plugin network><br> Server "192.168.200.99" "6045"<br></Plugin><br></code></pre></td></tr></table></figure><p>配置write_tsdb 插件数据</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">vim /opt/collectd/etc/collectd.conf<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs conf">LoadPlugin write_tsdb<br><Plugin write_tsdb><br> <Node><br> Host "192.168.200.99"<br> Port "6047"<br> HostTags "status=production"<br> StoreRates false<br> AlwaysAppendDS false<br> </Node><br></Plugin><br></code></pre></td></tr></table></figure><p>编辑配置文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">vim /etc/taos/taosadapter.toml<br></code></pre></td></tr></table></figure><figure class="highlight toml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs toml"><span class="hljs-section">[opentsdb_telnet]</span><br><span class="hljs-attr">enable</span> = <span class="hljs-literal">true</span><br><span class="hljs-attr">maxTCPConnections</span> = <span class="hljs-number">250</span><br><span class="hljs-attr">tcpKeepAlive</span> = <span class="hljs-literal">false</span><br><span class="hljs-attr">dbs</span> = [<span class="hljs-string">"opentsdb_telnet"</span>, <span class="hljs-string">"collectd"</span>, <span class="hljs-string">"icinga2"</span>, <span class="hljs-string">"tcollector"</span>]<br><span class="hljs-attr">ports</span> = [<span class="hljs-number">6046</span>, <span class="hljs-number">6047</span>, <span class="hljs-number">6048</span>, <span class="hljs-number">6049</span>]<br><span class="hljs-attr">user</span> = <span class="hljs-string">"user_mo"</span><br><span class="hljs-attr">password</span> = <span class="hljs-string">"Lunz2017"</span><br></code></pre></td></tr></table></figure><p>建立服务软连接</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">环境变量</span><br>ln -s /opt/collectd/sbin/collectd /usr/sbin/collectd<br><span class="hljs-meta prompt_">#</span><span class="language-bash">注册服务 这玩意不好使直接去库里验证数据吧</span><br>ln -s /opt/collectd/sbin/collectd /etc/init.d/collectd<br></code></pre></td></tr></table></figure><p>启动服务</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">collectd<br>ps -ef|grep collectd<br></code></pre></td></tr></table></figure><p>连接数据库查看collectd库内数据</p><p>验证存入数据</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql">use collectd;<br><span class="hljs-keyword">select</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">from</span> cpu_value limit <span class="hljs-number">10</span>;<br></code></pre></td></tr></table></figure><h2 id="应用3-另一种监控"><a href="#应用3-另一种监控" class="headerlink" title="应用3 另一种监控"></a>应用3 另一种监控</h2><p>安装 Telegraf 监控系统</p><p>添加镜像源</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs shell">cat <<EOF | sudo tee /etc/yum.repos.d/influxdata.repo<br>[influxdata]<br>name = InfluxData Repository - Stable<br>baseurl = https://repos.influxdata.com/stable/\$basearch/main<br>enabled = 1<br>gpgcheck = 1<br>gpgkey = https://repos.influxdata.com/influxdb.key<br>EOF<br></code></pre></td></tr></table></figure><p>安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo yum install telegraf<br></code></pre></td></tr></table></figure><p>rpm安装：</p><p><a href="https://cloud.189.cn/t/NZVVjaiYrYve%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Ait2m%EF%BC%89">https://cloud.189.cn/t/NZVVjaiYrYve（访问码：it2m）</a></p><p>修改配置文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">/etc/telegraf/telegraf.conf<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs conf">[[outputs.http]]<br> url = "http://192.168.200.99:6041/influxdb/v1/write?db=telegraf"<br> method = "POST"<br> timeout = "5s"<br> username = "user_mo"<br> password = "Lunz2017"<br> data_format = "influx"<br> influx_max_line_bytes = 250<br></code></pre></td></tr></table></figure><p>启动</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">sudo systemctl restart telegraf<br></code></pre></td></tr></table></figure><p>验证存入数据</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs sql">use telegraf;<br><span class="hljs-keyword">select</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">from</span> cpu limit <span class="hljs-number">10</span>;<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>数据库</category>
<category>TDEngine</category>
</categories>
<tags>
<tag>TDEngine</tag>
<tag>时序数据库</tag>
</tags>
</entry>
<entry>
<title>DataX-MySQL2MongoDB案例</title>
<link href="/2022/05/31/DataX-MySQL2MongoDB%E6%A1%88%E4%BE%8B/"/>
<url>/2022/05/31/DataX-MySQL2MongoDB%E6%A1%88%E4%BE%8B/</url>
<content type="html"><![CDATA[<h3 id="MySQL2MongoDB"><a href="#MySQL2MongoDB" class="headerlink" title="MySQL2MongoDB"></a>MySQL2MongoDB</h3><p>1.Josn配置文件可能加载失败，需要核对各参数，严格按照官方样例写参数，即使是Json格式也可能解析失败！</p><p>2.源库单表数据量过大时一定要加主键，加分片键 “<strong>splitPk</strong>“，分片键必需为主键、int整数型，此键可以不包含在所传输字段中；</p><p>3.报错：</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">Exception</span> <span class="hljs-keyword">in</span> thread "taskGroup-0" com.alibaba.datax.common.<span class="hljs-keyword">exception</span>.DataXException: Code:[DBUtilErrorCode<span class="hljs-number">-07</span>], Description:[读取数据库数据失败. 请检查您的配置的 <span class="hljs-keyword">column</span>/<span class="hljs-keyword">table</span>/<span class="hljs-keyword">where</span>/querySql或者向 DBA 寻求帮助.]. - 执行的<span class="hljs-keyword">SQL</span>为: <span class="hljs-keyword">select</span> WorkOrderId,CustStoreId,CustStoreName,CustStoreCode,PreCustStoreId,PreCustStoreName,CustSettleId,CustSettleName,IsCustomer,CustCoopType,ProCode,AppCode,ProName,CityCode,CityName,AreaCode,AreaName,InstallAddress,InstallTime,RequiredTime,LinkMan,LinkTel,SourceType,SecondLinkTel,SecondLinkMan,WarehouseId,WarehouseName,Remark,IsUrgent,CustUniqueSign,CreatePersonCode,CreatePersonName,EffectiveTime,OrderType,EffectiveSuccessfulTime,CreatedById,CreatedAt,UpdatedById,UpdatedAt,DeletedById,DeletedAt,Deleted,AppointStatus,ServiceCode,CreateType,ServiceName,RecCompleteTime,RecRecordPersonCode,RecRecordPersonName,WfOrderCompletetime,WfAssignor,WfDirectOrderAccept,WorkOdrTypeStatus,WorkOdrCloseStatusName,WorkOdrSuspendStatus,WorkStatus,WorkOdrClosedAt,AFCCreatedAt,AFCRetreatTime,AFCFollowFrequency,AFCIfRemove,AFCReturnedTypeCode,VHSAppraisalStatus,VHSReceiverLoginName,OperatingInfo,OperatedInfo,ServiceProviderCode,WorkflowInfo,WorkServiceSubject,AFCWorkFinanceRecord,AFCPaymentReport,VHSWorkFileReceiveInfo,CustomerId,CustomerName <span class="hljs-keyword">from</span> workjsoninfo <span class="hljs-keyword">where</span> (<span class="hljs-number">169036</span> <= Id <span class="hljs-keyword">AND</span> Id < <span class="hljs-number">338071</span>) 具体错误信息为：com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Application was streaming results <span class="hljs-keyword">when</span> the <span class="hljs-keyword">connection</span> failed. Consider raising <span class="hljs-keyword">value</span> <span class="hljs-keyword">of</span> <span class="hljs-string">'net_write_timeout'</span> <span class="hljs-keyword">on</span> the <span class="hljs-keyword">server</span>.<br></code></pre></td></tr></table></figure><p>解决方法：</p><p>增加源端 wait_timeout 参数值。</p><p>NULL值处理</p><p>问题：mysql2mongo数据库传输，不识别mysql内josn字段，因为datax会给json增加转义字符 “"</p><p>解决方法:源端字段类型改为varchar，入目标为array类型，mongodbwriter需要指定特殊解析数组分隔符；</p><h3 id="注意事项"><a href="#注意事项" class="headerlink" title="注意事项"></a>注意事项</h3><p>1.单 executor执行速度有限，尽量开多管道；</p>]]></content>
<categories>
<category>ETL工具</category>
<category>离线同步</category>
</categories>
<tags>
<tag>ETL</tag>
<tag>DataX</tag>
<tag>MySQL</tag>
<tag>MongoDB</tag>
</tags>
</entry>
<entry>
<title>DataHub安装</title>
<link href="/2022/05/25/DataHub%E5%AE%89%E8%A3%85/"/>
<url>/2022/05/25/DataHub%E5%AE%89%E8%A3%85/</url>
<content type="html"><![CDATA[<h1 id="产品介绍"><a href="#产品介绍" class="headerlink" title="产品介绍"></a>产品介绍</h1><p>前身为LinkedIn 的数据团队开发的 WhereHow，WhereHow 围绕着任何一个大数据系统中最重要的组成部分数据本身构建了一整套关于数据处理流程、使用人员和相关洞见的集中式平台和相应网页。</p><p>官网地址：<a href="https://datahubproject.io/">https://datahubproject.io/</a></p><p>Demo地址：<a href="https://demo.datahubproject.io/">https://demo.datahubproject.io/</a></p><p>github开源地址：<a href="https://github.com/datahub-project/datahub">https://github.com/datahub-project/datahub</a></p><p>CSDN介绍链接：<a href="https://blog.csdn.net/ZYC88888/article/details/119213446">https://blog.csdn.net/ZYC88888/article/details/119213446</a></p><p>CSDN专栏介绍：<a href="https://cloud.189.cn/t/YZ3Ejyn6nUzq%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9A9uhx%EF%BC%89">https://cloud.189.cn/t/YZ3Ejyn6nUzq（访问码：9uhx）</a></p><p>各元数据管理系统实现功能对比：</p><p><img src="https://img-blog.csdnimg.cn/20210729163327921.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1pZQzg4ODg4,size_16,color_FFFFFF,t_70" alt="img"></p><h1 id="使用场景"><a href="#使用场景" class="headerlink" title="使用场景"></a>使用场景</h1><ol><li>元数据收集、发现，结构变更通知；</li><li>数据血缘，开源版暂时不支持字段级血缘关系，不支持自动发现，需自己写脚本解析执行计划收集表字段关系；脚本样例：<a href="https://datahubproject.io/docs/lineage/sample_code">https://datahubproject.io/docs/lineage/sample_code</a></li><li>API接口获取元数据</li><li>标签管理</li><li>业务域划分</li><li>权限控制</li></ol><h1 id="部署方案"><a href="#部署方案" class="headerlink" title="部署方案"></a>部署方案</h1><h3 id="环境要求"><a href="#环境要求" class="headerlink" title="环境要求"></a>环境要求</h3><p>uname -a #CENTOS 7.9&8.5 安装成功，没特殊要求，其他发行版指令、依赖包不一样<br>python3 -V #0.8以后要求python版本大于3.6<br>java version #1.8<br>docker version #Docker version 20.10.16</p><p>部署附件：</p><p><a href="https://cloud.189.cn/t/6RNFNbYVzM3y%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Auqk6%EF%BC%89">https://cloud.189.cn/t/6RNFNbYVzM3y（访问码：uqk6）</a></p><p>下面这仨先不用下载</p><p><a href="https://cloud.189.cn/t/Fb6jIrZbEFzq%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Aoar2%EF%BC%89">https://cloud.189.cn/t/Fb6jIrZbEFzq（访问码：oar2）</a></p><p><a href="https://cloud.189.cn/t/yuuQZfryiUNf%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Aqym5%EF%BC%89">https://cloud.189.cn/t/yuuQZfryiUNf（访问码：qym5）</a></p><p><a href="https://cloud.189.cn/t/FR7v2mbee2y2%EF%BC%88%E8%AE%BF%E9%97%AE%E7%A0%81%EF%BC%9Amv0m%EF%BC%89">https://cloud.189.cn/t/FR7v2mbee2y2（访问码：mv0m）</a></p><h3 id="安装依赖包"><a href="#安装依赖包" class="headerlink" title="安装依赖包"></a>安装依赖包</h3><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs cmake">yum -y <span class="hljs-keyword">install</span> libffi-devel<br>yum -y <span class="hljs-keyword">install</span> zlib*<br>yum -y <span class="hljs-keyword">install</span> gcc<br>yum -y <span class="hljs-keyword">install</span> git<br>yum -y <span class="hljs-keyword">install</span> python3-devel<br>yum -y <span class="hljs-keyword">install</span> cyrus-sasl-devel<br></code></pre></td></tr></table></figure><p>安装必要插件时使用</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum groupinstall "Development tools"<br>yum install openldap-devel python-devel<br></code></pre></td></tr></table></figure><p>python 安装好后安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install pyopenssl ndg-httpsclient pyasn1<br>pip install -U urllib3<br>pip install -U channels<br>pip install requests[security]<br>pip3 install toml<br>pip --cert /user/local/cacert.pem install -U channels<br></code></pre></td></tr></table></figure><p>python编译安装 （python3 -V 大于3.6的忽略）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs shell">cd /opt/module/Python-3.8.0<br><span class="hljs-meta prompt_">#</span><span class="language-bash">编译安装</span><br>./configure --prefix=/usr/local/python3<br>sudo make && sudo make install<br><span class="hljs-meta prompt_">#</span><span class="language-bash">创建软连接</span><br>ln -s /usr/local/python3/bin/python3 /usr/bin/python3<br>ln -s /usr/local/python3/bin/pip3 /usr/local/bin/pip3<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看版本</span><br>python3 -V<br></code></pre></td></tr></table></figure><p>great_expectations 插件安装 （root环境执行pip指令会有提示权限问题，忽略即可，也可以进入虚拟环境运行）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">python -m venv /usr/bin/venv<br>source usr/bin/activate<br>pip install great_expectations<br></code></pre></td></tr></table></figure><p>下载安装docker-compose、jq</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell">wget https://github.com/docker/compose/releases/download/v2.5.0/docker-compose-linux-x86_64<br>wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64<br>mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose<br>mv jq-linux64 /usr/local/bin/jq<br>chmod +x /usr/local/bin/docker-compose<br>chmod +x /usr/local/bin/jq<br></code></pre></td></tr></table></figure><h3 id="部署"><a href="#部署" class="headerlink" title="部署"></a>部署</h3><p>复制后台安装包</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">git clone git://github.com/linkedin/datahub.git<br></code></pre></td></tr></table></figure><p>有时会连接失败，可以用上面网盘下载下来的安装包使用sftp上传至根目录</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs shell">mkdir -p /opt/datahub<br><span class="hljs-meta prompt_">#</span><span class="language-bash">解压至指定文件夹</span><br>tar -zxvf datahub-0.8.34.tar.gz -C /opt/datahub<br></code></pre></td></tr></table></figure><p>后台初始化运行/shell 运行</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">cd /opt/datahub/docker<br>mkdir logs<br>touch logs/quickstart.log<br>nohup ./quickstart.sh > logs/quickstart.log 2>&1 &<br>tail -200f logs/quickstart.log<br></code></pre></td></tr></table></figure><p>安装前台</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install --upgrade pip wheel setuptools<br>pip uninstall datahub acryl-datahub || true<br>pip install --upgrade acryl-datahub<br><br>datahub version<br></code></pre></td></tr></table></figure><p>检查前台运行情况</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip show acryl_datahub<br></code></pre></td></tr></table></figure><p>docker方式启动(上面是shell运行方式，两者不能同时跑) 后面可以指定启动配置文件，默认没有图数据库组件，即没血缘关系</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">datahub docker quickstart --quickstart-compose-file /opt/datahub/docker/quickstart/docker-compose.quickstart.yml<br></code></pre></td></tr></table></figure><p>检查各组件运行情况</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}\t{{.Status}}"<br></code></pre></td></tr></table></figure><blockquote><p>CONTAINER ID NAMES PORTS STATUS<br>feccca9dc28d datahub-frontend-react 0.0.0.0:9002->9002/tcp, :::9002->9002/tcp Up 2 hours (healthy)<br>cb58e6dd06c9 datahub-datahub-actions-1 Up 2 hours<br>fdc533e04c62 schema-registry 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp Up 2 hours<br>3d6985bdb797 broker 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:29092->29092/tcp, :::29092->29092/tcp Up 2 hours<br>e8552b179b85 datahub-gms 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp Up 2 hours (healthy)<br>dd45ab58216d mysql 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp Up 2 hours<br>8ce0c1069530 zookeeper 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp Up 2 hours<br>0c54c0cea09c neo4j 0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp Up 2 hours<br>87c2617c36da elasticsearch 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp Up 2 hours (healthy)</p></blockquote><p>#检查插件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">datahub check plugins<br></code></pre></td></tr></table></figure><p>安装数据源插件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install 'acryl-datahub[mysql]'<br>pip install 'acryl-datahub[bigquery]'<br>pip install 'acryl-datahub[bigquery-usage]'<br>pip install 'acryl-datahub[druid]'<br>pip install 'acryl-datahub[hive]'<br>pip install 'acryl-datahub[kafka]'<br>pip install 'acryl-datahub[kafka-connect]'<br>pip install 'acryl-datahub[metabase]'<br>pip install 'acryl-datahub[mongodb]'<br>pip install 'acryl-datahub[mssql]'<br>pip install 'acryl-datahub[mariadb]'<br>pip install 'acryl-datahub[openapi]'<br>pip install 'acryl-datahub[oracle]'<br>pip install 'acryl-datahub[postgres]'<br>pip install 'acryl-datahub[sqlalchemy]'<br></code></pre></td></tr></table></figure><p>目标端插件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install 'acryl-datahub[datahub-rest]'<br>pip install 'acryl-datahub[datahub-kafka]'<br></code></pre></td></tr></table></figure><p>airflow组件（暂时用不到，crontab可实现定时更新元数据）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install acryl-datahub[airflow]<br></code></pre></td></tr></table></figure><p><font color=red>降级great_expectations</font></p><blockquote><p>python 包great_expectation升级至最新版本后datahub未适配，必需降级至0.14.0</p></blockquote><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install --upgrade great-expectations==0.14.0<br></code></pre></td></tr></table></figure><p>血缘关系开启（未成功）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs SHELL"><span class="hljs-meta prompt_"># </span><span class="language-bash">For REST-based:</span><br>airflow connections add --conn-type 'datahub_rest' 'datahub_rest_default' --conn-host 'http://localhost:8080'<br><span class="hljs-meta prompt_"># </span><span class="language-bash">For Kafka-based (standard Kafka sink config can be passed via extras):</span><br>airflow connections add --conn-type 'datahub_kafka' 'datahub_kafka_default' --conn-host 'broker:9092' --conn-extra '{}'<br></code></pre></td></tr></table></figure><p>添加配置文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">vim ./airflow/airflow.cfg<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs cfg">[lineage]<br>backend = datahub_provider.lineage.datahub.DatahubLineageBackend<br>datahub_kwargs = {<br> "datahub_conn_id": "datahub_rest_default",<br> "cluster": "prod",<br> "capture_ownership_info": true,<br> "capture_tags_info": true,<br> "graceful_exceptions": true }<br># The above indentation is important!<br></code></pre></td></tr></table></figure><p>导入MySQL元数据</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">vim /opt/datahub/docker/ingestion/mysql_datahub.yml<br></code></pre></td></tr></table></figure><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs yaml"><span class="hljs-attr">source:</span><br> <span class="hljs-attr">type:</span> <span class="hljs-string">"mysql"</span><br> <span class="hljs-attr">config:</span><br> <span class="hljs-attr">username:</span> <span class="hljs-string">"user_xxx"</span><br> <span class="hljs-attr">password:</span> <span class="hljs-string">"xxxxx"</span><br> <span class="hljs-attr">database:</span> <span class="hljs-string">"yourdatabase"</span><br> <span class="hljs-attr">host_port:</span> <span class="hljs-string">"10.10.1.10:3306"</span><br><span class="hljs-attr">sink:</span><br> <span class="hljs-attr">type:</span> <span class="hljs-string">"datahub-rest"</span><br> <span class="hljs-attr">config:</span><br> <span class="hljs-attr">server:</span> <span class="hljs-string">'http://localhost:8080'</span><br></code></pre></td></tr></table></figure><p>导入操作</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">datahub ingest -c /opt/datahub/docker/ingestion/mysql_datahub.yml<br></code></pre></td></tr></table></figure><p>使用linux定时任务定时导入数据源，crontab 计算器<a href="https://www.matools.com/crontab">https://www.matools.com/crontab</a></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs shell"><span class="hljs-meta prompt_">#</span><span class="language-bash">编辑定时器</span><br>crontab -e<br><span class="hljs-meta prompt_">#</span><span class="language-bash">按 i 进入编辑模式 添加每3小时执行一次任务 后保存退出</span><br>0 0 0-3 * * ? datahub ingest -c /opt/datahub/docker/ingestion/mysql_datahub.yml<br><span class="hljs-meta prompt_">#</span><span class="language-bash">查看任务列表</span><br>crontab -l<br></code></pre></td></tr></table></figure><h3 id="查看"><a href="#查看" class="headerlink" title="查看"></a>查看</h3><p><a href="http://ip:9002/">http://ip:9002</a> （机器必需开启9002端口外网访问）</p><h3 id="销毁容器等"><a href="#销毁容器等" class="headerlink" title="销毁容器等"></a>销毁容器等</h3><p>关闭后台各docker容器 （保留数据）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">datahub docker nuke --keep-data<br></code></pre></td></tr></table></figure><p>docker关闭、删除所有容器</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">docker stop $(docker ps -a -q)<br>docker rm $(docker ps -aq)<br></code></pre></td></tr></table></figure><p>查看docker运行中项目日志</p><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs css">docker logs -f 容器ID<br>docker logs -f -t <span class="hljs-attr">--tail</span> <span class="hljs-number">100</span> 容器ID<br></code></pre></td></tr></table></figure><p>查看docker-compose运行日志</p><figure class="highlight r"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs r">docker<span class="hljs-operator">-</span>compose logs <span class="hljs-operator">-</span>f<br>docker<span class="hljs-operator">-</span>compose logs <br></code></pre></td></tr></table></figure><h1 id="使用指南-amp-FAQ"><a href="#使用指南-amp-FAQ" class="headerlink" title="使用指南&FAQ"></a>使用指南&FAQ</h1><p>待补充</p>]]></content>
<categories>
<category>数据治理</category>
<category>数据发现工具</category>
</categories>
<tags>
<tag>数据治理</tag>
<tag>DataHub</tag>
</tags>
</entry>
<entry>
<title>Apache-Alats安装部署</title>
<link href="/2022/05/19/Apache-Alats%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/"/>
<url>/2022/05/19/Apache-Alats%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/</url>
<content type="html"><![CDATA[<h1 id="产品介绍"><a href="#产品介绍" class="headerlink" title="产品介绍"></a>产品介绍</h1><h1 id="使用场景"><a href="#使用场景" class="headerlink" title="使用场景"></a>使用场景</h1><h1 id="部署方案"><a href="#部署方案" class="headerlink" title="部署方案"></a>部署方案</h1><p>系统版本</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">uname -a<br>Linux iZbp1e4qug6kl2vcvhr5f2Z 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux<br></code></pre></td></tr></table></figure><p>依赖包安装</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">yum -y install maven<br></code></pre></td></tr></table></figure><h2 id="使用指南-着重案例介绍-基础类知识可直接附带链接"><a href="#使用指南-着重案例介绍-基础类知识可直接附带链接" class="headerlink" title="使用指南(着重案例介绍,基础类知识可直接附带链接)"></a>使用指南(着重案例介绍,基础类知识可直接附带链接)</h2><h2 id="FAQ"><a href="#FAQ" class="headerlink" title="FAQ"></a>FAQ</h2>]]></content>
<categories>
<category>数据治理</category>
<category>数据发现工具</category>
</categories>
<tags>
<tag>Alats</tag>
<tag>数据治理</tag>
</tags>
</entry>
<entry>
<title>ADB开发要点</title>
<link href="/2021/12/29/ADB%E5%BC%80%E5%8F%91%E8%A6%81%E7%82%B9/"/>
<url>/2021/12/29/ADB%E5%BC%80%E5%8F%91%E8%A6%81%E7%82%B9/</url>
<content type="html"><![CDATA[<h1 id="基础相关"><a href="#基础相关" class="headerlink" title="基础相关"></a>基础相关</h1><h2 id="数据类型"><a href="#数据类型" class="headerlink" title="数据类型"></a>数据类型</h2><p>1.支持JSON、POINT空间类型字段；</p><p>2.定义时可不带长度；</p><p>3.数据类型介绍详见：<a href="https://help.aliyun.com/document_detail/123577.html">https://help.aliyun.com/document_detail/123577.html</a></p><h2 id="SQL语句差异"><a href="#SQL语句差异" class="headerlink" title="SQL语句差异"></a>SQL语句差异</h2><p>1.DDL/DML/DQL/DCL语句支持程度</p><p>2.不支持项：不支持函数、过程</p><p>3.官方介绍： <a href="https://help.aliyun.com/document_detail/197270.html">https://help.aliyun.com/document_detail/197270.html</a></p><h2 id="系统函数"><a href="#系统函数" class="headerlink" title="系统函数"></a>系统函数</h2><p>1.各函数与MySQL对比，支持度、兼容性对比。</p><p>2.官方介绍：<a href="https://help.aliyun.com/document_detail/197391.html">https://help.aliyun.com/document_detail/197391.html</a></p><h2 id="可编程性替代方案"><a href="#可编程性替代方案" class="headerlink" title="可编程性替代方案"></a>可编程性替代方案</h2><p>1.DMS数仓开发;</p><p>2.DataWorks开发（舍弃）;</p><p>3.DMS数仓开发暂无介绍，后期补充开发要点。</p><h2 id="优化要点"><a href="#优化要点" class="headerlink" title="优化要点"></a>优化要点</h2><h3 id="建表前提"><a href="#建表前提" class="headerlink" title="建表前提"></a>建表前提</h3><p>1.选好分布键，最佳选择为参与JOIN的分区键而非主键或自增主键; DISTRIBUTED BY HASH(column_name,…)</p><p>2.分区键，PARTITION BY {VALUE(column_name) | VALUE(date_format(column_name, ?)}</p><p>3.COUNT 出来得字段必须用 BIGINT存储</p><p>4.数据类型区别 不带长度</p><h3 id="SQL优化"><a href="#SQL优化" class="headerlink" title="SQL优化"></a>SQL优化</h3><p>1.主键自增不会报错！！！</p><p>2.JOIN 子查询内尽量较少数据行数</p><p>3.中间层定时全量刷新，可使用物化视图</p><p>4.严谨SELECT *</p><p>5.分布键必须包含在主键定义内</p>]]></content>
<categories>
<category>数据库</category>
<category>AnalyticDB</category>
</categories>
<tags>
<tag>ADB</tag>
<tag>FAQ</tag>
</tags>
</entry>
<entry>
<title>ADB建模要点</title>
<link href="/2021/12/25/ADB%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5/"/>
<url>/2021/12/25/ADB%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5/</url>
<content type="html"><</p><p>所以，只能在配置DTS时选好ods层表结构，分区、分布键，表属性（维度表还是分区表）</p><p>同步后只能通过变更DTS配置来改变表结构;</p><p>大数据量表按时间分区，待测试能够在线变更表结构。</p><p>冷热分区、分区信息不能在线修改；</p>]]></content>
<categories>
<category>数据库</category>
<category>AnalyticDB</category>
</categories>
<tags>
<tag>ADB</tag>
<tag>建模</tag>
</tags>
</entry>
<entry>
<title>ADB建模要点</title>
<link href="/2021/12/22/ADB%E5%BB%BA%E6%A8%A1%E8%A6%81%E7%82%B9/"/>
<url>/2021/12/22/ADB%E5%BB%BA%E6%A8%A1%E8%A6%81%E7%82%B9/</url>
<content type="html"><![CDATA[<h2 id="可以使用navicat替代，但是建表语句需要更改部分内容；"><a href="#可以使用navicat替代，但是建表语句需要更改部分内容；" class="headerlink" title="可以使用navicat替代，但是建表语句需要更改部分内容；"></a><strong>可以使用navicat替代，但是建表语句需要更改部分内容；</strong></h2><h2 id="建表字段区别"><a href="#建表字段区别" class="headerlink" title="建表字段区别"></a>建表字段区别</h2><p>1.非空默认当前时间 使用 CURRENT_TIMESTAMP </p><p>2.建表语句后缀： </p><p><code>DISTRIBUTE BY HASH(Id) INDEX_ALL='Y' STORAGE_POLICY='HOT' BLOCK_SIZE=8192 COMMENT = 'cdm_仓储中心_仓库信息';</code></p><p>可根据实际情况调整内容</p><p>3.</p>]]></content>
<categories>
<category>数据库</category>
<category>AnalyticDB</category>
</categories>
<tags>
<tag>ADB</tag>
<tag>建模</tag>
</tags>
</entry>
<entry>
<title>ADB表创建参数及性能优化</title>
<link href="/2021/12/20/ADB%E8%A1%A8%E5%88%9B%E5%BB%BA%E5%8F%82%E6%95%B0%E5%8F%8A%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96/"/>
<url>/2021/12/20/ADB%E8%A1%A8%E5%88%9B%E5%BB%BA%E5%8F%82%E6%95%B0%E5%8F%8A%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96/</url>
<content type="html"><![CDATA[<h2 id="分布键调整（包含主键和查询键）"><a href="#分布键调整（包含主键和查询键）" class="headerlink" title="分布键调整（包含主键和查询键）"></a>分布键调整（包含主键和查询键）</h2><h3 id="创建表、写入数据"><a href="#创建表、写入数据" class="headerlink" title="创建表、写入数据"></a>创建表、写入数据</h3><p><code>DROP TABLE IF EXISTS tb_workorderinfo;</code><br><code>Create Table tb_workorderinfo (</code><br> <code>Id varchar NOT NULL COMMENT '主键(WI)',</code><br> <code>AppCode varchar COMMENT '申请编码',</code><br> <code>SourceType smallint COMMENT '工单来源(0工单下单1智车品2客户接口3LCRM下单4OCRM下单5平安私有云6金融SAAS推送7车务推送)',</code><br> <code>OrderType varchar COMMENT '工单类型(sys_ordertypeinfo.Code)',</code><br> <code>CreateType smallint COMMENT '工单建单方式(来源二级分类)(1批量导入)',</code><br> <code>WorkStatus smallint COMMENT '工单状态(0待提交1正在审核2审核不通过3正在寻找工程师4等待工程师接单5已拒单，重新寻找工程师6等待提交结果7等待客户确认8客户驳回，重新处理9已完成10关闭11质检12质检不通过)',</code><br> <code>ServiceProviderCode varchar COMMENT '服务供应商Code',</code><br> <code>CustomerId varchar COMMENT '客户Id',</code><br> <code>CustomerName varchar COMMENT '客户名称',</code><br> <code>CustStoreId varchar COMMENT '门店Id',</code><br> <code>CustStoreName varchar COMMENT '门店名称',</code><br> <code>CustStoreCode varchar COMMENT '门店Code',</code><br> <code>PreCustStoreId varchar COMMENT '客户接口下单门店Id',</code><br> <code>PreCustStoreName varchar COMMENT '客户接口下单门店名称',</code><br> <code>CustSettleId varchar COMMENT '结算单位Id',</code><br> <code>CustSettleName varchar COMMENT '结算单位名称',</code><br> <code>IsCustomer tinyint NOT NULL COMMENT '是否为客户(0否1是)',</code><br> <code>CustCoopType smallint COMMENT '客户是否有合作商品(1中瑞2非中瑞3中瑞+非中瑞)',</code><br> <code>ProCode varchar COMMENT '省份Code',</code><br> <code>ProName varchar COMMENT '省份名称',</code><br> <code>CityCode varchar COMMENT '城市Code',</code><br> <code>CityName varchar COMMENT '城市名称',</code><br> <code>AreaCode varchar COMMENT '地区Code',</code><br> <code>AreaName varchar COMMENT '地区名称',</code><br> <code>InstallAddress varchar COMMENT '加装地址',</code><br> <code>InstallTime datetime COMMENT '加装时间',</code><br> <code>RequiredTime datetime COMMENT '要求时间',</code><br> <code>LinkMan varchar COMMENT '联系人姓名',</code><br> <code>LinkTel varchar COMMENT '联系电话',</code><br> <code>SecondLinkTel varchar COMMENT '第二联系电话',</code><br> <code>SecondLinkMan varchar COMMENT '第二联系人',</code><br> <code>WarehouseId varchar COMMENT '发货一级仓Id',</code><br> <code>WarehouseName varchar COMMENT '发货一级仓名称',</code><br> <code>Remark varchar COMMENT '客户备注',</code><br> <code>IsUrgent tinyint NOT NULL DEFAULT '0' COMMENT '检修工单是否紧急(0否1是)',</code><br> <code>CustUniqueSign varchar COMMENT '客户唯一标识',</code><br> <code>CreatePersonCode varchar COMMENT '建单人Code',</code><br> <code>CreatePersonName varchar COMMENT '建单人姓名',</code><br> <code>EffectiveTime datetime COMMENT '工单生效时间',</code><br> <code>EffectiveSuccessfulTime datetime COMMENT '下单成功生效时间',</code><br> <code>CreatedById varchar COMMENT '',</code><br> <code>CreatedAt datetime COMMENT '',</code><br> <code>UpdatedById varchar COMMENT '',</code><br> <code>UpdatedAt datetime COMMENT '',</code><br> <code>DeletedById varchar COMMENT '',</code><br> <code>DeletedAt datetime COMMENT '',</code><br> <code>Deleted tinyint NOT NULL DEFAULT '0' COMMENT '',</code><br> <code>primary key (id)</code><br><code>) DISTRIBUTE BY HASH(id) INDEX_ALL='Y' STORAGE_POLICY='HOT' BLOCK_SIZE=8192</code><br><code>;</code></p><p><code>DROP TABLE IF EXISTS tb_workcarinfo;</code><br><code>Create Table tb_workcarinfo (</code><br> <code>Id varchar NOT NULL COMMENT '主键(WC)',</code><br> <code>WorkOrderId varchar COMMENT '工单主单Id(tb_workorderinfo.Id)',</code><br> <code>VinNumber varchar COMMENT '车架号',</code><br> <code>PlateNumber varchar COMMENT '车牌号',</code><br> <code>EngineNumber varchar COMMENT '发动机号',</code><br> <code>CarModelId varchar COMMENT '车型Id',</code><br> <code>CarModelName varchar COMMENT '车型名称',</code><br> <code>CarSeriesId varchar COMMENT '车系Id',</code><br> <code>CarSeriesName varchar COMMENT '车系名称',</code><br> <code>CarBrandId varchar COMMENT '车辆品牌Id',</code><br> <code>CarBrandName varchar COMMENT '车辆品牌名称',</code><br> <code>CarFullName varchar COMMENT '车型全称',</code><br> <code>Color varchar COMMENT '车辆颜色',</code><br> <code>CarPrice decimal(18, 2) COMMENT '车辆价格',</code><br> <code>IsNewCar smallint COMMENT '车辆类型(0二手车1新车)',</code><br> <code>CarType smallint COMMENT '车辆类型(1新车2二手车)',</code><br> <code>UserName varchar COMMENT '车主姓名',</code><br> <code>UserTel varchar COMMENT '车主联系方式',</code><br> <code>UserCityCode varchar COMMENT '车主所在城市Code',</code><br> <code>UserCityName varchar COMMENT '车主所在城市名称',</code><br> <code>UserAddress varchar COMMENT '车主地址',</code><br> <code>ShortVin varchar COMMENT '车架号后6位',</code><br> <code>Remark varchar COMMENT '车辆备注',</code><br> <code>CreatedById varchar COMMENT '',</code><br> <code>CreatedAt datetime COMMENT '',</code><br> <code>UpdatedById varchar COMMENT '',</code><br> <code>UpdatedAt datetime COMMENT '',</code><br> <code>DeletedById varchar COMMENT '',</code><br> <code>DeletedAt datetime COMMENT '',</code><br> <code>Deleted tinyint NOT NULL DEFAULT '0' COMMENT '',</code><br> <code>primary key (id)</code><br><code>) DISTRIBUTE BY HASH(id) INDEX_ALL='Y' STORAGE_POLICY='HOT' BLOCK_SIZE=8192</code><br><code>;</code></p><p><code>INSERT INTO tb_workorderinfo</code><br><code>SELECT *</code><br><code>FROM ods_serviceordercenter.tb_workorderinfo ;</code></p><p><code>INSERT INTO tb_workcarinfo</code><br><code>SELECT *</code><br><code>FROM ods_serviceordercenter.tb_workcarinfo ;</code></p><h3 id="查询性能"><a href="#查询性能" class="headerlink" title="查询性能"></a>查询性能</h3><p><code>SELECT /*+ filter_not_pushdown_columns=[ods_testcenter.tb_workcarinfo:Deleted|WorkOrderId] */</code> #禁止索引下推，不使用Deleted字段索引，如果不加全部索引的话，这个可以不设置<br><code>a.AppCode,b.VinNumber</code> #非必要一定不要写 *<br><code>FROM tb_workorderinfo a</code><br><code>LEFT JOIN tb_workcarinfo b</code><br> <code>ON b.WorkOrderId=a.Id</code><br> <code>AND b.Deleted=0</code><br><code>WHERE a.Deleted=0</code><br> <code>AND a.CreatedAt>='2021-01-01'</code><br><code>LIMIT 100;</code></p><p>800ms左右</p><h3 id="更换分区方式"><a href="#更换分区方式" class="headerlink" title="更换分区方式"></a>更换分区方式</h3><p><code>DROP TABLE IF EXISTS tb_workorderinfo;</code><br><code>Create Table tb_workorderinfo (</code><br> <code>Id varchar NOT NULL COMMENT '主键(WI)',</code><br> <code>AppCode varchar COMMENT '申请编码',</code><br> <code>SourceType smallint COMMENT '工单来源(0工单下单1智车品2客户接口3LCRM下单4OCRM下单5平安私有云6金融SAAS推送7车务推送)',</code><br> <code>OrderType varchar COMMENT '工单类型(sys_ordertypeinfo.Code)',</code><br> <code>CreateType smallint COMMENT '工单建单方式(来源二级分类)(1批量导入)',</code><br> <code>WorkStatus smallint COMMENT '工单状态(0待提交1正在审核2审核不通过3正在寻找工程师4等待工程师接单5已拒单，重新寻找工程师6等待提交结果7等待客户确认8客户驳回，重新处理9已完成10关闭11质检12质检不通过)',</code><br> <code>ServiceProviderCode varchar COMMENT '服务供应商Code',</code><br> <code>CustomerId varchar COMMENT '客户Id',</code><br> <code>CustomerName varchar COMMENT '客户名称',</code><br> <code>CustStoreId varchar COMMENT '门店Id',</code><br> <code>CustStoreName varchar COMMENT '门店名称',</code><br> <code>CustStoreCode varchar COMMENT '门店Code',</code><br> <code>PreCustStoreId varchar COMMENT '客户接口下单门店Id',</code><br> <code>PreCustStoreName varchar COMMENT '客户接口下单门店名称',</code><br> <code>CustSettleId varchar COMMENT '结算单位Id',</code><br> <code>CustSettleName varchar COMMENT '结算单位名称',</code><br> <code>IsCustomer tinyint NOT NULL COMMENT '是否为客户(0否1是)',</code><br> <code>CustCoopType smallint COMMENT '客户是否有合作商品(1中瑞2非中瑞3中瑞+非中瑞)',</code><br> <code>ProCode varchar COMMENT '省份Code',</code><br> <code>ProName varchar COMMENT '省份名称',</code><br> <code>CityCode varchar COMMENT '城市Code',</code><br> <code>CityName varchar COMMENT '城市名称',</code><br> <code>AreaCode varchar COMMENT '地区Code',</code><br> <code>AreaName varchar COMMENT '地区名称',</code><br> <code>InstallAddress varchar COMMENT '加装地址',</code><br> <code>InstallTime datetime COMMENT '加装时间',</code><br> <code>RequiredTime datetime COMMENT '要求时间',</code><br> <code>LinkMan varchar COMMENT '联系人姓名',</code><br> <code>LinkTel varchar COMMENT '联系电话',</code><br> <code>SecondLinkTel varchar COMMENT '第二联系电话',</code><br> <code>SecondLinkMan varchar COMMENT '第二联系人',</code><br> <code>WarehouseId varchar COMMENT '发货一级仓Id',</code><br> <code>WarehouseName varchar COMMENT '发货一级仓名称',</code><br> <code>Remark varchar COMMENT '客户备注',</code><br> <code>IsUrgent tinyint NOT NULL DEFAULT '0' COMMENT '检修工单是否紧急(0否1是)',</code><br> <code>CustUniqueSign varchar COMMENT '客户唯一标识',</code><br> <code>CreatePersonCode varchar COMMENT '建单人Code',</code><br> <code>CreatePersonName varchar COMMENT '建单人姓名',</code><br> <code>EffectiveTime datetime COMMENT '工单生效时间',</code><br> <code>EffectiveSuccessfulTime datetime COMMENT '下单成功生效时间',</code><br> <code>CreatedById varchar COMMENT '',</code><br> <code>CreatedAt datetime COMMENT '',</code><br> <code>UpdatedById varchar COMMENT '',</code><br> <code>UpdatedAt datetime COMMENT '',</code><br> <code>DeletedById varchar COMMENT '',</code><br> <code>DeletedAt datetime COMMENT '',</code><br> <code>Deleted tinyint NOT NULL DEFAULT '0' COMMENT '',</code><br> <code>primary key (id,CreatedAt),</code><br> <code>KEY NON-AppCode (AppCode),</code><br> <code>KEY NON-OrderType (OrderType),</code><br> <code>KEY NON-ProCode (ProCode),</code><br> <code>KEY NON-CustStoreCode (CustStoreCode),</code><br> <code>KEY NON-CustStoreId (CustStoreId),</code><br> <code>KEY NON-WorkStatus (WorkStatus),</code><br> <code>KEY NON-CreatedAt (CreatedAt),</code><br> <code>KEY NON-CreatePersonName (CreatePersonName),</code><br> <code>KEY NON-LinkMan (LinkMan),</code><br> <code>KEY NON-CustomerId (CustomerId),</code><br> <code>KEY NON-CreateType (CreateType),</code><br> <code>KEY NON-InstallTime (InstallTime),</code><br> <code>KEY NON-RequiredTime (RequiredTime),</code><br> <code>KEY NON-CustomerName (CustomerName),</code><br> <code>KEY NON-CustSettleId (CustSettleId)</code><br><code>) DISTRIBUTE BY HASH(id)</code><br><code>PARTITION BY VALUE (DATE_FORMAT(CreatedAt,'%Y%m')) LIFECYCLE 200</code><br><code>STORAGE_POLICY='MIXED' HOT_PARTITION_COUNT=2</code><br><code>BLOCK_SIZE=8192;</code></p><p><code>DROP TABLE IF EXISTS tb_workcarinfo;</code><br><code>Create Table tb_workcarinfo (</code><br> <code>Id varchar NOT NULL COMMENT '主键(WC)',</code><br> <code>WorkOrderId varchar COMMENT '工单主单Id(tb_workorderinfo.Id)',</code><br> <code>VinNumber varchar COMMENT '车架号',</code><br> <code>PlateNumber varchar COMMENT '车牌号',</code><br> <code>EngineNumber varchar COMMENT '发动机号',</code><br> <code>CarModelId varchar COMMENT '车型Id',</code><br> <code>CarModelName varchar COMMENT '车型名称',</code><br> <code>CarSeriesId varchar COMMENT '车系Id',</code><br> <code>CarSeriesName varchar COMMENT '车系名称',</code><br> <code>CarBrandId varchar COMMENT '车辆品牌Id',</code><br> <code>CarBrandName varchar COMMENT '车辆品牌名称',</code><br> <code>CarFullName varchar COMMENT '车型全称',</code><br> <code>Color varchar COMMENT '车辆颜色',</code><br> <code>CarPrice decimal(18, 2) COMMENT '车辆价格',</code><br> <code>IsNewCar smallint COMMENT '车辆类型(0二手车1新车)',</code><br> <code>CarType smallint COMMENT '车辆类型(1新车2二手车)',</code><br> <code>UserName varchar COMMENT '车主姓名',</code><br> <code>UserTel varchar COMMENT '车主联系方式',</code><br> <code>UserCityCode varchar COMMENT '车主所在城市Code',</code><br> <code>UserCityName varchar COMMENT '车主所在城市名称',</code><br> <code>UserAddress varchar COMMENT '车主地址',</code><br> <code>ShortVin varchar COMMENT '车架号后6位',</code><br> <code>Remark varchar COMMENT '车辆备注',</code><br> <code>CreatedById varchar COMMENT '',</code><br> <code>CreatedAt datetime COMMENT '',</code><br> <code>UpdatedById varchar COMMENT '',</code><br> <code>UpdatedAt datetime COMMENT '',</code><br> <code>DeletedById varchar COMMENT '',</code><br> <code>DeletedAt datetime COMMENT '',</code><br> <code>Deleted tinyint NOT NULL DEFAULT '0' COMMENT '',</code><br> <code>primary key (id,WorkOrderId,CreatedAt),</code><br><code>KEY NON-PlateNumber (PlateNumber),</code><br><code>KEY NON-UserName (UserName),</code><br><code>KEY NON-WorkOrderId (WorkOrderId),</code><br><code>KEY NON-VinNumber (VinNumber),</code><br><code>KEY NON-ShortVin (ShortVin)</code><br><code>) DISTRIBUTE BY HASH(WorkOrderId)</code><br><code>PARTITION BY VALUE (DATE_FORMAT(CreatedAt,'%Y%m')) LIFECYCLE 200</code><br><code>STORAGE_POLICY='MIXED' HOT_PARTITION_COUNT=2STORAGE_POLICY='HOT' BLOCK_SIZE=8192;</code></p><p><code>INSERT INTO tb_workorderinfo</code><br><code>SELECT *</code><br><code>FROM ods_serviceordercenter.tb_workorderinfo ;</code></p><p><code>INSERT INTO tb_workcarinfo</code><br><code>SELECT *</code><br><code>FROM ods_serviceordercenter.tb_workcarinfo ;</code></p><p>可以看出差距不大；</p><p><code>DROP TABLE IF EXISTS tb_alarminfo;</code><br><code>CREATE TABLE tb_alarminfo (</code><br> <code>Id varchar COMMENT '主键(AI)',</code><br> <code>CarId varchar COMMENT '车辆Id',</code><br> <code>DeviceId varchar COMMENT '设备Id',</code><br> <code>DeviceNumber varchar COMMENT '设备号',</code><br> <code>AlarmType smallint COMMENT '预警类型:1 光敏;2 断电;3 翻转;4 伪基站;5 SOS;6 急加速;7 急减速;8 急转弯;9 震动;10 碰撞;11 电瓶低电压;12 硬件超速预警;13 事故预警;14 摔车报警;15 倾倒预警',</code><br> <code>DeviceLat decimal(11,8) COMMENT '原始纬度',</code><br> <code>DeviceLng decimal(11,8) COMMENT '原始经度',</code><br> <code>Speed decimal(10,2) COMMENT '速度',</code><br> <code>FaultCode varchar COMMENT '故障码',</code><br> <code>AlarmAmount int COMMENT '当前预警次数',</code><br> <code>AlarmTime datetime COMMENT '预警时间',</code><br> <code>IsRead tinyint(1) DEFAULT '0' COMMENT '是否已读: 0 否;1 是',</code><br> <code>IsSend tinyint(1) DEFAULT '0' COMMENT '是否推送:0 否;1 是',</code><br> <code>IsFrozen tinyint(1) NOT NULL DEFAULT '0' COMMENT '是否冻结:0 否(最新版本);1 是(历史版本)',</code><br> <code>InsertTime datetime DEFAULT NULL COMMENT '预警生成时间',</code><br> <code>Deleted tinyint(1) DEFAULT '0' COMMENT '是否删除:0 否;1 是',</code><br> <code>PRIMARY KEY (Id,CarId,AlarmTime),</code><br> <code>KEY NON-AlarmType (AlarmType),</code><br> <code>KEY NON-DeviceNumber (DeviceNumber),</code><br> <code>KEY NON-IsRead (IsRead),</code><br> <code>KEY NON-IsSend (IsSend),</code><br> <code>KEY NON-DeviceId (DeviceId),</code><br> <code>KEY NON-IsFrozen (IsFrozen),</code><br> <code>KEY NON-InsertTime (InsertTime),</code><br> <code>KEY NON-AlarmTime (AlarmTime)</code><br><code>) COMMENT='预警域_硬件预警记录'</code><br><code>DISTRIBUTE BY HASH(CarId)</code><br><code>PARTITION BY VALUE (DATE_FORMAT(AlarmTime,'%Y%m')) LIFECYCLE 200</code><br><code>STORAGE_POLICY='MIXED' HOT_PARTITION_COUNT=3 BLOCK_SIZE=8192;</code></p><p><code>DROP TABLE IF EXISTS tb_distalarminfo;</code><br><code>CREATE TABLE tb_distalarminfo (</code><br> <code>Id varchar COMMENT '主键(DA)倒序',</code><br> <code>CarId varchar COMMENT '车辆Id',</code><br> <code>DeviceId varchar COMMENT '设备Id',</code><br> <code>DeviceNumber varchar COMMENT '设备号',</code><br> <code>AlarmGrade smallint COMMENT '危险等级:0 安全;1 低(跨市);2 中(跨省);3 高(跨国)',</code><br> <code>DeviceLat decimal(11,8) COMMENT '原始纬度',</code><br> <code>Devicelng decimal(11,8) COMMENT '原始经度',</code><br> <code>AlarmTime datetime COMMENT '预警时间',</code><br> <code>SafeProCode varchar COMMENT '安全省份Code',</code><br> <code>SafeCityCode varchar COMMENT '安全城市Code',</code><br> <code>NowProCode varchar COMMENT '目前省份Code',</code><br> <code>NowCityCode varchar COMMENT '目前城市Code',</code><br> <code>AlarmAmount int COMMENT '当前预警次数',</code><br> <code>UnAlarmAmount int COMMENT '当前解除预警次数',</code><br> <code>IsAlarm tinyint(1) DEFAULT '1' COMMENT '是否报警:0 解除报警;1 报警',</code><br> <code>IsRead tinyint(1) DEFAULT '0' COMMENT '是否已读:0 否;1 是',</code><br> <code>IsSend tinyint(1) DEFAULT '0' COMMENT '是否推送:0 否;1 是',</code><br> <code>IsFrozen tinyint(1) NOT NULL DEFAULT '0' COMMENT '是否冻结:0 否(最新数据);1 是(历史数据)',</code><br> <code>InsertTime datetime COMMENT '预警生成时间',</code><br> <code>Deleted tinyint(1) DEFAULT '0' COMMENT '是否删除:0 否;1 是',</code><br> <code>PRIMARY KEY (Id,CarId,AlarmTime),</code><br> <code>KEY NON-DeviceNumber (DeviceNumber),</code><br> <code>KEY NON-AlarmGrade (AlarmGrade),</code><br> <code>KEY NON-IsFrozen (IsFrozen),</code><br> <code>KEY NON-IsSend (IsSend),</code><br> <code>KEY NON-DeviceId (DeviceId),</code><br> <code>KEY NON-AlarmTime (AlarmTime),</code><br> <code>KEY NON-SafeCityCode (SafeCityCode),</code><br> <code>KEY NON-NowCityCode (NowCityCode),</code><br> <code>KEY NON-SafeProCode (SafeProCode),</code><br> <code>KEY NON-NowProCode (NowProCode)</code><br><code>) COMMENT='预警域_跨区预警信息'</code><br><code>DISTRIBUTE BY HASH(CarId)</code><br><code>PARTITION BY VALUE (DATE_FORMAT(AlarmTime,'%Y%m')) LIFECYCLE 200</code><br><code>STORAGE_POLICY='MIXED' HOT_PARTITION_COUNT=3 BLOCK_SIZE=8192;</code></p>]]></content>
<categories>
<category>数据库</category>
<category>AnalyticDB</category>
</categories>
<tags>
<tag>ADB</tag>
<tag>结构设计</tag>
</tags>
</entry>
</search>