-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
1908 lines (1668 loc) · 917 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title><![CDATA[深入浅出Android Chrome浏览器源代码 - 书签功能]]></title>
<url>https://amao12580.github.io/post/2017/01/Propose-a-perfect-solution-to-crack-android-chrome-browser's-bookmarks-function/</url>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>很久没更博了，自从16年国庆节后开始着手<a href="http://www.coolapk.com/apk/pro.kisscat.www.bookmarkhelper" target="_blank" rel="external">书签助手APP</a>的开发，更新了二十余个版本后，正逢年底，工作一下子紧张起来，暂时将APP的开发工作搁置了近一个月。</p>
<h1 id="问题描述"><a href="#问题描述" class="headerlink" title="问题描述"></a>问题描述</h1><p>最近几天有了时间来继续开发，计划17年的第一版更新，完成23款浏览器的书签合并到Android Chrome浏览器，尽最大努力，保持合并后的Chrome浏览器不出现FC、ANR等问题，最好能支持使用Android Google Accout同步到云端。而书签功能作为浏览器的底层基础，很多上层功能需要以此为基点，什么样的方案才能避免蝴蝶效应引起牵一发而动全身呢？</p>
<h1 id="可行性分析"><a href="#可行性分析" class="headerlink" title="可行性分析"></a>可行性分析</h1><p>前提说明：以下内容，大量使用<a href="https://github.com/alibaba/fastjson" target="_blank" rel="external">fastjson</a> api，请提前温习。</p>
<h2 id="Bookmarks文件"><a href="#Bookmarks文件" class="headerlink" title="Bookmarks文件"></a>Bookmarks文件</h2><p>我们都知道，Android Chrome浏览器的书签数据，存储在“/data/data/com.android.chrome/app_chrome/Default/Bookmarks”文件中，下文简称“Bookmarks文件”。文件大小视书签数据量而定，300条书签，接近250KB，将这个文件拷贝出来，简单的使用EditPlus软件打开。一眼看出是经 pretty formatter 处理的json文件，序列化时，为方便阅读，适当加上换行和空格，以体现视觉层次感。</p>
<p><img src="/img/chrome/Bookmarks-Overview.jpg" alt=""></p>
<p>稍加留意，可以知道Bookmarks文件中，书签与文件夹的关系被处理为一棵树形结构。事实上，我在开发Chrome浏览器提取书签，合并到Via浏览器时，就是读取了Bookmarks文件，使用递归将每条书签的文件夹完整路径提取出来。但这毕竟是read-only操作，不涉及到写入。</p>
<p>那么是不是将书签数据，按照既定的json格式组织起来，写入到Bookmarks文件就可以了呢？先暂且认为是可以的。那么如何安全的写入呢？我考虑到有如下几个关键点：</p>
<p>1.写入时，确保Chrome浏览器安全退出，避免多进程同时写，结果相互覆盖。由于GAPPS很有可能同Chrome一起安装，需要写入时监测GMS，避免唤醒Chrome浏览器，比如定期唤醒Chrome同步浏览器数据到云端。</p>
<p><img src="/img/chrome/Google-Account-Sync.jpg" alt=""></p>
<p>2.读取时，确保读取完整已有书签数据，避免在回写时数据丢失。</p>
<p>3.写入时，将已有Bookmarks文件做一次backup，避免程序bug或崩溃（OS Crash）时，可以回退到写入前（restore）。</p>
<p>针对第1点，分为两种情况。</p>
<p>1.如果没有安装其他的GAPPS套件，我们可以在书签合并之前，提示用户保存好浏览数据，然后手动退出（可以是强制停止），也可以由书签助手APP利用Root权限强行杀进程。</p>
<p>2.如果有安装其他的GAPPS套件，这种情况就复杂了，在不卸载的前提下，是否可以调用相关API对GAPPS套件进行完整启动和停止，以避免唤醒Chrome浏览器，可行性是未知的，理论上Google不会提供这样的API。以借用Root权限杀进程(kill -9 ?)的方式强制退出，会引发数据丢失的问题，何况完成书签合并后，还需要将GAPPS启动回来，那将有一段时间的运行中断，例如GMS通知等重要业务将没办法使用。强制退出的GAPPS会在书签合并中途重新启动吗？broadcast receiver 接收到系统event而启动主进程，这是可能的。这个方案目前问题太多，几乎不可行。也许有其他方式可以解决，先搁置这个问题，思考在可以解决的前提下，第2点和第3点该如何应对？</p>
<p>针对第2点，在下一小节里说明。</p>
<p>针对第3点，目前的程序逻辑，已经有了backup策略，但是restore是缺失的，这个作为健壮性设计在下一版本迭代中完成。</p>
<h3 id="读取完整书签数据"><a href="#读取完整书签数据" class="headerlink" title="读取完整书签数据"></a>读取完整书签数据</h3><p>针对上文中的第2点，确保读取完整已有书签数据。Bookmarks文件作为json格式的文件，完整读取，是需要将所有key-value对 都转化为java object，并保持好key-value对 之间的逻辑依赖关系，比如名为folderA的文件夹，共有5个field，包含了2条书签，分别是bookmarkB和bookmarkC，而bookmarkB有12个field，bookmarkC有6个field。别惊讶，下文中我将说明为何field数量出现不一致，这是真实存在的。想要完整读取，就需要这种描述信息由json格式，转化为java object。</p>
<p>在之前的读取时，只需要针对基本的4个field进行解析就可以：name，url，type，children。而且这些field的排列规则是非常固定的。我们可以抽象出来，组成Node对象，更加方便后续的程序逻辑解析。</p>
<p><img src="/img/chrome/Node-old.jpg" alt=""></p>
<p>那为了读取完整的书签数据，这一点也需要调整，整理出全新的Node对象，包含9个field。可以留意到，为了保持序列化的写入顺序，加入了注解：“@JSONField(ordinal = 5)”，这很重要，最小化影响Chrome浏览器功能，经观察Bookmarks文件中的key是以当前深度下的key字母正向顺序写入的。</p>
<p><img src="/img/chrome/Node-new.jpg" alt=""></p>
<h4 id="参差不齐的JSON"><a href="#参差不齐的JSON" class="headerlink" title="参差不齐的JSON"></a>参差不齐的JSON</h4><p>参差不齐的JSON是指统一深度下的节点对象，出现field数量变化。这一点体现在Node对象的meta_info field。</p>
<figure class="highlight less"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="variable">@Setter</span></div><div class="line"><span class="variable">@Getter</span></div><div class="line"><span class="variable">@JSONField</span>(ordinal = <span class="number">5</span>)</div><div class="line">private MetaInfo meta_info;</div></pre></td></tr></table></figure>
<p><img src="/img/chrome/Node-meta_info-noRules.jpg" alt=""></p>
<p>经过仔细观察，遍历一个比较大和复杂的Bookmarks文件，我们发现field数量变化也出现规律性。1.MetaInfo的自身field交叉，field数量不相同的MetaInfo对象，存在交叉关系，部分或完全重叠。2.有的Node对象不存在meta_info field，例如新添加的书签数据，这点在下文中说明。</p>
<p>MetaInfo也是一个对象，整理出可能的field后，得到以下pojo。</p>
<p><img src="/img/chrome/Node-meta_info-pojo.jpg" alt=""></p>
<p>你可以看出来，又有了新的注解：“@JSONField(ordinal = 10, name = synonymsPrefix + “userEdit”)”，由于java对于field的定义规范，不允许“.”。我们将key与field name的映射关系，做了特别处理。</p>
<p>仅MetaInfo对象就有11个field，数量不是问题，关键是无法确定，是否完整罗列了所有field ？</p>
<p>不能接受数据丢失，差之毫厘谬以千里。</p>
<h3 id="JSON-parseObject"><a href="#JSON-parseObject" class="headerlink" title="JSON.parseObject"></a>JSON.parseObject</h3><p>在上一小节，除了GAPPS的问题外，还遇到了meta_info对象不确定的问题，接下来solve it。</p>
<p>有没有可能在不定义pojo或不完整定义pojo的前提下，解析json string呢？在此基础上经过修改，还能在序列化为string不丢失数据呢？</p>
<p>这有点像在原有的json string上，再合适的位置，打开一道口子，放进新的书签数据段，然后重新缝合起来，变成一个全新的json string，就像是给计算机多加一个内存条，增加的不影响已有的，并且能协作一起运行。</p>
<p>多方尝试发现是可以的，整理出如下代码段，可以完成在“bookmark_bar”节点下，添加一个书签，进行序列化时，没有数据丢失，原有的格式被完整保留。</p>
<figure class="highlight haxe"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div></pre></td><td class="code"><pre><div class="line"><span class="comment">//将Bookmarks文件，完整读取到内存</span></div><div class="line"><span class="keyword">String</span> content = FileUtil.read(tmpFilePath);</div><div class="line"><span class="keyword">if</span> (content == <span class="literal">null</span> || content.isEmpty()) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:content is null or empty."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//解析为JSONObject，保持原有key排列顺序</span></div><div class="line">JSONObject top = JSON.parseObject(content, Feature.OrderedField);</div><div class="line"><span class="keyword">if</span> (top == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:top is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//获取名为roots的子节点</span></div><div class="line">JSONObject roots = top.getJSONObject(<span class="string">"roots"</span>);</div><div class="line"><span class="keyword">if</span> (roots == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:roots is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//获取名为bookmark_bar的子节点</span></div><div class="line">JSONObject bookmarkBar = roots.getJSONObject(<span class="string">"bookmark_bar"</span>);</div><div class="line"><span class="keyword">if</span> (bookmarkBar == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:bookmarkBar is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//获取名为children的子节点</span></div><div class="line">JSONArray children = bookmarkBar.getJSONArray(<span class="string">"children"</span>);</div><div class="line"><span class="keyword">if</span> (children == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:children is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line">LogHelper.v(<span class="string">"children origin size:"</span> + children.size());</div><div class="line"></div><div class="line">Node node = <span class="keyword">new</span> <span class="type">Node</span>();</div><div class="line">node.setName(<span class="string">"test01"</span>);</div><div class="line">node.setUrl(<span class="string">"http://www.github.com"</span>);</div><div class="line">node.setType(MetaData.FOLDER_TYPE_DEFAULT_NAME);</div><div class="line">node.setId(<span class="string">"99999999999999999"</span>);</div><div class="line"><span class="comment">//添加一个书签，名为：“test01”</span></div><div class="line">children.add(node);</div><div class="line"><span class="comment">//覆盖原有的children节点</span></div><div class="line">JsonUtil.replace(bookmarkBar, <span class="string">"children"</span>, children);</div><div class="line"><span class="comment">//关于checksum的处理下文说明</span></div><div class="line">JsonUtil.update(top, <span class="string">"checksum"</span>, <span class="string">""</span>);</div><div class="line"><span class="comment">//获取添加书签后的完整json数据</span></div><div class="line"><span class="keyword">String</span> finalContent = JsonUtil.toJson(top, <span class="literal">true</span>);</div><div class="line"><span class="keyword">if</span> (finalContent == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:finalContent is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div></pre></td></tr></table></figure>
<p>效果图：<br><img src="/img/chrome/Bookmarks-addOne-fake.jpg" alt=""></p>
<h3 id="字段的分析"><a href="#字段的分析" class="headerlink" title="字段的分析"></a>字段的分析</h3><p>按上一小节的代码段逻辑，我们对原有的书签json string，是不做修改的，这样是否可以被chrome浏览器识别为正常完整的新Bookmarks文件呢？经模拟器（Genymotion：V2.8.1）环境测试（完整安装GAPPS），是不可行的。chrome浏览器的书签页面，不显示新加的书签“test01”，但在Bookmarks文件中确实是存在的，这说明chrome浏览器不认可这一数据段。</p>
<p>经查证，修改完Bookmarks文件后，恢复启动chrome浏览器，新书签“test01”数据段被删除，这说明chrome浏览器在启动时，有verify机制和rollback机制，检测到不正常的书签数据hack，自动恢复到上一个savepoint，那这些机制的运作细节是怎样的呢？如何应对？</p>
<p>同时发现，chrome浏览器页面中，无法再添加新书签了，提示：“无法添加书签。”，这很可能是在异常修改后，chrome浏览器的一种自我防御机制，还只是猜想。</p>
<p>也许完整分析Bookmarks文件的字段会有一些灵感？在字面上来分析业务含义，很容易知道大部分字段的功能性和算法。</p>
<figure class="highlight delphi"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">String</span> id;<span class="comment">//【猜测】唯一主键，数据来源不明，经观察，存在自增特点，步长不确定。同级路径从上到下，存在顺序性，值大于内层目录id值</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> <span class="keyword">name</span>;<span class="comment">//书签或者文件夹名称</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> url;<span class="comment">//书签的url，文件夹无此field</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> <span class="keyword">type</span>;<span class="comment">//类型，区分是书签还是文件夹，固定枚举值</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> date_added;<span class="comment">//数据被添加时的时间戳，固定存在</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> date_modified;<span class="comment">//数据被修改时的时间戳，不一定存在。如果修改，则该书签或文件夹新增该field，同时上级路径也会有该field</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> checksum;<span class="comment">//【猜测】roots节点才有，书签完整性和有效性校验值，具体算法不明。</span></div><div class="line"></div><div class="line">Integer version;<span class="comment">//【猜测】roots节点才有，版本？具体产生算法不明。</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> sync_transaction_version;<span class="comment">//【猜测】chrome浏览器的书签想要同步到云端，就需要这个字段控制版本</span></div><div class="line"></div><div class="line"><span class="keyword">Object</span> meta_info;<span class="comment">//【猜测】观察仅书签才有，可能记录了书签的元数据。</span></div><div class="line"></div><div class="line"><span class="keyword">Object</span>[] children;<span class="comment">//文件夹才有，描述该文件夹所包含的书签数据</span></div></pre></td></tr></table></figure>
<p>未知的字段，大概占了50%左右，仅仅从字面含义观察，已经不能满足分析要求了。</p>
<p>可以看出来，chrome浏览器对于运行稳定性以及健壮性，有很多的考虑，如何不破坏这些机制，完成书签添加呢？</p>
<h2 id="黑盒测试验证"><a href="#黑盒测试验证" class="headerlink" title="黑盒测试验证"></a>黑盒测试验证</h2><p>在静态分析meta_info field 阶段时，意外的留意到，不是每个字段都会出现，呈现不整齐的现象。那在实际的chrome浏览器中，新增书签或文件夹时是否也存在呢？如果存在，意味着有些字段的产生规则和业务含义可以不必理会，这需要在模拟器环境来验证猜想。</p>
<h3 id="新书签"><a href="#新书签" class="headerlink" title="新书签"></a>新书签</h3><p>在模拟器环境，完整GAPPS套件安装，新安装最新版chrome浏览器，完成Google account 同步【FQ】，包括浏览器信息同步。此时正常完整退出chrome浏览器，保存一个Bookmarks文件副本。打开chrome浏览器，在“书签栏”路径下，新加一个书签，名为“test02”（新书签，之前是不存在的），正常完整的退出chrome浏览器，回过头来再看Bookmarks文件的变化。</p>
<p>添加书签：<br><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-bookmark-one.jpg" alt=""></p>
<p>Bookmarks文件变化：</p>
<p>第1部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-bookmark-one-after-change.jpg" alt=""></p>
<p>第2部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-bookmark-one-after-change-checksum.jpg" alt=""></p>
<p>我们可以发现，新的书签，不仅仅影响局部数据，像原有的date_modified、checksum等值也会跟随变化。好消息是，观察新书签数据结构，发现新书签数据段，字段结构并不复杂，大部分字段可以由外部程序提供，但还是存在疑点：“id”、“sync_transaction_version”、“checksum”等产生规则是什么呢？</p>
<h3 id="新文件夹"><a href="#新文件夹" class="headerlink" title="新文件夹"></a>新文件夹</h3><p>已经知道新书签的添加，对Bookmarks文件的影响，继续来观察新文件夹对Bookmarks文件的影响。按照上一节的流程添加，只是添加时改为添加文件夹，名为“新建文件夹10”,同时为这个新文件夹，加入一个书签，名为：“test03”。</p>
<p>添加文件夹：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-one.jpg" alt=""></p>
<p>Bookmarks文件变化：</p>
<p>第1部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-bookmark-one-after-change.jpg" alt=""></p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-bookmark-one-after-change2.jpg" alt=""></p>
<p>第2部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-and-bookmark-one-after-change-checksum.jpg" alt=""></p>
<p>同上一小节，“id”、“sync_transaction_version”、“checksum”等字段的产生规则还不明确。</p>
<h3 id="SyncData-sqlite3-不确定问题"><a href="#SyncData-sqlite3-不确定问题" class="headerlink" title="SyncData.sqlite3 不确定问题"></a>SyncData.sqlite3 不确定问题</h3><p>在观察新加书签和新加文件夹的时候，观察的重点对象是Bookmarks文件。经过上文的观察结果，有一些疑问不能仅仅通过Bookmarks文件来得到答案。在观察Bookmarks文件的同时，我们使用文件系统自身的watch机制，来感知其他数据文件的变化，得到了另一重点对象：SyncData.sqlite3，它的路径位于：“/data/data/com.android.chrome/app_chrome/Default/Sync Data/SyncData.sqlite3”，经分析，这是一个典型的sqlite3数据库。</p>
<p>将这个文件拷贝出来，用Navicat Premium（11.1.12）打开，果然是不需要密码，可以直接打开查看。</p>
<p><img src="/img/chrome/SyncData-sqlite3-overview.jpg" alt=""></p>
<p>5张表的结构也出来了<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">CREATE TABLE deleted_metas (metahandle bigint primary key ON CONFLICT FAIL,base_version bigint<span class="built_in"> default </span>-1,server_version bigint<span class="built_in"> default </span>0,local_external_id bigint<span class="built_in"> default </span>0,transaction_version bigint<span class="built_in"> default </span>0,mtime bigint<span class="built_in"> default </span>0,server_mtime bigint<span class="built_in"> default </span>0,ctime bigint<span class="built_in"> default </span>0,server_ctime bigint<span class="built_in"> default </span>0,id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,server_parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,is_unsynced bit<span class="built_in"> default </span>0,is_unapplied_update bit<span class="built_in"> default </span>0,is_del bit<span class="built_in"> default </span>0,is_dir bit<span class="built_in"> default </span>0,server_is_dir bit<span class="built_in"> default </span>0,server_is_del bit<span class="built_in"> default </span>0,non_unique_name varchar,server_non_unique_name varchar(255),unique_server_tag varchar,unique_client_tag varchar,unique_bookmark_tag varchar,specifics blob,server_specifics blob,base_server_specifics blob,server_unique_position blob,unique_position blob,attachment_metadata blob,server_attachment_metadata blob);</div><div class="line"></div><div class="line">CREATE TABLE metas(metahandle bigint primary key ON CONFLICT FAIL,base_version bigint<span class="built_in"> default </span>-1,server_version bigint<span class="built_in"> default </span>0,local_external_id bigint<span class="built_in"> default </span>0,transaction_version bigint<span class="built_in"> default </span>0,mtime bigint<span class="built_in"> default </span>0,server_mtime bigint<span class="built_in"> default </span>0,ctime bigint<span class="built_in"> default </span>0,server_ctime bigint<span class="built_in"> default </span>0,id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,server_parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,is_unsynced bit<span class="built_in"> default </span>0,is_unapplied_update bit<span class="built_in"> default </span>0,is_del bit<span class="built_in"> default </span>0,is_dir bit<span class="built_in"> default </span>0,server_is_dir bit<span class="built_in"> default </span>0,server_is_del bit<span class="built_in"> default </span>0,non_unique_name varchar,server_non_unique_name varchar(255),unique_server_tag varchar,unique_client_tag varchar,unique_bookmark_tag varchar,specifics blob,server_specifics blob,base_server_specifics blob,server_unique_position blob,unique_position blob,attachment_metadata blob,server_attachment_metadata blob);</div><div class="line"></div><div class="line">CREATE TABLE models (model_id BLOB primary key, progress_marker BLOB, transaction_version BIGINT<span class="built_in"> default </span>0,context BLOB);</div><div class="line"></div><div class="line">CREATE TABLE share_info (id TEXT primary key, name TEXT, store_birthday TEXT, cache_guid TEXT, bag_of_chips BLOB);</div><div class="line"></div><div class="line">CREATE TABLE share_version (id VARCHAR(128) primary key, data INT);</div></pre></td></tr></table></figure></p>
<p>经过对比分析，metas表，是主要观察对象，这个表的数据量1000+，保存了多处登录的所有设备上：浏览历史记录、当前打开的标签页、书签信息。当数据<br>新加书签“test02”的对应记录：</p>
<p><img src="/img/chrome/SyncData-sqlite3-metas-bookmarks-newBookmark-test02.jpg" alt=""></p>
<p>新加文件夹“test03”的对应记录：</p>
<p><img src="/img/chrome/SyncData-sqlite3-metas-folder-newFolder-test03.jpg" alt=""></p>
<p>很明显的，metas.local_external_id 字段，对应着Bookmarks文件中，id字段；metas.transaction_version 字段，对应着Bookmarks文件中，sync_transaction_version字段，checksum字段暂时没有找到映射关系。</p>
<p>还发现，metas表中，有10多个字段尚有疑问，</p>
<figure class="highlight applescript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">“metahandle”，“local_external_id”，“transaction_version”，</div><div class="line">“unique_bookmark_tag”，“specifics”，“server_specifics”，</div><div class="line">“server_unique_position”，“unique_position”，“<span class="built_in">id</span>”，“parent_id”，</div><div class="line">“server_parent_id”</div></pre></td></tr></table></figure>
<p>很容易从字面上猜测到业务含义，但这些字段的产生算法是怎么样的呢？</p>
<p>还好在上一小节中的：“id”、“sync_transaction_version”这两个字段在SyncData.sqlite3文件中得到印证。总算没有白费功夫，但是“checksum”是什么呢？猜测的是否有误呢？</p>
<p>checksum字段的算法，经google找到一小部分非官方说明：<a href="http://stackoverflow.com/questions/11308603/logic-behind-creating-bookmark-checksum-in-google-chrome" target="_blank" rel="external">http://stackoverflow.com/questions/11308603/logic-behind-creating-bookmark-checksum-in-google-chrome</a>。关键的代码段在chromium开源项目中：<a href="https://chromium.googlesource.com/chromium/chromium/+/20f8aa123f98b2bcb0d346af0d78ad7a8ddea5d0/chrome/browser/bookmarks/bookmark_codec.cc" target="_blank" rel="external">https://chromium.googlesource.com/chromium/chromium/+/20f8aa123f98b2bcb0d346af0d78ad7a8ddea5d0/chrome/browser/bookmarks/bookmark_codec.cc</a></p>
<figure class="highlight xl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div></pre></td><td class="code"><pre><div class="line">Value* BookmarkCodec::Encode(BookmarkModel* model) {</div><div class="line"> <span class="function"><span class="title">return</span> Encode(model-></span>bookmark_bar_node(),</div><div class="line"> <span class="function"><span class="title">model</span>-></span>other_node(),</div><div class="line"> <span class="function"><span class="title">model</span>-></span>mobile_node(),</div><div class="line"> <span class="function"><span class="title">model</span>-></span><span class="function"><span class="title">root_node</span>()-></span>meta_info_str());</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line">Value* BookmarkCodec::Encode(const BookmarkNode* bookmark_bar_node,</div><div class="line"> const BookmarkNode* other_folder_node,</div><div class="line"> const BookmarkNode* mobile_folder_node,</div><div class="line"> const std::string& model_meta_info) {</div><div class="line"> ids_reassigned_ = <span class="literal">false</span>;</div><div class="line"> InitializeChecksum();</div><div class="line"> DictionaryValue* roots = new DictionaryValue();</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>Set(kRootFolderNameKey, EncodeNode(bookmark_bar_node));</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>Set(kOtherBookmarkFolderNameKey, EncodeNode(other_folder_node));</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>Set(kMobileBookmarkFolderNameKey, EncodeNode(mobile_folder_node));</div><div class="line"> <span class="keyword">if</span> (!model_meta_info.empty())</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>SetString(kMetaInfo, model_meta_info);</div><div class="line"> DictionaryValue* main = new DictionaryValue();</div><div class="line"> <span class="function"><span class="title">main</span>-></span>SetInteger(kVersionKey, kCurrentVersion);</div><div class="line"> FinalizeChecksum();</div><div class="line"> <span class="comment">// We are going to store the computed checksum. So set stored checksum to be</span></div><div class="line"> <span class="comment">// the same as computed checksum.</span></div><div class="line"> stored_checksum_ = computed_checksum_;</div><div class="line"> <span class="function"><span class="title">main</span>-></span>Set(kChecksumKey, new base::StringValue(computed_checksum_));</div><div class="line"> <span class="function"><span class="title">main</span>-></span>Set(kRootsKey, roots);</div><div class="line"> return main;</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line">Value* BookmarkCodec::EncodeNode(const BookmarkNode* node) {</div><div class="line"> DictionaryValue* value = new DictionaryValue();</div><div class="line"> <span class="function"><span class="title">std</span>::string id = base::Int64ToString(node-></span>id());</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kIdKey, id);</div><div class="line"> <span class="function"><span class="title">const</span> string16& <span class="built_in">title</span> = node-></span>GetTitle();</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kNameKey, <span class="built_in">title</span>);</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kDateAddedKey,</div><div class="line"> <span class="function"><span class="title">base</span>::Int64ToString(node-></span>date_added().ToInternalValue()));</div><div class="line"> <span class="function"><span class="title">if</span> (node-></span>is_url()) {</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kTypeKey, kTypeURL);</div><div class="line"> <span class="function"><span class="title">std</span>::string url = node-></span>url().possibly_invalid_spec();</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kURLKey, url);</div><div class="line"> UpdateChecksumWithUrlNode(id, <span class="built_in">title</span>, url);</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kTypeKey, kTypeFolder);</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kDateModifiedKey,</div><div class="line"> <span class="function"><span class="title">base</span>::Int64ToString(node-></span>date_folder_modified().</div><div class="line"> ToInternalValue()));</div><div class="line"> UpdateChecksumWithFolderNode(id, <span class="built_in">title</span>);</div><div class="line"> ListValue* child_values = new ListValue();</div><div class="line"> <span class="function"><span class="title">value</span>-></span>Set(kChildrenKey, child_values);</div><div class="line"> <span class="function"><span class="title">for</span> (int i = 0; i < node-></span>child_count(); ++i)</div><div class="line"> <span class="function"><span class="title">child_values</span>-></span>A<span class="function"><span class="title">ppend</span>(EncodeNode(node-></span>GetChild(i)));</div><div class="line"> }</div><div class="line"> <span class="function"><span class="title">if</span> (!node-></span>meta_info_str().empty())</div><div class="line"> <span class="function"><span class="title">value</span>-></span>S<span class="function"><span class="title">etString</span>(kMetaInfo, node-></span>meta_info_str());</div><div class="line"> return value;</div><div class="line">}</div><div class="line"></div><div class="line">void BookmarkCodec::FinalizeChecksum() {</div><div class="line"> base::MD5Digest digest;</div><div class="line"> base::MD5Final(&digest, &md5_context_);</div><div class="line"> computed_checksum_ = base::MD5DigestToBase16(digest);</div><div class="line">}</div></pre></td></tr></table></figure>
<p>进行语义逻辑分析，可以看出来，Encode方法将bookmark_bar_node对象，other_node对象，mobile_node对象，meta_info_str对象作为入参，多维度进行摘要算法计算，字段结果就是checksum值。也就是说，Bookmarks文件中，任意书签的任意字段信息发生改变，checksum跟随变化，意思是所有书签数据的hashTag。摘要算法的细节还不清楚，在下文中说明。</p>
<h2 id="思路转换"><a href="#思路转换" class="headerlink" title="思路转换"></a>思路转换</h2><p>真是一波未平一波又起，chrome浏览器的复杂性远远超出预期，看来简单的通过分析android data目录文件，没办法满足需求了。换个角度思考，chrome浏览器的复杂性和稳定性是同时提供的，这意味着内部的复杂度被chrome上层程序掩盖。还知道chrome pc版是支持<a href="https://chrome.google.com/webstore/category/extensions?hl=zh-CN" target="_blank" rel="external">extension</a>的，这是一大亮点，意味着pc版开放了一些api给extension调用，辅助chrome运行。那有没有可能，由android chrome浏览器提供一个书签管理api，开放给书签助手APP来调用呢？</p>
<h2 id="content-provider"><a href="#content-provider" class="headerlink" title="content provider"></a>content provider</h2><p>通过一番Google搜索（通过google解决google的问题？），找到了可能的解决办法：<a href="http://stackoverflow.com/questions/14433480/add-bookmark-to-chrome-browser-on-android?rq=1" target="_blank" rel="external">Add bookmark to chrome browser on Android</a>。原理是android chrome浏览器开放了一对权限来管理书签和历史浏览：com.android.browser.permission.READ_HISTORY_BOOKMARKS，com.android.browser.permission.WRITE_HISTORY_BOOKMARKS，app在获取这些权限后，可以通过content provider的方式，将书签数据组织为chrome浏览器需要的格式，提交给chrome浏览器来存储。</p>
<p>“com.android.browser.permission.READ_HISTORY_BOOKMARKS”，该权限被用作查询chrome浏览器的书签和历史浏览记录，可以按条件过滤，<a href="http://stackoverflow.com/questions/31872028/which-is-the-alternate-permission-for-reading-bookmarks-in-android-m" target="_blank" rel="external">这个权限在6..0及以上的android版本已经被删除，意味着不再可用。</a></p>
<p>这2个的api由android chrome浏览器官方提供，可行性高了不少，附带的GAPPS问题和同步到云端的问题，也屏蔽了解构底层Bookmarks文件和SyncData.sqlite3文件。问题可以就此解决吗？</p>
<h3 id="solution-demo"><a href="#solution-demo" class="headerlink" title="solution demo"></a>solution demo</h3><p>整理出添加书签的demo代码段：<br><figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">private</span> <span class="keyword">void</span> test(){</div><div class="line"> ContentValues values = <span class="keyword">new</span> ContentValues();</div><div class="line"> values.put(BookmarkColumns.TITLE, <span class="string">"title"</span>);</div><div class="line"> values.put(BookmarkColumns.URL, <span class="string">"http://ss.com"</span>);</div><div class="line"> values.put(BookmarkColumns.BOOKMARK, <span class="number">1</span>);</div><div class="line"> values.put(BookmarkColumns.CREATED, <span class="number">0</span>);</div><div class="line"> values.put(BookmarkColumns.DATE, <span class="number">0</span>);</div><div class="line"> values.put(<span class="string">"parentId"</span>, <span class="number">3</span>); <span class="comment">// just for Chrome</span></div><div class="line"> getContentResolver().insert(Uri.parse(<span class="string">"content://com.android.chrome.ChromeBrowserProvider/bookmarks"</span>), values);</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">public</span> <span class="keyword">static</span> class BookmarkColumns implements BaseColumns {</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> URL = <span class="string">"url"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> VISITS = <span class="string">"visits"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> DATE = <span class="string">"date"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> BOOKMARK = <span class="string">"bookmark"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> TITLE = <span class="string">"title"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> CREATED = <span class="string">"created"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> FAVICON = <span class="string">"favicon"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> THUMBNAIL = <span class="string">"thumbnail"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> TOUCH_ICON = <span class="string">"touch_icon"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> USER_ENTERED = <span class="string">"user_entered"</span>;</div><div class="line">}</div></pre></td></tr></table></figure></p>
<p>添加文件夹的api也是类似的写法吗？google没有找到印证。</p>
<h3 id="chromium源代码"><a href="#chromium源代码" class="headerlink" title="chromium源代码"></a>chromium源代码</h3><p>想起来chrome是基于chromium开源项目的，那content provider这部分的代码也是开源的吗？找到chromium源代码online page:<a href="https://www.chromium.org/developers/how-tos/get-the-code" target="_blank" rel="external">https://www.chromium.org/developers/how-tos/get-the-code</a>，选择<a href="https://chromium.googlesource.com/chromium/src/+/master/docs/android_build_instructions.md" target="_blank" rel="external">Android</a>版，先不下载，翻阅相关代码段：<a href="https://chromium.googlesource.com/chromium/src/+/master/chrome/android/java/src/org/chromium/chrome/browser/provider/ChromeBrowserProvider.java" target="_blank" rel="external">https://chromium.googlesource.com/chromium/src/+/master/chrome/android/java/src/org/chromium/chrome/browser/provider/ChromeBrowserProvider.java</a>，405行。</p>
<figure class="highlight groovy"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">public</span> Uri insert(Uri uri, ContentValues values) {</div><div class="line"> <span class="keyword">if</span> (!canHandleContentProviderApiCall() || !hasWriteAccess()) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">int</span> match = mUriMatcher.match(uri);</div><div class="line"> Uri res = <span class="literal">null</span>;</div><div class="line"> <span class="keyword">long</span> id;</div><div class="line"> <span class="keyword">switch</span> (match) {</div><div class="line"> <span class="keyword">case</span> <span class="string">URI_MATCH_BOOKMARKS:</span></div><div class="line"> id = addBookmark(values);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_BOOKMARK_ID) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_BOOKMARK_CONTENT:</span></div><div class="line"> values.put(BookmarkColumns.BOOKMARK, <span class="number">1</span>);</div><div class="line"> <span class="comment">//$FALL-THROUGH$</span></div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_BOOKMARK:</span></div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_HISTORY_CONTENT:</span></div><div class="line"> id = addBookmarkFromAPI(values);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_CONTENT_PROVIDER_ID) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_SEARCHES:</span></div><div class="line"> id = addSearchTermFromAPI(values);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_CONTENT_PROVIDER_ID) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"><span class="symbol"> default:</span></div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> IllegalArgumentException(TAG + <span class="string">": insert - unknown URL "</span> + uri);</div><div class="line"> }</div><div class="line"> res = ContentUris.withAppendedId(uri, id);</div><div class="line"> notifyChange(res);</div><div class="line"> <span class="keyword">return</span> res;</div><div class="line">}</div></pre></td></tr></table></figure>
<p>正好对应了content provider部分的对接部分，使用git将chromium android源代码clone下来，因为网速影响，只拉取master分支的最后一个commit版本，history change都先不下载了，约2.5GB。</p>
<figure class="highlight crmsh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">git <span class="keyword">clone</span> <span class="title">https</span>://chromium.googlesource.com/chromium/src -b <span class="keyword">master</span> <span class="title">--depth</span>=<span class="number">1</span></div></pre></td></tr></table></figure>
<p>android源代码，使用Android Studio浏览应该是最好的选择，因为机器配置（4核奔腾+8GB）不够强，对如此浩瀚的代码仓库，编译索引起来力不从心，一度死机。尝试缩减源代码范围，只编译“\chromium\src\chrome\android”部分，约30MB，很快就打开了，但是依赖包没有编译，无法正常浏览。</p>
<p>针对如此大规模的源代码，有更好的源代码浏览工具吗？可能混杂的编程语言有python、java、c系列。</p>
<h4 id="Source-Insight"><a href="#Source-Insight" class="headerlink" title="Source Insight"></a>Source Insight</h4><p>Source Insight.它是一个面向项目开发的程序编辑器和代码浏览器。Source Insight能分析你的源代码并在你工作的同时动态维护它自己的符号数据库，并自动为你显示有用的上下文信息。 它的强大之处在于不仅仅是可编辑的源代码，还包括对于代码中的变量和类进行关联和查找。比如java语言，你可以清晰的看到一个类中的成员变量以及方法，而且source insight 还提供了类的预览，比如源码中有一个类，那么你可以解转到那个类里查看源码。</p>
<p>Source Insight支持的编程语言列表：<br><img src="/img/chrome/sourceInsight-support-language.jpg" alt=""></p>
<h4 id="ChromeBrowserProvider-java"><a href="#ChromeBrowserProvider-java" class="headerlink" title="ChromeBrowserProvider.java"></a>ChromeBrowserProvider.java</h4><p>上一小节，已经提到ChromeBrowserProvider.java文件，这一节的就此着手，抽丝剥茧，一路挖下去吧！</p>
<p>当入参uri，是URI_MATCH_BOOKMARKS时，进入到addBookmark(values)代码段，URI_MATCH_BOOKMARKS定义：private static final int URI_MATCH_BOOKMARKS = 0;</p>
<h5 id="addBookmark方法"><a href="#addBookmark方法" class="headerlink" title="addBookmark方法"></a>addBookmark方法</h5><p>接下来看addBookmark方法：</p>
<figure class="highlight maxima"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line">private long addBookmark(ContentValues <span class="built_in">values</span>) {</div><div class="line"> String url = <span class="built_in">values</span>.getAsString(BookmarkColumns.URL);</div><div class="line"> String <span class="built_in">title</span> = <span class="built_in">values</span>.getAsString(BookmarkColumns.TITLE);</div><div class="line"> boolean isFolder = <span class="literal">false</span>;</div><div class="line"> <span class="keyword">if</span> (<span class="built_in">values</span>.containsKey(BOOKMARK_IS_FOLDER_PARAM)) {</div><div class="line"> isFolder = <span class="built_in">values</span>.getAsBoolean(BOOKMARK_IS_FOLDER_PARAM);</div><div class="line"> }</div><div class="line"> long parentId = INVALID_BOOKMARK_ID;</div><div class="line"> <span class="keyword">if</span> (<span class="built_in">values</span>.containsKey(BOOKMARK_PARENT_ID_PARAM)) {</div><div class="line"> parentId = <span class="built_in">values</span>.getAsLong(BOOKMARK_PARENT_ID_PARAM);</div><div class="line"> }</div><div class="line"> long id = nativeAddBookmark(mNativeChromeBrowserProvider, url, <span class="built_in">title</span>, isFolder, parentId);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_BOOKMARK_ID) <span class="built_in">return</span> id;</div><div class="line"></div><div class="line"> <span class="keyword">if</span> (isFolder) {</div><div class="line"> updateLastModifiedBookmarkFolder(id);</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> updateLastModifiedBookmarkFolder(parentId);</div><div class="line"> }</div><div class="line"> <span class="built_in">return</span> id;</div><div class="line">}</div></pre></td></tr></table></figure>
<p>第2、3两行，分别获取传入的URL和标题字段值，不做多说。第4至11行，验证是否为文件夹，以及获取上层文件夹的id值，主要是为了第15至19行，进行文件夹的最后更新时间做计算，猜测对应前文的date_modified字段，进行级联修改。BOOKMARK_IS_FOLDER_PARAM与BOOKMARK_PARENT_ID_PARAM的定义如下。<br><figure class="highlight php"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="comment">/** The parameter used to specify whether this is a bookmark folder. */</span></div><div class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String BOOKMARK_IS_FOLDER_PARAM = <span class="string">"isFolder"</span>;</div><div class="line"></div><div class="line"><span class="comment">/** The parameter used to specify a bookmark parent ID in ContentValues. */</span></div><div class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String BOOKMARK_PARENT_ID_PARAM = <span class="string">"parentId"</span>;</div></pre></td></tr></table></figure></p>
<p>可以很明显的感受到Google对Java代码的规范性，类名采用大驼峰；方法名、参数名、局部变量、成员变量名，采用小驼峰；常量名采用全大写蛇形，这里有一份完整的:<a href="http://www.hawstein.com/posts/google-java-style.html" target="_blank" rel="external">Google Java编程风格指南</a></p>
<h4 id="nativeAddBookmark方法"><a href="#nativeAddBookmark方法" class="headerlink" title="nativeAddBookmark方法"></a>nativeAddBookmark方法</h4><p>上文中的第12行代码，调用nativeAddBookmark方法后，返回long类型的id字段。这个方法对应的源码段是：<br><figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">private</span> <span class="keyword">native</span> <span class="keyword">long</span> nativeAddBookmark(<span class="keyword">long</span> nativeChromeBrowserProvider,</div><div class="line"> <span class="keyword">String</span> url, <span class="keyword">String</span> title, <span class="built_in">boolean</span> isFolder, <span class="keyword">long</span> parentId);</div></pre></td></tr></table></figure></p>
<p>方法定义为native，从这一步开始，书签添加的逻辑实现，从Java层交棒给了C层。第一个想到的是在同级目录下会有一个：org_chromium_chrome_browser_provider_nativeAddBookmark.h文件，但是没找到。</p>
<p>在“\chromium\src\chrome\android\”目录下，并没有关于jni的代码实现，我们使用EditPlus，对整个chromium src层级目录“\chromium\src\”，进行递归搜索“nativeAddBookmark”关键字，范围限定为：“<em>.cc </em>.h”。耗时 1 分 14 秒完成搜索，但一无所获，难道jni部分并未开源吗？或者在其他路径下开源，未囊括到chromium项目？前一种有可能，后一种可能性很低。</p>
<p>我们都知道，JNI函数的注册有两种方法，一种是静态方法，需要用javah为每个声明了native函数的java类编译出的class文件生成一个头文件，另一种是动态注册，通过数据结构保存关联关系实现注册。搜索“nativeAddBookmark”关键字进行jni函数查找的方式，是以静态方法为前提的。那目标jni函数可能是采用动态的方式注册的吗？</p>
<p>在搜索jni函数的过程中，发现android与ios（”\chromium\src\ios\chrome）共用了同一jni函数，不知道pc版是不是也会共用？一方面性能相比上层代码高，另一方面OS底层都对c代码有支持，产生了一定的跨平台性。</p>
<h2 id="content-provider事务性思考"><a href="#content-provider事务性思考" class="headerlink" title="content provider事务性思考"></a>content provider事务性思考</h2><p>——————-还没写完，持续更新中。2017年1月16日10:01:51——————-</p>
<h1 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h1><h1 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h1><h2 id="思路"><a href="#思路" class="headerlink" title="思路"></a>思路</h2><ul>
<li><p><a href="http://stackoverflow.com/questions/14433480/add-bookmark-to-chrome-browser-on-android?rq=1" target="_blank" rel="external">stackoverflow:about use content provider way to add bookmarks</a></p>
</li>
<li><p><a href="https://developer.android.com/reference/android/provider/Browser.BookmarkColumns.html" target="_blank" rel="external">Browser.BookmarkColumns 释义</a></p>
</li>
</ul>
<h2 id="Source-Insight-使用"><a href="#Source-Insight-使用" class="headerlink" title="Source Insight 使用"></a>Source Insight 使用</h2><ul>
<li><p><a href="http://www.cnblogs.com/octobershiner/archive/2012/03/18/2400805.html" target="_blank" rel="external">source insight 基础使用教程</a></p>
</li>
<li><p><a href="http://zydecx.github.io/2015/05/source-insight-for-java.html" target="_blank" rel="external">source insight 添加jdk源码</a>，android sdk 源码添加同理</p>
</li>
</ul>
<h2 id="Chromium源代码结构"><a href="#Chromium源代码结构" class="headerlink" title="Chromium源代码结构"></a>Chromium源代码结构</h2><ul>
<li><p><a href="http://www.jianshu.com/p/4afe92418bd9" target="_blank" rel="external">Chromium 源代码目录结构 - 1</a></p>
</li>
<li><p><a href="http://hyshucom.iteye.com/blog/1694642" target="_blank" rel="external">Chromium 源代码目录结构 - 2</a></p>
</li>
</ul>
<h2 id="助攻"><a href="#助攻" class="headerlink" title="助攻"></a>助攻</h2><ul>
<li><p><a href="http://stackoverflow.com/questions/22732001/connect-to-vpn-in-genymotion-android" target="_blank" rel="external">genymotion vpn setting</a></p>
</li>
<li><p><a href="https://www.genymotion.com/help/desktop/faq/#network-vpn" target="_blank" rel="external">Why doesn’t Genymotion run with my VPN client?</a></p>
</li>
</ul>
<h1 id="Tips"><a href="#Tips" class="headerlink" title="Tips"></a>Tips</h1><h2 id="Genymotion"><a href="#Genymotion" class="headerlink" title="Genymotion"></a>Genymotion</h2><h3 id="virtual-device-启动过慢"><a href="#virtual-device-启动过慢" class="headerlink" title="virtual device 启动过慢"></a>virtual device 启动过慢</h3><p>有时候virtual device 启动过慢，花了10分钟，还停留在android黑底白字阶段，甚至启动失败。这可能是genymotion自身进程冲突，现在强制退出启动，退出其他genymotion软件，在windows任务管理器，进程列表中，找到adb、vbox相关进程，强制结束。再次启动，发现快了很多！</p>
<h2 id="Linux"><a href="#Linux" class="headerlink" title="Linux"></a>Linux</h2><h3 id="busybox"><a href="#busybox" class="headerlink" title="busybox"></a>busybox</h3><p>BusyBox 是一个集成了一百多个最常用linux命令和工具的软件。BusyBox 包含了一些简单的工具，例如ls、 cat 和 echo等等，还包含了一些更大、更复杂的工具，例如 grep、find、mount 以及 telnet。有些人将 BusyBox 称为 Linux 工具里的瑞士军刀。简单的说BusyBox就好像是个大工具箱，它集成压缩了 Linux 的许多工具和命令。也包含了 Android 系统的自带的shell。</p>
<p>为什么要在Android中加入busybox?</p>
<p>用过adb shell的人应该知道，在默认情况下，adb shell下是不能用clear,grep, find,vi等指令的，甚至连Tab链自动补全功能都不能用，对于已经习惯了使用这些指令的码农们来说，这是件比较悲摧的事情。幸运地是，我们有了busybox!</p>
<p>Genymotion环境下，使用DOS-ADB安装Busybox</p>
<figure class="highlight vim"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div></pre></td><td class="code"><pre><div class="line">download link :http<span class="variable">s:</span>//busybox.net/downloads/binaries/<span class="number">1.21</span>.<span class="number">1</span>/busybox-armv7l</div><div class="line"></div><div class="line">D:\Android\sdk\platform-tools>adb <span class="keyword">shell</span></div><div class="line">root@vbox86p:/ # <span class="keyword">cd</span> /data/</div><div class="line">root@vbox86p:/data # <span class="built_in">mkdir</span> busybox</div><div class="line">root@vbox86p:/data # <span class="keyword">exit</span></div><div class="line"></div><div class="line">D:\Android\sdk\platform-tools>adb push busybox /data/busybox/</div><div class="line">[<span class="number">100</span>%] /data/busybox/busybox</div><div class="line"></div><div class="line">D:\Android\sdk\platform-tools>adb <span class="keyword">shell</span></div><div class="line"></div><div class="line">root@vbox86p:/ # chmod <span class="number">777</span> /data/busybox/busybox</div><div class="line"></div><div class="line">root@vbox86p:/ # /data/busybox/busybox</div><div class="line"></div><div class="line">BusyBox v1.<span class="number">26.2</span> (<span class="number">2017</span>-<span class="number">01</span>-<span class="number">11</span> <span class="number">08</span>:<span class="number">43</span>:<span class="number">16</span> UTC) multi-<span class="keyword">call</span> binary.</div><div class="line">BusyBox <span class="keyword">is</span> copyrighted by many authors between <span class="number">1998</span>-<span class="number">2015</span>.</div><div class="line">Licensed under GPLv2. See <span class="keyword">source</span> distribution <span class="keyword">for</span> detailed</div><div class="line">copyright notices.</div><div class="line"></div><div class="line">Usage: busybox [<span class="function"><span class="keyword">function</span> [<span class="title">arguments</span>]...]</span></div><div class="line"> <span class="built_in">or</span>: busybox --<span class="keyword">list</span>[-full]</div><div class="line"> <span class="built_in">or</span>: busybox --install [-s] [DIR]</div><div class="line"> <span class="built_in">or</span>: <span class="function"><span class="keyword">function</span> [<span class="title">arguments</span>]...</span></div><div class="line"></div><div class="line"> BusyBox <span class="keyword">is</span> <span class="keyword">a</span> multi-<span class="keyword">call</span> binary that combines many common Unix</div><div class="line"> utilities into <span class="keyword">a</span> single <span class="built_in">executable</span>. Most people will create <span class="keyword">a</span></div><div class="line"> link <span class="keyword">to</span> busybox <span class="keyword">for</span> each <span class="function"><span class="keyword">function</span> <span class="title">they</span> <span class="title">wish</span> <span class="title">to</span> <span class="title">use</span> <span class="title">and</span> <span class="title">BusyBox</span></span></div><div class="line"> will act like whatever it was invoked <span class="keyword">as</span>.</div><div class="line"></div><div class="line">Currently defined <span class="keyword">function</span><span class="variable">s:</span></div><div class="line"> [, [[, acpid, <span class="built_in">add</span>-<span class="keyword">shell</span>, addgroup, adduser, adjtimex, arp, arping, ash,</div><div class="line"> awk, base64, basename, beep, blkdiscard, blkid, blockdev, bootchartd,</div><div class="line"> brctl, bunzip2, bzcat, bzip2, <span class="keyword">cal</span>, <span class="keyword">cat</span>, catv, chat, chattr, chgrp,</div><div class="line"> chmod, chown, chpasswd, chpst, chroot, chrt, chvt, cksum, clear, cmp,</div><div class="line"> comm, conspy, <span class="keyword">cp</span>, cpio, crond, crontab, cryptpw, cttyhack, cut, date,</div><div class="line"> dc, dd, deallocvt, delgroup, deluser, depmod, devmem, df, dhcprelay,</div><div class="line"> diff, dirname, dmesg, dnsd, dnsdomainname, dos2unix, dpkg, dpkg-<span class="keyword">deb</span>,</div><div class="line"> du, dumpkmap, dumpleases, <span class="keyword">echo</span>, ed, egrep, eject, env, envdir,</div><div class="line"> envuidgid, ether-wake, <span class="built_in">expand</span>, expr, fakeidentd, false, fatattr, fbset,</div><div class="line"> fbsplash, fdflush, fdformat, fdisk, fgconsole, fgrep, <span class="keyword">find</span>, findfs,</div><div class="line"> flock, <span class="keyword">fold</span>, free, freeramdisk, fsck, fsck.minix, fstrim, fsync, ftpd,</div><div class="line"> ftpget, ftpput, fuser, getopt, getty, <span class="keyword">grep</span>, groups, gunzip, gzip, halt,</div><div class="line"> hd, hdparm, head, hexdump, hostid, <span class="built_in">hostname</span>, httpd, hush, hwclock,</div><div class="line"> i2cdetect, i2cdump, i2cget, i2cset, id, ifconfig, ifdown, ifenslave,</div><div class="line"> ifplugd, ifup, inetd, init, insmod, install, ionice, iostat, ip,</div><div class="line"> ipaddr, ipcalc, ipcrm, ipcs, iplink, ipneigh, iproute, iprule,</div><div class="line"> iptunnel, kbd_mode, kill, killall, killall5, klogd, less, linux32,</div><div class="line"> linux64, linuxrc, <span class="keyword">ln</span>, loadfont, loadkmap, logger, login, logname,</div><div class="line"> logread, losetup, lpd, lpq, lpr, <span class="keyword">ls</span>, lsattr, lsmod, lsof, lspci, lsusb,</div><div class="line"> lzcat, lzma, lzop, lzopcat, makedevs, makemime, man, md5sum, mdev,</div><div class="line"> mesg, microcom, <span class="built_in">mkdir</span>, mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.minix,</div><div class="line"> mkfs.vfat, mknod, mkpasswd, mkswap, mktemp, modinfo, modprobe, more,</div><div class="line"> mount, mountpoint, mpstat, mt, mv, nameif, nanddump, nandwrite,</div><div class="line"> nbd-client, nc, netstat, nice, nmeter, nohup, nslookup, ntpd, od,</div><div class="line"> openvt, passwd, patch, pgrep, pidof, ping, ping6, pipe_progress,</div><div class="line"> pivot_root, pkill, pmap, popmaildir, poweroff, powertop, printenv,</div><div class="line"> <span class="built_in">printf</span>, <span class="keyword">ps</span>, pscan, pstree, <span class="keyword">pwd</span>, pwdx, raidautorun, rdate, rdev,</div><div class="line"> readahead, readlink, readprofile, realpath, reboot, reformime,</div><div class="line"> <span class="built_in">remove</span>-<span class="keyword">shell</span>, renice, reset, <span class="keyword">resize</span>, rev, rm, rmdir, rmmod, route, rpm,</div><div class="line"> rpm2cpio, rtcwake, run-parts, runsv, runsvdir, rx, script,</div><div class="line"> scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont,</div><div class="line"> setkeycodes, setlogcons, setserial, setsid, setuidgid, <span class="keyword">sh</span>, sha1sum,</div><div class="line"> sha256sum, sha3sum, sha512sum, showkey, shuf, slattach, <span class="keyword">sleep</span>, smemcap,</div><div class="line"> softlimit, <span class="keyword">sort</span>, <span class="keyword">split</span>, start-<span class="keyword">stop</span>-daemon, stat, strings, stty, su,</div><div class="line"> sulogin, sum, <span class="keyword">sv</span>, svc, svlogd, swapoff, swapon, switch_root, <span class="keyword">sync</span>,</div><div class="line"> sysctl, syslogd, tac, tail, tar, tcpsvd, tee, telnet, telnetd, test,</div><div class="line"> tftp, tftpd, time, timeout, top, touch, <span class="keyword">tr</span>, traceroute, traceroute6,</div><div class="line"> true, truncate, tty, ttysize, tunctl, ubiattach, ubidetach, ubimkvol,</div><div class="line"> ubirename, ubirmvol, ubirsvol, ubiupdatevol, udhcpc, udhcpd, udpsvd,</div><div class="line"> uevent, umount, uname, unexpand, uniq, unix2dos, unlink, unlzma,</div><div class="line"> unlzop, unxz, unzip, uptime, usleep, uudecode, uuencode, vconfig, <span class="keyword">vi</span>,</div><div class="line"> vlock, volname, watch, watchdog, wc, wget, which, whoami, whois, xargs,</div><div class="line"> xz, xzcat, yes, zcat, zcip</div></pre></td></tr></table></figure>
<h4 id="FAQ"><a href="#FAQ" class="headerlink" title="FAQ:"></a>FAQ:</h4><figure class="highlight vim"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">Q:adb push命令出现错误：ad<span class="variable">b:</span> error: failed <span class="keyword">to</span> <span class="keyword">copy</span> <span class="string">'***'</span> <span class="keyword">to</span> <span class="string">'/data/***'</span>: Read-<span class="keyword">only</span> <span class="keyword">file</span> <span class="built_in">system</span></div><div class="line"></div><div class="line">A:请在模拟器内，使用RE，手动挂载为可读写，再次重试</div></pre></td></tr></table></figure>
<h3 id="inotify"><a href="#inotify" class="headerlink" title="inotify"></a>inotify</h3><p>inotify是用来监视文件系统事件的机制，在linux 2.6.13内核中引入。该机制可以用来监视文件和目录，当文件或目录发生变化时，内核会将文件或目录的变化发送给inotify文件描述符，在应用层只需调用read()就可以读取这些事件，非常的方便。更好的是，inotify文件描述符还可以使用select、poll、epoll这些接口来监听，当有事件发生是，inotify文件描述符会可读。</p>
<p>Genymotion模拟器也支持该特性，内核版本：root@vbox86p:/ # uname -r ——–> 3.10.0-genymotion-g08e528d</p>
<ul>
<li><a href="http://blog.csdn.net/justlinux2010/article/details/8680585" target="_blank" rel="external">Linux下监测目录或文件的变化</a></li>
<li><a href="系统监控工具----Inotify-Tools">系统监控工具—-Inotify-Tools</a></li>
</ul>
]]></content>
<categories>
<category> Record </category>
</categories>
<tags>
<tag> Geek </tag>
<tag> Android </tag>
<tag> Chrome </tag>
<tag> Chromium </tag>
<tag> SourceCode </tag>
<tag> SQLite </tag>
<tag> JSON </tag>
<tag> Fastjson </tag>
<tag> Java </tag>
<tag> C++ </tag>
<tag> SourceInsight </tag>
<tag> SublimText </tag>
<tag> EditPlus </tag>
</tags>
</entry>
<entry>
<title><![CDATA[Geek - Flyme书签导出到Via浏览器]]></title>
<url>https://amao12580.github.io/post/2016/09/flyme-broswer-bookmarks-export/</url>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>结尾部分的解析导出代码，已经整理完毕：<a href="https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java" target="_blank" rel="external">https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java</a></p>
<p>换了神族手机一年多，感觉各种受限制，没有官解Bootload，ROM没得刷，也就只能在线主题买买买，给老黄贡献订单，其实整来整去就那点界面变化。不Root也被各种广告、内置应用通知栏弹幕，索性Root折腾一把了！</p>
<p>Flyme内置浏览器（(V 4.5.0033)），不那么好用，实用的自定义UA功能不支持（突破各种云盘下载你懂的），我想android还是开放自由的吧，应该换个<a href="http://www.coolapk.com/apk/mark.via" target="_blank" rel="external">Via浏览器</a>(V 1.8.4，体积不超过300k，超轻量，可玩性也非常高)，再把Flyme浏览器里50+书签导出来，却发现没有导入导出书签的设置，WTF？？？</p>
<p><img src="/img/flyme-broswer-setting.jpg" alt=""><img src="/img/flyme-broswer-bookmarks.jpg" alt=""></p>
<p>好在咱也不是技术小白啊，这点小事，还是可以搞定的！</p>
<h1 id="实验"><a href="#实验" class="headerlink" title="实验"></a>实验</h1><p>既然魅族浏览器，在UI界面上没有提供书签导入导出，那我保存的书签记录在哪里呢？猜想是在/data/data/*目录，折腾前为防止数据丢失，先用<a href="http://www.coolapk.com/apk/com.keramidas.TitaniumBackup" target="_blank" rel="external">钛备份</a> backup一下魅族浏览器的程序和数据吧！</p>
<p>完成备份后，关闭魅族浏览器以及强制停止进程，尝试在内置存储目录（/storage/emulated/0/），使用<a href="http://www.coolapk.com/apk/com.speedsoftware.rootexplorer" target="_blank" rel="external">RE文件管理器</a>删除相关文件夹：/storage/emulated/0/.com.android.broswer/，重新打开，发现书签和设置并没有丢失！而且/storage/emulated/0/.com.android.broswer/文件夹又自动生成了！可见这个文件夹存放了一些临时文件，对用户的设置和书签数据不影响。</p>
<p>在/data/data/目录下（需Root才能读写），找了一番，定位到魅族浏览器的数据目录：/data/data/com.android.broswer/，啊这个文件夹还真挺丰富的！<br><img src="/img/flyme-broswer-datas.jpg" alt=""></p>
<p>究竟那个才是我们的书签存放目录呢？？？</p>
<h1 id="定位"><a href="#定位" class="headerlink" title="定位"></a>定位</h1><p>我想书签是一个用户的常用功能，会存在添加书签和删除的操作，应该会把数据放在一个适合快速存储和妥善保存的地方，比如说：database？还真有/data/data/com.android.broswer/databases/这个目录，啊哈，又近了一步呢！这个文件夹下都有些啥？竟是十多个数据库和临时WAL文件！<br><img src="/img/flyme-broswer-databases.jpg" alt=""><br>哪一个才是存放书签的呢？上排除法！根据一些命名过滤掉一些不可能的选项，余下的就是correct。正确答案是：/data/data/com.android.broswer/databases/broswer2.db这个文件，一起打开看看！（长按，右上角，选择打开方式，SQLite数据库浏览器）<br><img src="/img/flyme-broswer-databases-db2.jpg" alt=""></p>
<p>有很多大家熟悉的db：元数据、书签、历史访问、最常访问、图片、最近搜索、用户设置、账户同步。相信大家都知道选哪个了：bookmarks。终于看到需要的信息，所有的书签都在这里哦，想想办法弄出来吧！<br><img src="/img/flyme-broswer-databases-db2-bookmarks.jpg" alt=""></p>
<h1 id="导出"><a href="#导出" class="headerlink" title="导出"></a>导出</h1><p>为了编辑方便，我们将/data/data/com.android.broswer/databases/broswer2.db导出到PC操作，安装Navicat Premium（version:11.1.12），新建连接，选择SQLite。<br><img src="/img/flyme-broswer-databases-db2-bookmarks-navicat.jpg" alt=""></p>
<p>打开连接，选择：bookmarks，发现在、数据与RE管理器里看到是一样的！<br><img src="/img/flyme-broswer-databases-db2-bookmarks-navicat-datas.jpg" alt=""><br>我们直接使用navicat的数据导出向导，导出为json文件（还有很多格式可以选择，json方便下一步处理）bookmarks.json，可用的字段是title、url。<br><figure class="highlight perl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div><div class="line">226</div><div class="line">227</div><div class="line">228</div><div class="line">229</div><div class="line">230</div><div class="line">231</div></pre></td><td class="code"><pre><div class="line"><span class="comment">#表结构：</span></div><div class="line"></div><div class="line">CREATE TABLE bookmarks(_id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT,url TEXT,host TEXT, folder INTEGER NOT NULL DEFAULT <span class="number">0</span>,parent INTEGER,position INTEGER NOT NULL,insert_after INTEGER,deleted INTEGER NOT NULL DEFAULT <span class="number">0</span>,account_name TEXT,account_type TEXT,sourceid TEXT,version INTEGER NOT NULL DEFAULT <span class="number">1</span>,created INTEGER,modified INTEGER,dirty INTEGER NOT NULL DEFAULT <span class="number">0</span>,sync1 TEXT,sync2 TEXT,sync3 TEXT,sync4 TEXT,sync5 TEXT,mapping INTEGER NOT NULL DEFAULT <span class="number">0</span>);</div><div class="line"></div><div class="line"><span class="comment">#查询title、url两个字段</span></div><div class="line"></div><div class="line">SELECT title,url from bookmarks;</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">{</div><div class="line"><span class="string">"RECORDS"</span>:[</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"默认目录"</span>,</div><div class="line"><span class="string">"url"</span>:null</div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Google 镜像站搜集 - Techzero"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.itechzero.com\/google-mirror-sites-collect.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.baidu.com\/from=879a\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.1_18_\/baiduid=4C04AB048A505E6697DCB5C3C808A263\/w=0_10_oop+ooa+ood\/t=iphone\/l=3\/tc?ref=www_iphone&lid=2022227202196170648&order=5&vit=osres&tj=www_normal_5_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=9624&di=1c941110077479c6&bdenc=1&tch=124.237.42.912.2.930&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-7agTCcgsFOd7GdWWUm7hp0rK&eqid=1c1064ed1a08e8001000000556a6ba3c&wd="</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java NIO和IO的区别_java_脚本之家"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.jb51.net\/article\/50621.htm"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.importnew.com\/1993.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"java常用设计模式 - Leo Chin - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.cnblogs.com\/hnrainll\/archive\/2011\/12\/29\/2305582.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"正则表达式给力小伙伴 | 行思錄 | Travel Coder"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/liudanking.me\/smarttool\/regexp_smarttool\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"进阶正则表达式 - Barret Lee - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.cnblogs.com\/hustskyking\/p\/how-regular-expressions-work.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.myexception.cn\/operating-system\/1932122.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/my.oschina.net\/u\/554660\/blog\/284653"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/blog.csdn.net\/shagoo\/article\/details\/8191346"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"深圳市五险一金及税后工资计算器"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/salarycalculator.sinaapp.com\/city\/shenzhen"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"多图详解Spring框架的设计理念与设计模式(1) - 51CTO.COM"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/developer.51cto.com\/art\/201006\/205212.htm"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"java中的设计模式如何分类"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/zhidao.baidu.com\/question\/223541008.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=0&from=879a&uid=0&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java 设计模式分类"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=4732764"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"来自投资银行的20个Java面试题 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/20-java-interview-questions-from-investment-banks.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/bbym010.iteye.com\/blog\/2100868"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"HAProxy简介 - 张善友 - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.cnblogs.com\/shanyou\/archive\/2012\/10\/16\/2726768.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"什么是“极限编程”？"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/zhidao.baidu.com\/question\/5153690.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=0&from=879a&uid=0&pu=usm@1,sz@1320_1004,ta@iphone_2_5.1_18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"MongoDB Sharding机制分析 » 囧囧孙"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.jiongsun.com\/2013\/03\/45.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.baidu.com\/from=1001560s\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.1_3_537\/baiduid=4C04AB048A505E6697DCB5C3C808A263\/w=0_10_kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF\/t=iphone\/l=3\/tc?ref=www_iphone&lid=11304546645738609731&order=1&vit=osres&tj=www_normal_1_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D..._%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=10463&di=b0551844ad65db76&bdenc=1&tch=124.0.0.0.0.0&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=9ce1d147be1560001000000356d4d7f0&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"互联网系统可靠性基础：正确的异常处理"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.tuicool.com\/m\/articles\/UvayInY"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"服务器端 概述 - 极光文档"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/docs.jpush.io\/server\/server_overview\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"从ACID到CAP到BASE - xixicat - SegmentFault"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/segmentfault.com\/a\/1190000004468442?url_type=39&object_type=webpage&pos=1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Netty系列之Netty可靠性分析"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.infoq.com\/cn\/articles\/netty-reliability\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.tuicool.com\/m\/articles\/feYfInn"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"有趣的bloom过滤器 | OurMySQL"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/ourmysql.com\/archives\/510?f=wb"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=27109467"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/java-string-ansi-unicode-bmp-utf.html?url_type=39&object_type=webpage&pos=1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"看了极光推送技术原理的几点思考"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=8227469"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"腾讯微信技术架构"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=8581852"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"鸟哥的 Linux 私房菜 -- 学习 bash shell"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/vbird.dic.ksu.edu.tw\/linux_basic\/0320bash.php"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/www.zhihu.com\/question\/20962240"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"动态规划：从新手到专家"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.hawstein.com\/posts\/dp-novice-to-advanced.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"商品搜索引擎—推荐系统设计 - ImportNew"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.importnew.com\/19834.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/coolshell.cn\/articles\/6790.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"32个算法"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/media.weibo.cn\/article?id=2309403979794248269609&jumpfrom=weibocom"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java对象之死-云栖社区"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/yq.aliyun.com\/articles\/53681?spm=5176.100239.bloglist.55.lOez9A"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"amao12580 · GitHub"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/github.com\/amao12580\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"微服务与架构师"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA3MDMwOTcwMg==&mid=2650004550&idx=1&sn=b2299d1f2456541fc6b600a480c27dd8&scene=1&srcid=0714I4IHCwFfqnIGigCv3pJb#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"图灵社区 : 阅读 : 什么时候你不应该使用微服务"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.ituring.com.cn\/article\/217430"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"微服务的两种模式：应用中心和任务中心 - DockOne.io"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/dockone.io\/m\/article\/1335"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/blogread.cn\/it\/wap\/article\/7770?f=wb"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"一个资深架构师是这样理解数据结构的"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"高性能服务器架构思路 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java编程常见问题汇总 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/blogread.cn\/it\/wap\/article\/7816?f=wb"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"微服务实战：从架构到部署 - DockOne.io"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/dockone.io\/article\/1519"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"优酷蓝鲸近千节点的Redis集群运维经验总结"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"spring ioc DI 理解 - WhyWin - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/zm6.sm-tc.cn\/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/dockone.io\/article\/1582"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"技改之路：从单块应用到微服务，我的血泪总结"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"华为内部如何实施微服务架构？基本就靠这5大原则"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"悲观的看：互联网技术岗位红利期逐渐到头"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/media.weibo.cn\/article?id=2309404020851660960391&jumpfrom=weibocom"</span></div><div class="line">}</div><div class="line">]</div><div class="line">}</div></pre></td></tr></table></figure></p>
<h1 id="导入"><a href="#导入" class="headerlink" title="导入"></a>导入</h1><p>目前各家浏览器的导入导出格式并没有统一标准，我们来看看Via 浏览器的导出文件：/storage/emulated/0/Download/Via_书签_20160929.txt，与导出的文件一致，也是json格式。</p>
<figure class="highlight json"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">{<span class="attr">"title"</span>:<span class="string">"Cat's Blog - 一饮一啄，莫非前定. - Steven Cheng"</span>,<span class="attr">"url"</span>:<span class="string">"https:\/\/amao12580.github.io\/"</span>,<span class="attr">"folder"</span>:<span class="string">""</span>,<span class="attr">"order"</span>:<span class="number">0</span>}</div><div class="line">{<span class="attr">"title"</span>:<span class="string">"十周年 - Google 搜索"</span>,<span class="attr">"url"</span>:<span class="string">"https:\/\/www.google.com.hk\/search?q=%E5%8D%81%E5%91%A8%E5%B9%B4"</span>,<span class="attr">"folder"</span>:<span class="string">""</span>,<span class="attr">"order"</span>:<span class="number">0</span>}</div></pre></td></tr></table></figure>
<p>既然数据格式一致，剩下的事就简单了，只需要将魅族浏览器导出的broswer2.db文件整理成Via可接受的结构就行，写一段小程序吧，decode + code！</p>
<figure class="highlight perl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> org.jooq.example;</div><div class="line"></div><div class="line">import java.util.List;</div><div class="line"></div><div class="line">/**</div><div class="line"> * Created with IntelliJ IDEA.</div><div class="line"> * User:ChengLiang</div><div class="line"> * Date:<span class="number">2016</span>/<span class="number">9</span>/<span class="number">29</span></div><div class="line"> * Time:<span class="number">15</span>:<span class="number">03</span></div><div class="line"> *<span class="regexp">/</span></div><div class="line"><span class="regexp">@lombok.Data</span></div><div class="line"><span class="regexp">public class MeizuBookmark {</span></div><div class="line"><span class="regexp"> List<item> RECORDS;</span></div><div class="line"><span class="regexp"> @lombok.Data</span></div><div class="line"><span class="regexp"> class item{</span></div><div class="line"><span class="regexp"> String title;</span></div><div class="line"><span class="regexp"> String url;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp">}</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">package org.jooq.example;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">import com.alibaba.fastjson.annotation.JSONField;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">/</span>**</div><div class="line"> * Created with IntelliJ IDEA.</div><div class="line"> * User:ChengLiang</div><div class="line"> * Date:<span class="number">2016</span>/<span class="number">9</span>/<span class="number">29</span></div><div class="line"> * Time:<span class="number">15</span>:<span class="number">03</span></div><div class="line"> *<span class="regexp">/</span></div><div class="line"><span class="regexp">@lombok.Data</span></div><div class="line"><span class="regexp">public class ViaBookmark {</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 1)</span></div><div class="line"><span class="regexp"> String title;</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 2)</span></div><div class="line"><span class="regexp"> String url;</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 3)</span></div><div class="line"><span class="regexp"> String folder = "";</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 4)</span></div><div class="line"><span class="regexp"> int order = 0;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"> public ViaBookmark(String title, String url,int order) {</span></div><div class="line"><span class="regexp"> this.title=title;</span></div><div class="line"><span class="regexp"> this.url=url;</span></div><div class="line"><span class="regexp"> this.order=order;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp">}</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">package org.jooq.example;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">import com.alibaba.fastjson.JSON;</span></div><div class="line"><span class="regexp">import com.alibaba.fastjson.JSONReader;</span></div><div class="line"><span class="regexp">import org.junit.Test;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">import java.io.File;</span></div><div class="line"><span class="regexp">import java.io.FileNotFoundException;</span></div><div class="line"><span class="regexp">import java.io.FileReader;</span></div><div class="line"><span class="regexp">import java.util.ArrayList;</span></div><div class="line"><span class="regexp">import java.util.List;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">/</span>**</div><div class="line"> * Created with IntelliJ IDEA.</div><div class="line"> * User:ChengLiang</div><div class="line"> * Date:<span class="number">2016</span>/<span class="number">9</span>/<span class="number">29</span></div><div class="line"> * Time:<span class="number">15</span>:<span class="number">00</span></div><div class="line"> * <p></div><div class="line"> * fastJson 解析魅族浏览器书签json文件，输出via浏览可导入的结构，整理为json文件</div><div class="line"> *<span class="regexp">/</span></div><div class="line"><span class="regexp">public class BroswerTest {</span></div><div class="line"><span class="regexp"> @Test</span></div><div class="line"><span class="regexp"> public void readJSONFromFile() throws FileNotFoundException {</span></div><div class="line"><span class="regexp"> String fullPath = "C:\\Users\\Administrator\\Desktop\\bookmarks3.json";</span></div><div class="line"><span class="regexp"> JSONReader jsonReader = new JSONReader(new FileReader(new File(fullPath)));</span></div><div class="line"><span class="regexp"> MeizuBookmark meizuBookmark = jsonReader.readObject(MeizuBookmark.class);</span></div><div class="line"><span class="regexp"> jsonReader.close();</span></div><div class="line"><span class="regexp"> System.out.println("meizuBookmark:" + JSON.toJSONString(meizuBookmark));</span></div><div class="line"><span class="regexp"> if (meizuBookmark == null || meizuBookmark.getRECORDS() == null || meizuBookmark.getRECORDS().isEmpty()) {</span></div><div class="line"><span class="regexp"> System.out.println("解码失败，或源文件没有书签");</span></div><div class="line"><span class="regexp"> return;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> int index = 0;</span></div><div class="line"><span class="regexp"> int seq = 0;</span></div><div class="line"><span class="regexp"> System.out.println("完成解码，开始解析书签.");</span></div><div class="line"><span class="regexp"> List<ViaBookmark> results=new ArrayList<>();</span></div><div class="line"><span class="regexp"> for (MeizuBookmark.item item : meizuBookmark.getRECORDS()) {</span></div><div class="line"><span class="regexp"> index++;</span></div><div class="line"><span class="regexp"> System.out.println("第" + index + "条书签：" + JSON.toJSONString(item));</span></div><div class="line"><span class="regexp"> if(item.getTitle()==null||item.getTitle().isEmpty()||item.getUrl()==null||item.getUrl().isEmpty()){</span></div><div class="line"><span class="regexp"> System.out.println("数据不完整，跳过处理.");</span></div><div class="line"><span class="regexp"> continue;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> results.add(new ViaBookmark(item.getTitle(),item.getUrl(),seq));</span></div><div class="line"><span class="regexp"> seq++;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> System.out.println("viaBookmark:");</span></div><div class="line"><span class="regexp"> for(ViaBookmark viaBookmark:results){</span></div><div class="line"><span class="regexp"> System.out.println(JSON.toJSONString(viaBookmark));</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> System.out.println("完成解析书签.");</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp">}</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">Console：</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">"D:\Program Files\Java\jdk1.8.0_74\bin\java" -ea -Didea.launcher.port=7537 -Didea.launcher.bin.path=D:\IDEA14\bin -Dfile.encoding=UTF-8 -classpath "D:\IDEA14\lib\idea_rt.jar;D:\IDEA14\plugins\junit\lib\junit-rt.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\charsets.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\deploy.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\access-bridge-64.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\cldrdata.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\dnsns.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\jaccess.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\jfxrt.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\localedata.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\nashorn.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunec.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunjce_provider.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunmscapi.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunpkcs11.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\zipfs.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\javaws.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jce.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jfr.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jfxswt.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jsse.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\management-agent.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\plugin.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\resources.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\rt.jar;D:\GitHub\JOOQ-With-Spring\target\test-classes;D:\GitHub\JOOQ-With-Spring\target\classes;D:\maven\repository\org\apache\logging\log4j\log4j-api\2.6.2\log4j-api-2.6.2.jar;D:\maven\repository\org\apache\logging\log4j\log4j-core\2.6.2\log4j-core-2.6.2.jar;D:\maven\repository\org\slf4j\slf4j-api\1.7.21\slf4j-api-1.7.21.jar;D:\maven\repository\org\apache\logging\log4j\log4j-slf4j-impl\2.6.2\log4j-slf4j-impl-2.6.2.jar;D:\maven\repository\org\springframework\spring-context\4.2.5.RELEASE\spring-context-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-aop\4.2.5.RELEASE\spring-aop-4.2.5.RELEASE.jar;D:\maven\repository\aopalliance\aopalliance\1.0\aopalliance-1.0.jar;D:\maven\repository\org\springframework\spring-beans\4.2.5.RELEASE\spring-beans-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-core\4.2.5.RELEASE\spring-core-4.2.5.RELEASE.jar;D:\maven\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;D:\maven\repository\org\springframework\spring-expression\4.2.5.RELEASE\spring-expression-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-jdbc\4.2.5.RELEASE\spring-jdbc-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-tx\4.2.5.RELEASE\spring-tx-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-webmvc\4.2.5.RELEASE\spring-webmvc-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-web\4.2.5.RELEASE\spring-web-4.2.5.RELEASE.jar;D:\maven\repository\org\aspectj\aspectjrt\1.5.4\aspectjrt-1.5.4.jar;D:\maven\repository\org\aspectj\aspectjweaver\1.8.9\aspectjweaver-1.8.9.jar;D:\maven\repository\javax\servlet\javax.servlet-api\4.0.0-b01\javax.servlet-api-4.0.0-b01.jar;D:\maven\repository\javax\servlet\jsp\jsp-api\2.2.1-b03\jsp-api-2.2.1-b03.jar;D:\maven\repository\junit\junit\4.12\junit-4.12.jar;D:\maven\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;D:\maven\repository\org\springframework\spring-test\4.2.5.RELEASE\spring-test-4.2.5.RELEASE.jar;D:\maven\repository\org\jooq\jooq-codegen\3.8.4\jooq-codegen-3.8.4.jar;D:\maven\repository\org\jooq\jooq\3.8.4\jooq-3.8.4.jar;D:\maven\repository\org\jooq\jooq-meta\3.8.4\jooq-meta-3.8.4.jar;D:\maven\repository\mysql\mysql-connector-java\5.1.38\mysql-connector-java-5.1.38.jar;D:\maven\repository\com\zaxxer\HikariCP\2.4.7\HikariCP-2.4.7.jar;D:\maven\repository\org\projectlombok\lombok\1.16.8\lombok-1.16.8.jar;D:\maven\repository\joda-time\joda-time\2.9.4\joda-time-2.9.4.jar;D:\maven\repository\com\alibaba\fastjson\1.2.15\fastjson-1.2.15.jar" com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 org.jooq.example.BroswerTest,readJSONFromFile</span></div><div class="line"><span class="regexp">meizuBookmark:{"rECORDS":[{"title":"默认目录"},{"title":"Google 镜像站搜集 - Techzero","url":"http:/</span><span class="regexp">/www.itechzero.com/google</span>-mirror-sites-collect.html<span class="string">"},{"</span>title<span class="string">":"</span>OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.baidu.com/from=<span class="number">879</span>a/bd_page_type=<span class="number">1</span>/ssid=<span class="number">0</span>/uid=<span class="number">0</span>/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.<span class="number">1_18_</span>/baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263/w=<span class="number">0_10_</span>oop+ooa+ood/t=iphone/l=<span class="number">3</span>/tc?<span class="keyword">ref</span>=www_iphone&lid=<span class="number">2022227202196170648</span>&order=<span class="number">5</span>&vit=osres&tj=www_normal_5_0_10_title&<span class="keyword">m</span>=<span class="number">8</span>&srd=<span class="number">1</span>&cltj=cloud_title&dict=<span class="number">30</span>&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=<span class="number">9624</span>&di=<span class="number">1</span>c941110077479c6&bdenc=<span class="number">1</span>&tch=<span class="number">124.237</span>.<span class="number">42.912</span>.<span class="number">2.930</span>&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-<span class="number">7</span>agTCcgsFOd7GdWWUm7hp0rK&eqid=<span class="number">1</span>c1064ed1a08e8001000000556a6ba3c&wd=<span class="string">"},{"</span>title<span class="string">":"</span>Java NIO和IO的区别_java<span class="number">_</span>脚本之家<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.jb51.net/article/<span class="number">50621</span>.htm<span class="string">"},{"</span>title<span class="string">":"</span>成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.importnew.com/<span class="number">1993</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>java常用设计模式 - Leo Chin - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.cnblogs.com/hnrainll/archive/<span class="number">2011</span>/<span class="number">12</span>/<span class="number">29</span>/<span class="number">2305582</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>正则表达式给力小伙伴 | 行思錄 | Travel Coder<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//liudanking</span>.me/smarttool/regexp_smarttool/<span class="string">"},{"</span>title<span class="string">":"</span>进阶正则表达式 - Barret Lee - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.cnblogs.com/hustskyking/p/how-regular-expressions-work.html<span class="string">"},{"</span>title<span class="string">":"</span>[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.myexception.cn/operating-<span class="keyword">system</span>/<span class="number">1932122</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//my</span>.oschina.net/u/<span class="number">554660</span>/blog/<span class="number">284653</span><span class="string">"},{"</span>title<span class="string">":"</span>[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//blog</span>.csdn.net/shagoo/article/details/<span class="number">8191346</span><span class="string">"},{"</span>title<span class="string">":"</span>深圳市五险一金及税后工资计算器<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//salarycalculator</span>.sinaapp.com/city/shenzhen<span class="string">"},{"</span>title<span class="string">":"</span>多图详解Spring框架的设计理念与设计模式(<span class="number">1</span>) - <span class="number">51</span>CTO.COM<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//developer</span>.<span class="number">51</span>cto.com/art/<span class="number">201006</span>/<span class="number">205212</span>.htm<span class="string">"},{"</span>title<span class="string">":"</span>java中的设计模式如何分类<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//zhidao</span>.baidu.com/question/<span class="number">223541008</span>.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=<span class="number">0</span>&from=<span class="number">879</span>a&uid=<span class="number">0</span>&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=<span class="number">1</span>&baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1<span class="string">"},{"</span>title<span class="string">":"</span>Java 设计模式分类<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">4732764</span><span class="string">"},{"</span>title<span class="string">":"</span>来自投资银行的<span class="number">20</span>个Java面试题 – 码农网<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.codeceo.com/article/<span class="number">20</span>-java-interview-questions-from-investment-banks.html<span class="string">"},{"</span>title<span class="string">":"</span>JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//bbym</span>01<span class="number">0</span>.iteye.com/blog/<span class="number">2100868</span><span class="string">"},{"</span>title<span class="string">":"</span>HAProxy简介 - 张善友 - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.cnblogs.com/shanyou/archive/<span class="number">2012</span>/<span class="number">10</span>/<span class="number">16</span>/<span class="number">2726768</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>什么是“极限编程”？<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//zhidao</span>.baidu.com/question/<span class="number">5153690</span>.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=<span class="number">0</span>&from=<span class="number">879</span>a&uid=<span class="number">0</span>&pu=usm@1,sz@1320_1004,ta@iphone_2_5.<span class="number">1_18_</span>&bd_page_type=<span class="number">1</span>&baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1<span class="string">"},{"</span>title<span class="string">":"</span>MongoDB Sharding机制分析 » 囧囧孙<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.jiongsun.com/<span class="number">2013</span>/<span class="number">03</span>/<span class="number">45</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.baidu.com/from=<span class="number">1001560</span><span class="keyword">s</span>/bd_page_type=<span class="number">1</span>/ssid=<span class="number">0</span>/uid=<span class="number">0</span>/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.<span class="number">1_3_537</span>/baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263/w=<span class="number">0_10_</span>kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF/t=iphone/l=<span class="number">3</span>/tc?<span class="keyword">ref</span>=www_iphone&lid=<span class="number">11304546645738609731</span>&order=<span class="number">1</span>&vit=osres&tj=www_normal_1_0_10_title&<span class="keyword">m</span>=<span class="number">8</span>&srd=<span class="number">1</span>&cltj=cloud_title&dict=<span class="number">30</span>&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D...<span class="number">_</span>%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=<span class="number">10463</span>&di=b0551844ad65db76&bdenc=<span class="number">1</span>&tch=<span class="number">124.0</span>.<span class="number">0</span>.<span class="number">0</span>.<span class="number">0</span>.<span class="number">0</span>&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=<span class="number">9</span>ce1d147be1560001000000356d4d7f<span class="number">0</span>&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D<span class="string">"},{"</span>title<span class="string">":"</span>互联网系统可靠性基础：正确的异常处理<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.tuicool.com/<span class="regexp">m/articles/</span>UvayInY<span class="string">"},{"</span>title<span class="string">":"</span>服务器端 概述 - 极光文档<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//docs</span>.jpush.io/server/server_overview/<span class="string">"},{"</span>title<span class="string">":"</span>从ACID到CAP到BASE - xixicat - SegmentFault<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//segmentfault</span>.com/a/<span class="number">1190000004468442</span>?url_type=<span class="number">39</span>&object_type=webpage&<span class="keyword">pos</span>=<span class="number">1</span><span class="string">"},{"</span>title<span class="string">":"</span>Netty系列之Netty可靠性分析<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.infoq.com/cn/articles/netty-reliability/<span class="string">"},{"</span>title<span class="string">":"</span>【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.tuicool.com/<span class="regexp">m/articles/fe</span>YfInn<span class="string">"},{"</span>title<span class="string">":"</span>有趣的bloom过滤器 | OurMySQL<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//ourmysql</span>.com/archives/<span class="number">510</span>?f=wb<span class="string">"},{"</span>title<span class="string">":"</span>Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">27109467</span><span class="string">"},{"</span>title<span class="string">":"</span>从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.codeceo.com/article/java-string-ansi-unicode-bmp-utf.html?url_type=<span class="number">39</span>&object_type=webpage&<span class="keyword">pos</span>=<span class="number">1</span><span class="string">"},{"</span>title<span class="string">":"</span>看了极光推送技术原理的几点思考<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">8227469</span><span class="string">"},{"</span>title<span class="string">":"</span>腾讯微信技术架构<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">8581852</span><span class="string">"},{"</span>title<span class="string">":"</span>鸟哥的 Linux 私房菜 -- 学习 bash shell<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//vbird</span>.dic.ksu.edu.tw/linux_basic/<span class="number">0320</span>bash.php<span class="string">"},{"</span>title<span class="string">":"</span>如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//www</span>.zhihu.com/question/<span class="number">20962240</span><span class="string">"},{"</span>title<span class="string">":"</span>动态规划：从新手到专家<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.hawstein.com/posts/dp-novice-to-advanced.html<span class="string">"},{"</span>title<span class="string">":"</span>商品搜索引擎—推荐系统设计 - ImportNew<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.importnew.com/<span class="number">19834</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//coolshell</span>.cn/articles/<span class="number">6790</span>.html<span class="string">"},{"</span>title<span class="string">":"</span><span class="number">32</span>个算法<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//media</span>.weibo.cn/article?id=<span class="number">2309403979794248269609</span>&jumpfrom=weibocom<span class="string">"},{"</span>title<span class="string">":"</span>Java对象之死-云栖社区<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//yq</span>.aliyun.com/articles/<span class="number">53681</span>?spm=<span class="number">5176.100239</span>.bloglist.<span class="number">55</span>.lOez9A<span class="string">"},{"</span>title<span class="string">":"</span>amao1258<span class="number">0</span> · GitHub<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//github</span>.com/amao1258<span class="number">0</span>/<span class="string">"},{"</span>title<span class="string">":"</span>微服务与架构师<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//mp</span>.weixin.qq.com/<span class="keyword">s</span>?__biz=MzA3MDMwOTcwMg==&mid=<span class="number">2650004550</span>&idx=<span class="number">1</span>&sn=b2299d1f2456541fc6b600a480c27dd8&scene=<span class="number">1</span>&srcid=<span class="number">0714</span>I4IHCwFfqnIGigCv3pJb<span class="comment">#rd"},{"title":"图灵社区 : 阅读 : 什么时候你不应该使用微服务","url":"http://www.ituring.com.cn/article/217430"},{"title":"微服务的两种模式：应用中心和任务中心 - DockOne.io","url":"http://dockone.io/m/article/1335"},{"title":"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！","url":"http://blogread.cn/it/wap/article/7770?f=wb"},{"title":"一个资深架构师是这样理解数据结构的","url":"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"},{"title":"高性能服务器架构思路 – 码农网","url":"http://www.codeceo.com/article/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"},{"title":"Java编程常见问题汇总 – 码农网","url":"http://www.codeceo.com/article/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"},{"title":"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！","url":"http://blogread.cn/it/wap/article/7816?f=wb"},{"title":"微服务实战：从架构到部署 - DockOne.io","url":"http://dockone.io/article/1519"},{"title":"优酷蓝鲸近千节点的Redis集群运维经验总结","url":"http://mp.weixin.qq.com/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"},{"title":"spring ioc DI 理解 - WhyWin - 博客园","url":"https://zm6.sm-tc.cn/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"},{"title":"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io","url":"http://dockone.io/article/1582"},{"title":"技改之路：从单块应用到微服务，我的血泪总结","url":"http://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"},{"title":"华为内部如何实施微服务架构？基本就靠这5大原则","url":"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"},{"title":"悲观的看：互联网技术岗位红利期逐渐到头","url":"http://media.weibo.cn/article?id=2309404020851660960391&jumpfrom=weibocom"}]}</span></div><div class="line">完成解码，开始解析书签.</div><div class="line">第<span class="number">1</span>条书签：{<span class="string">"title"</span>:<span class="string">"默认目录"</span>}</div><div class="line">数据不完整，跳过处理.</div><div class="line">第<span class="number">2</span>条书签：{<span class="string">"title"</span>:<span class="string">"Google 镜像站搜集 - Techzero"</span>,<span class="string">"url"</span>:<span class="string">"http://www.itechzero.com/google-mirror-sites-collect.html"</span>}</div><div class="line">第<span class="number">3</span>条书签：{<span class="string">"title"</span>:<span class="string">"OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=879a/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.1_18_/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_oop+ooa+ood/t=iphone/l=3/tc?ref=www_iphone&lid=2022227202196170648&order=5&vit=osres&tj=www_normal_5_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=9624&di=1c941110077479c6&bdenc=1&tch=124.237.42.912.2.930&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-7agTCcgsFOd7GdWWUm7hp0rK&eqid=1c1064ed1a08e8001000000556a6ba3c&wd="</span>}</div><div class="line">第<span class="number">4</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java NIO和IO的区别_java_脚本之家"</span>,<span class="string">"url"</span>:<span class="string">"http://m.jb51.net/article/50621.htm"</span>}</div><div class="line">第<span class="number">5</span>条书签：{<span class="string">"title"</span>:<span class="string">"成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/1993.html"</span>}</div><div class="line">第<span class="number">6</span>条书签：{<span class="string">"title"</span>:<span class="string">"java常用设计模式 - Leo Chin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hnrainll/archive/2011/12/29/2305582.html"</span>}</div><div class="line">第<span class="number">7</span>条书签：{<span class="string">"title"</span>:<span class="string">"正则表达式给力小伙伴 | 行思錄 | Travel Coder"</span>,<span class="string">"url"</span>:<span class="string">"http://liudanking.me/smarttool/regexp_smarttool/"</span>}</div><div class="line">第<span class="number">8</span>条书签：{<span class="string">"title"</span>:<span class="string">"进阶正则表达式 - Barret Lee - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hustskyking/p/how-regular-expressions-work.html"</span>}</div><div class="line">第<span class="number">9</span>条书签：{<span class="string">"title"</span>:<span class="string">"[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统"</span>,<span class="string">"url"</span>:<span class="string">"http://m.myexception.cn/operating-system/1932122.html"</span>}</div><div class="line">第<span class="number">10</span>条书签：{<span class="string">"title"</span>:<span class="string">"从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://my.oschina.net/u/554660/blog/284653"</span>}</div><div class="line">第<span class="number">11</span>条书签：{<span class="string">"title"</span>:<span class="string">"[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET"</span>,<span class="string">"url"</span>:<span class="string">"http://blog.csdn.net/shagoo/article/details/8191346"</span>}</div><div class="line">第<span class="number">12</span>条书签：{<span class="string">"title"</span>:<span class="string">"深圳市五险一金及税后工资计算器"</span>,<span class="string">"url"</span>:<span class="string">"http://salarycalculator.sinaapp.com/city/shenzhen"</span>}</div><div class="line">第<span class="number">13</span>条书签：{<span class="string">"title"</span>:<span class="string">"多图详解Spring框架的设计理念与设计模式(1) - 51CTO.COM"</span>,<span class="string">"url"</span>:<span class="string">"http://developer.51cto.com/art/201006/205212.htm"</span>}</div><div class="line">第<span class="number">14</span>条书签：{<span class="string">"title"</span>:<span class="string">"java中的设计模式如何分类"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/223541008.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=0&from=879a&uid=0&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1"</span>}</div><div class="line">第<span class="number">15</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java 设计模式分类"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=4732764"</span>}</div><div class="line">第<span class="number">16</span>条书签：{<span class="string">"title"</span>:<span class="string">"来自投资银行的20个Java面试题 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/20-java-interview-questions-from-investment-banks.html"</span>}</div><div class="line">第<span class="number">17</span>条书签：{<span class="string">"title"</span>:<span class="string">"JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站"</span>,<span class="string">"url"</span>:<span class="string">"http://bbym010.iteye.com/blog/2100868"</span>}</div><div class="line">第<span class="number">18</span>条书签：{<span class="string">"title"</span>:<span class="string">"HAProxy简介 - 张善友 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/shanyou/archive/2012/10/16/2726768.html"</span>}</div><div class="line">第<span class="number">19</span>条书签：{<span class="string">"title"</span>:<span class="string">"什么是“极限编程”？"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/5153690.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=0&from=879a&uid=0&pu=usm@1,sz@1320_1004,ta@iphone_2_5.1_18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1"</span>}</div><div class="line">第<span class="number">20</span>条书签：{<span class="string">"title"</span>:<span class="string">"MongoDB Sharding机制分析 » 囧囧孙"</span>,<span class="string">"url"</span>:<span class="string">"http://www.jiongsun.com/2013/03/45.html"</span>}</div><div class="line">第<span class="number">21</span>条书签：{<span class="string">"title"</span>:<span class="string">"kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=1001560s/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.1_3_537/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF/t=iphone/l=3/tc?ref=www_iphone&lid=11304546645738609731&order=1&vit=osres&tj=www_normal_1_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D..._%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=10463&di=b0551844ad65db76&bdenc=1&tch=124.0.0.0.0.0&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=9ce1d147be1560001000000356d4d7f0&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D"</span>}</div><div class="line">第<span class="number">22</span>条书签：{<span class="string">"title"</span>:<span class="string">"互联网系统可靠性基础：正确的异常处理"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/UvayInY"</span>}</div><div class="line">第<span class="number">23</span>条书签：{<span class="string">"title"</span>:<span class="string">"服务器端 概述 - 极光文档"</span>,<span class="string">"url"</span>:<span class="string">"http://docs.jpush.io/server/server_overview/"</span>}</div><div class="line">第<span class="number">24</span>条书签：{<span class="string">"title"</span>:<span class="string">"从ACID到CAP到BASE - xixicat - SegmentFault"</span>,<span class="string">"url"</span>:<span class="string">"https://segmentfault.com/a/1190000004468442?url_type=39&object_type=webpage&pos=1"</span>}</div><div class="line">第<span class="number">25</span>条书签：{<span class="string">"title"</span>:<span class="string">"Netty系列之Netty可靠性分析"</span>,<span class="string">"url"</span>:<span class="string">"http://www.infoq.com/cn/articles/netty-reliability/"</span>}</div><div class="line">第<span class="number">26</span>条书签：{<span class="string">"title"</span>:<span class="string">"【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/feYfInn"</span>}</div><div class="line">第<span class="number">27</span>条书签：{<span class="string">"title"</span>:<span class="string">"有趣的bloom过滤器 | OurMySQL"</span>,<span class="string">"url"</span>:<span class="string">"http://ourmysql.com/archives/510?f=wb"</span>}</div><div class="line">第<span class="number">28</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=27109467"</span>}</div><div class="line">第<span class="number">29</span>条书签：{<span class="string">"title"</span>:<span class="string">"从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-string-ansi-unicode-bmp-utf.html?url_type=39&object_type=webpage&pos=1"</span>}</div><div class="line">第<span class="number">30</span>条书签：{<span class="string">"title"</span>:<span class="string">"看了极光推送技术原理的几点思考"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8227469"</span>}</div><div class="line">第<span class="number">31</span>条书签：{<span class="string">"title"</span>:<span class="string">"腾讯微信技术架构"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8581852"</span>}</div><div class="line">第<span class="number">32</span>条书签：{<span class="string">"title"</span>:<span class="string">"鸟哥的 Linux 私房菜 -- 学习 bash shell"</span>,<span class="string">"url"</span>:<span class="string">"http://vbird.dic.ksu.edu.tw/linux_basic/0320bash.php"</span>}</div><div class="line">第<span class="number">33</span>条书签：{<span class="string">"title"</span>:<span class="string">"如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎"</span>,<span class="string">"url"</span>:<span class="string">"https://www.zhihu.com/question/20962240"</span>}</div><div class="line">第<span class="number">34</span>条书签：{<span class="string">"title"</span>:<span class="string">"动态规划：从新手到专家"</span>,<span class="string">"url"</span>:<span class="string">"http://www.hawstein.com/posts/dp-novice-to-advanced.html"</span>}</div><div class="line">第<span class="number">35</span>条书签：{<span class="string">"title"</span>:<span class="string">"商品搜索引擎—推荐系统设计 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/19834.html"</span>}</div><div class="line">第<span class="number">36</span>条书签：{<span class="string">"title"</span>:<span class="string">"多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn"</span>,<span class="string">"url"</span>:<span class="string">"http://coolshell.cn/articles/6790.html"</span>}</div><div class="line">第<span class="number">37</span>条书签：{<span class="string">"title"</span>:<span class="string">"32个算法"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309403979794248269609&jumpfrom=weibocom"</span>}</div><div class="line">第<span class="number">38</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java对象之死-云栖社区"</span>,<span class="string">"url"</span>:<span class="string">"https://yq.aliyun.com/articles/53681?spm=5176.100239.bloglist.55.lOez9A"</span>}</div><div class="line">第<span class="number">39</span>条书签：{<span class="string">"title"</span>:<span class="string">"amao12580 · GitHub"</span>,<span class="string">"url"</span>:<span class="string">"https://github.com/amao12580/"</span>}</div><div class="line">第<span class="number">40</span>条书签：{<span class="string">"title"</span>:<span class="string">"微服务与架构师"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA3MDMwOTcwMg==&mid=2650004550&idx=1&sn=b2299d1f2456541fc6b600a480c27dd8&scene=1&srcid=0714I4IHCwFfqnIGigCv3pJb#rd"</span>}</div><div class="line">第<span class="number">41</span>条书签：{<span class="string">"title"</span>:<span class="string">"图灵社区 : 阅读 : 什么时候你不应该使用微服务"</span>,<span class="string">"url"</span>:<span class="string">"http://www.ituring.com.cn/article/217430"</span>}</div><div class="line">第<span class="number">42</span>条书签：{<span class="string">"title"</span>:<span class="string">"微服务的两种模式：应用中心和任务中心 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/m/article/1335"</span>}</div><div class="line">第<span class="number">43</span>条书签：{<span class="string">"title"</span>:<span class="string">"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7770?f=wb"</span>}</div><div class="line">第<span class="number">44</span>条书签：{<span class="string">"title"</span>:<span class="string">"一个资深架构师是这样理解数据结构的"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"</span>}</div><div class="line">第<span class="number">45</span>条书签：{<span class="string">"title"</span>:<span class="string">"高性能服务器架构思路 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"</span>}</div><div class="line">第<span class="number">46</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java编程常见问题汇总 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"</span>}</div><div class="line">第<span class="number">47</span>条书签：{<span class="string">"title"</span>:<span class="string">"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7816?f=wb"</span>}</div><div class="line">第<span class="number">48</span>条书签：{<span class="string">"title"</span>:<span class="string">"微服务实战：从架构到部署 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1519"</span>}</div><div class="line">第<span class="number">49</span>条书签：{<span class="string">"title"</span>:<span class="string">"优酷蓝鲸近千节点的Redis集群运维经验总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"</span>}</div><div class="line">第<span class="number">50</span>条书签：{<span class="string">"title"</span>:<span class="string">"spring ioc DI 理解 - WhyWin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"https://zm6.sm-tc.cn/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"</span>}</div><div class="line">第<span class="number">51</span>条书签：{<span class="string">"title"</span>:<span class="string">"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1582"</span>}</div><div class="line">第<span class="number">52</span>条书签：{<span class="string">"title"</span>:<span class="string">"技改之路：从单块应用到微服务，我的血泪总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"</span>}</div><div class="line">第<span class="number">53</span>条书签：{<span class="string">"title"</span>:<span class="string">"华为内部如何实施微服务架构？基本就靠这5大原则"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"</span>}</div><div class="line">第<span class="number">54</span>条书签：{<span class="string">"title"</span>:<span class="string">"悲观的看：互联网技术岗位红利期逐渐到头"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309404020851660960391&jumpfrom=weibocom"</span>}</div><div class="line">viaBookmark:</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Google 镜像站搜集 - Techzero"</span>,<span class="string">"url"</span>:<span class="string">"http://www.itechzero.com/google-mirror-sites-collect.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">0</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=879a/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.1_18_/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_oop+ooa+ood/t=iphone/l=3/tc?ref=www_iphone&lid=2022227202196170648&order=5&vit=osres&tj=www_normal_5_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=9624&di=1c941110077479c6&bdenc=1&tch=124.237.42.912.2.930&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-7agTCcgsFOd7GdWWUm7hp0rK&eqid=1c1064ed1a08e8001000000556a6ba3c&wd="</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">1</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java NIO和IO的区别_java_脚本之家"</span>,<span class="string">"url"</span>:<span class="string">"http://m.jb51.net/article/50621.htm"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">2</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/1993.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">3</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"java常用设计模式 - Leo Chin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hnrainll/archive/2011/12/29/2305582.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">4</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"正则表达式给力小伙伴 | 行思錄 | Travel Coder"</span>,<span class="string">"url"</span>:<span class="string">"http://liudanking.me/smarttool/regexp_smarttool/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">5</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"进阶正则表达式 - Barret Lee - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hustskyking/p/how-regular-expressions-work.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">6</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统"</span>,<span class="string">"url"</span>:<span class="string">"http://m.myexception.cn/operating-system/1932122.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">7</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://my.oschina.net/u/554660/blog/284653"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">8</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET"</span>,<span class="string">"url"</span>:<span class="string">"http://blog.csdn.net/shagoo/article/details/8191346"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">9</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"深圳市五险一金及税后工资计算器"</span>,<span class="string">"url"</span>:<span class="string">"http://salarycalculator.sinaapp.com/city/shenzhen"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">10</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"多图详解Spring框架的设计理念与设计模式(1) - 51CTO.COM"</span>,<span class="string">"url"</span>:<span class="string">"http://developer.51cto.com/art/201006/205212.htm"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">11</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"java中的设计模式如何分类"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/223541008.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=0&from=879a&uid=0&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">12</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java 设计模式分类"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=4732764"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">13</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"来自投资银行的20个Java面试题 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/20-java-interview-questions-from-investment-banks.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">14</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站"</span>,<span class="string">"url"</span>:<span class="string">"http://bbym010.iteye.com/blog/2100868"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">15</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"HAProxy简介 - 张善友 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/shanyou/archive/2012/10/16/2726768.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">16</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"什么是“极限编程”？"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/5153690.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=0&from=879a&uid=0&pu=usm@1,sz@1320_1004,ta@iphone_2_5.1_18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">17</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"MongoDB Sharding机制分析 » 囧囧孙"</span>,<span class="string">"url"</span>:<span class="string">"http://www.jiongsun.com/2013/03/45.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">18</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=1001560s/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.1_3_537/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF/t=iphone/l=3/tc?ref=www_iphone&lid=11304546645738609731&order=1&vit=osres&tj=www_normal_1_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D..._%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=10463&di=b0551844ad65db76&bdenc=1&tch=124.0.0.0.0.0&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=9ce1d147be1560001000000356d4d7f0&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">19</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"互联网系统可靠性基础：正确的异常处理"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/UvayInY"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">20</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"服务器端 概述 - 极光文档"</span>,<span class="string">"url"</span>:<span class="string">"http://docs.jpush.io/server/server_overview/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">21</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"从ACID到CAP到BASE - xixicat - SegmentFault"</span>,<span class="string">"url"</span>:<span class="string">"https://segmentfault.com/a/1190000004468442?url_type=39&object_type=webpage&pos=1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">22</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Netty系列之Netty可靠性分析"</span>,<span class="string">"url"</span>:<span class="string">"http://www.infoq.com/cn/articles/netty-reliability/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">23</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/feYfInn"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">24</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"有趣的bloom过滤器 | OurMySQL"</span>,<span class="string">"url"</span>:<span class="string">"http://ourmysql.com/archives/510?f=wb"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">25</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=27109467"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">26</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-string-ansi-unicode-bmp-utf.html?url_type=39&object_type=webpage&pos=1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">27</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"看了极光推送技术原理的几点思考"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8227469"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">28</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"腾讯微信技术架构"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8581852"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">29</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"鸟哥的 Linux 私房菜 -- 学习 bash shell"</span>,<span class="string">"url"</span>:<span class="string">"http://vbird.dic.ksu.edu.tw/linux_basic/0320bash.php"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">30</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎"</span>,<span class="string">"url"</span>:<span class="string">"https://www.zhihu.com/question/20962240"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">31</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"动态规划：从新手到专家"</span>,<span class="string">"url"</span>:<span class="string">"http://www.hawstein.com/posts/dp-novice-to-advanced.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">32</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"商品搜索引擎—推荐系统设计 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/19834.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">33</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn"</span>,<span class="string">"url"</span>:<span class="string">"http://coolshell.cn/articles/6790.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">34</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"32个算法"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309403979794248269609&jumpfrom=weibocom"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">35</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java对象之死-云栖社区"</span>,<span class="string">"url"</span>:<span class="string">"https://yq.aliyun.com/articles/53681?spm=5176.100239.bloglist.55.lOez9A"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">36</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"amao12580 · GitHub"</span>,<span class="string">"url"</span>:<span class="string">"https://github.com/amao12580/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">37</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"微服务与架构师"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA3MDMwOTcwMg==&mid=2650004550&idx=1&sn=b2299d1f2456541fc6b600a480c27dd8&scene=1&srcid=0714I4IHCwFfqnIGigCv3pJb#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">38</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"图灵社区 : 阅读 : 什么时候你不应该使用微服务"</span>,<span class="string">"url"</span>:<span class="string">"http://www.ituring.com.cn/article/217430"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">39</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"微服务的两种模式：应用中心和任务中心 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/m/article/1335"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">40</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7770?f=wb"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">41</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"一个资深架构师是这样理解数据结构的"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">42</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"高性能服务器架构思路 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">43</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java编程常见问题汇总 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">44</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7816?f=wb"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">45</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"微服务实战：从架构到部署 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1519"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">46</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"优酷蓝鲸近千节点的Redis集群运维经验总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">47</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"spring ioc DI 理解 - WhyWin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"https://zm6.sm-tc.cn/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">48</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1582"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">49</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"技改之路：从单块应用到微服务，我的血泪总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">50</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"华为内部如何实施微服务架构？基本就靠这5大原则"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">51</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"悲观的看：互联网技术岗位红利期逐渐到头"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309404020851660960391&jumpfrom=weibocom"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">52</span>}</div><div class="line">完成解析书签.</div><div class="line"></div><div class="line">Process finished with <span class="keyword">exit</span> code <span class="number">0</span></div></pre></td></tr></table></figure>
<p><img src="/img/via-broswer-bookmarks.jpg" alt=""></p>
<p>解析导出代码，已经整理完毕：<a href="https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java" target="_blank" rel="external">https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java</a></p>
<p>有心的小伙伴，完全可以按照这篇文章，写个lite app，帮助神族用户脱坑啦，哈哈哈！Android的可玩性也就体现出来啦！有时间我也做一个App自动化完成吧，maybe later！</p>
<h1 id="UA"><a href="#UA" class="headerlink" title="UA"></a>UA</h1><p>书签导入<a href="http://www.coolapk.com/apk/mark.via" target="_blank" rel="external">Via</a>后的事大家都懂得啦，第一要务就是改UA（User-Agent），分享几个我收集的几个UA。<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line">Mozilla/<span class="number">5.0</span> (BB10; Touch) AppleWebKit/<span class="number">537.10</span>+ (KHTML, like Gecko) Version/<span class="number">10.0</span><span class="number">.9</span><span class="number">.2372</span> Mobile Safari/<span class="number">537.10</span>+（BlackBerry Z30）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; U; Android <span class="number">4.3</span>; en-us; SM-N900T Build/JSS15J) AppleWebKit/<span class="number">534.30</span> (KHTML, like Gecko) Version/<span class="number">4.0</span> Mobile Safari/<span class="number">534.30</span>（Galaxy Note <span class="number">3</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Windows Phone <span class="number">10.0</span>; Android <span class="number">4.2</span><span class="number">.1</span>; Microsoft; Lumia <span class="number">950</span>) AppleWebKit/<span class="number">537.36</span> (KHTML, like Gecko) Chrome/<span class="number">46.0</span><span class="number">.2486</span><span class="number">.0</span> Mobile Safari/<span class="number">537.36</span> Edge/<span class="number">14.14263</span>（Microsoft Lumia <span class="number">950</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; U; en-us; KFAPWI Build/JDQ39) AppleWebKit/<span class="number">535.19</span> (KHTML, like Gecko) Silk/<span class="number">3.13</span> Safari/<span class="number">535.19</span> Silk-Accelerated=true（Kindle Fire HDX）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; Android <span class="number">4.3</span>; Nexus <span class="number">10</span> Build/JSS15Q) AppleWebKit/<span class="number">537.36</span> (KHTML, like Gecko) Chrome/<span class="number">48.0</span><span class="number">.2564</span><span class="number">.23</span> Safari/<span class="number">537.36</span>（Nexus <span class="number">10</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPad; CPU OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPad）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPad; CPU OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPad mini）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPhone; CPU iPhone OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPnone <span class="number">6</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPhone; CPU iPhone OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPnone <span class="number">6</span> plus）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; U; Android; unknown; zh-CN) AppleWebKit/<span class="number">534.30</span> (KHTML, like Gecko) Version/<span class="number">4.0</span> UCBrowser/<span class="number">10.10</span><span class="number">.8</span><span class="number">.822</span> U3/<span class="number">0.8</span><span class="number">.0</span> Mobile Safari/<span class="number">534.30</span>（UC）</div><div class="line"></div><div class="line">netdisk;<span class="number">7.0</span><span class="number">.0</span>（百度云手机版<span class="number">7.0</span><span class="number">.0</span>）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.2</span><span class="number">.0</span>（百度云手机版<span class="number">5.2</span><span class="number">.0</span>）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.2</span><span class="number">.7</span>;PC;PC-Windows;<span class="number">6.2</span><span class="number">.9200</span>;WindowsBaiduYunGuanJia（百度云电脑版<span class="number">5.2</span><span class="number">.7</span>）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.4</span><span class="number">.3</span>;PC;PC-Windows;<span class="number">6.2</span><span class="number">.9200</span>;WindowsBaiduYunGuanJia（百度云电脑版<span class="number">5.4</span><span class="number">.3</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; Android; unknown; zh-CN) AppleWebKit/<span class="number">537.36</span> (KHTML, like Gecko) Version/<span class="number">4.0</span> Chrome/<span class="number">53.0</span><span class="number">.2785</span><span class="number">.80</span> Mobile Safari/<span class="number">537.36</span> MicroMessenger/<span class="number">6.3</span><span class="number">.18</span><span class="number">.800</span> NetType/WIFI Language/zh_CN（微信标识）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.4</span><span class="number">.10</span>;PC;PC-Windows;<span class="number">6.2</span><span class="number">.9200</span>;WindowsBaiduYunGuanJia（百度云电脑版<span class="number">5.4</span><span class="number">.10</span>）</div></pre></td></tr></table></figure></p>
<h1 id="小结"><a href="#小结" class="headerlink" title="小结"></a>小结</h1><p>有编程基础就是好，啥事难不倒！爱折腾，爱android！</p>
]]></content>
<categories>
<category> Record </category>
</categories>
<tags>
<tag> Geek </tag>
<tag> SQLite </tag>
<tag> JSON </tag>
<tag> Java </tag>
<tag> Flyme </tag>
<tag> Bookmarks </tag>
</tags>
</entry>
<entry>
<title><![CDATA[Solr实战 - 生产环境的设计原则]]></title>
<url>https://amao12580.github.io/post/2016/09/solr-design-principles/</url>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>中午追番（大力推荐【<a href="https://movie.douban.com/subject/25774052/" target="_blank" rel="external">寄生兽</a>】）没睡觉，现在有点困，洗了把脸，思路恢复清晰了。这篇文章经过数天的腹稿终于要发出来，早上来公司草拟了提纲就先搁置了，没想着一口气写完，但下午想到blog是<a href="http://amao12580.github.io/post/2016/06/How-to-better-write-the-blog/">自动up to date的</a>，不能给个提纲就完事啊！读者的感受在哪里？太不负责了！！！</p>
<p>自从接盘离职同事的代码，一直在做搜索引擎(Solr)方面的工作，都忙成狗了，各个业务线都有搜索的需求，零零碎碎20多个接口吧！维护的过程，看着十多年经验的同事写的代码，有褒有贬，自己也学到了一些小技巧，最最最主要的，对搜索引擎不再陌生，在工作中应用、调优没有问题了。</p>
<p>代码维护是艰难的事，维护shit般的代码是想死的事！如果从头到尾按照统一的规范来构建搜索业务，也许没这么多后续痛苦，看着没有编码规范和零注释的代码，经过数位同事的修改，变得一片浑浊，想理清却已找不到人，只能自己加班看，梳理思路，同时还得整理重构方案！体会到团队的规范性在后期节省了多少维护成本，然而这个收益在前期是看不到或容易被忽视的的，并且在工期和人力的压迫下，是最容易被leader妥协的点！不加规范的代码和设计就像脱缰的野马一样，早期不仔细调教，后面再想驯服就难上加难，也就是技术债务吧！接盘是个苦差事，希望大家谨慎，努力不给后人留坑吧！</p>
<p><img src="/img/rough-and-tumble.jpg" alt=""></p>
<h1 id="Core设计原则"><a href="#Core设计原则" class="headerlink" title="Core设计原则"></a>Core设计原则</h1><h2 id="理解Core"><a href="#理解Core" class="headerlink" title="理解Core"></a>理解Core</h2><p>在Client操作Solr时，通过HTTP RESTful接口完成request-response，使用Core指明操作范围，Core还是RESTful中最后一级资源路径。</p>
<p>Client常用的操作有5种：index、delete、query、optimize，含义同字面意义（自描述）。还有不常用的操作：create、rename、unload、reload、swap、ping、commit、rollback、flush。其中index、delete、query、optimize、rename、unload、reload、swap需要指定Core才可以操作，delete和optimize、flush有些特殊，可以不指定Core，默认对所有Core执行。而create、rename、unload、reload、swap仅在Solr-Admin页面(http ://your solr ip:8983/)上提供，在SolrJ客户端(面向Java语言)无法直接使用，不过可以使用HTTPClient组装HTTP报文模拟操作！</p>
<p>何不将Core想象成MySQL中的Table？在MySQL中，对Table进行CRUD操作时，不是也要加上table name吗？create table “test”… 、alter table “test” … 、select * from “test”…、delete from “test”…大多数的sql需要指定表名，但也有例外：show process list、show variables like ‘character ‘、show innodb engine status;等等。除了在查询时，solr的Core与MySQL的表有相同的功能（事实上Solr的查询远比MySQL的查询强大）。Solr就像是个操作系统，安装在操作系统中的软件就是Core，每个Core有自身的配置文件及数据。在磁盘的角度看，Core是一个文件夹，比如我在线上有名为user和order的2个Core，在Solr的工作目录(%SolrHome%/server/solr)下，有类似的文件目录：</p>
<p><img src="/img/Solr-core-dir-desc.jpg" alt=""><br>可以看到Solr依靠文件夹管理索引文件和日志文件，我们将在后文中说明这些文件的作用和相对路径配置！</p>
<h2 id="设计Core"><a href="#设计Core" class="headerlink" title="设计Core"></a>设计Core</h2><p>既然Core与Table类似，那我们是不是按照传统在MySQL的思路，按照业务功能，拆分出符合三范式的Table呢？例如在MySQL中有user和order两张表。按照这种思路，我们也在Solr建立两个核心user、order，数据一一对应即可？</p>
<p>值得注意的是，Solr对多核心join提供非常有限的支持。</p>
<p>1.仅支持最多2个Core进行join query（formIndex=user toIndex=order）<br>2.仅支持对formIndex中的Core field进行filter query以及response，这是打折的inner join<br>3.两个core都在一个solr instance上</p>
<p>这些限制意味着什么呢？在一次query中，2个核心之间的交互度很低。为了更方便的讲清楚这个问题，以及考虑下文，我们假设现在的user和order核心的结构定义文件（schema.xml）的内容如下:</p>
<figure class="highlight axapta"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">user与<span class="keyword">order</span>是<span class="number">1</span>:N的关系，即：一个用户拥有多个订单，一个订单只从属一个用户</div></pre></td></tr></table></figure>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">user/conf/schema.xml</div><div class="line"></div><div class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"utf-8"</span><span class="meta">?></span></span></div><div class="line"></div><div class="line"><span class="tag"><<span class="name">schema</span> <span class="attr">name</span>=<span class="string">"user"</span> <span class="attr">version</span>=<span class="string">"1.5"</span>></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"_version_"</span> <span class="attr">type</span>=<span class="string">"long"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">uniqueKey</span>></span>Id<span class="tag"></<span class="name">uniqueKey</span>></span></div><div class="line"> <span class="comment"><!-- 用户唯一编号 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"Id"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=普通用户；1=VIP用户 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"type"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 昵称 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"nickname"</span> <span class="attr">type</span>=<span class="string">"simplChinese"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=失效；1=有效 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"status"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 账户最后一次活动时间 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"updateTime"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"string"</span> <span class="attr">class</span>=<span class="string">"solr.StrField"</span> <span class="attr">sortMissingLast</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"int"</span> <span class="attr">class</span>=<span class="string">"solr.TrieIntField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"long"</span> <span class="attr">class</span>=<span class="string">"solr.TrieLongField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"simplChinese"</span> <span class="attr">class</span>=<span class="string">"solr.TextField"</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"index"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"query"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"></<span class="name">fieldType</span>></span></div><div class="line"><span class="tag"></<span class="name">schema</span>></span></div></pre></td></tr></table></figure>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div></pre></td><td class="code"><pre><div class="line">order/conf/schema.xml</div><div class="line"></div><div class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"utf-8"</span><span class="meta">?></span></span></div><div class="line"></div><div class="line"><span class="tag"><<span class="name">schema</span> <span class="attr">name</span>=<span class="string">"order"</span> <span class="attr">version</span>=<span class="string">"1.5"</span>></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"_version_"</span> <span class="attr">type</span>=<span class="string">"long"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">uniqueKey</span>></span>Id<span class="tag"></<span class="name">uniqueKey</span>></span></div><div class="line"> <span class="comment"><!-- 订单唯一编号 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"Id"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=微信订单；1=唯品会订单 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"type"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 订单概览 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"title"</span> <span class="attr">type</span>=<span class="string">"simplChinese"</span>/></span></div><div class="line"> <span class="comment"><!-- 订单备注 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"remark"</span> <span class="attr">type</span>=<span class="string">"simplChinese"</span>/></span></div><div class="line"> <span class="comment"><!-- 下单用户Id --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"uId"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=创建；1=付款；2=付款超时 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"status"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 金额、以分为单位 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"amount"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 订单最后一次更新时间 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"updateTime"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"string"</span> <span class="attr">class</span>=<span class="string">"solr.StrField"</span> <span class="attr">sortMissingLast</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"int"</span> <span class="attr">class</span>=<span class="string">"solr.TrieIntField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"long"</span> <span class="attr">class</span>=<span class="string">"solr.TrieLongField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"simplChinese"</span> <span class="attr">class</span>=<span class="string">"solr.TextField"</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"index"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"query"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"></<span class="name">fieldType</span>></span></div><div class="line"><span class="tag"></<span class="name">schema</span>></span></div></pre></td></tr></table></figure>
<h2 id="超级Core"><a href="#超级Core" class="headerlink" title="超级Core"></a>超级Core</h2><p>假设有需求功能：”查询有已付款订单的用户昵称和用户Id”，类似的SQL：SELECT u.Id,u.nickname FROM user u INNER JOIN order o ON u.Id=o.uId WHERE o.status=1;<a href="/post/2016/09/solr-design-principles/#研究Join-query">这样简单的查询需求是可以完成的</a>，但对于复杂需求，例如”查询在今天活跃过的用户和这些用户在今天的最新一笔有效微信订单，要求订单金额不能低于5元，需要打印用户和订单基本信息”，这种就无法满足了！如果必须要复杂join query，常规思路是加新Core，如userAndOrder，主体信息为user和order，因为1:N的关系，需要新加自定义联合主键Id，在index阶段，以用户和订单两个角度来维护数据。在单个用户角度，删除该用户的所有数据（deleteByQuery(“uId:1000”)），重新查询订单信息，index多条。在单个订单角度，删除一条数据（deleteByQuery(“orderId:5000”)），查询所属用户，index一条。为了减少冗余，原有的user Core和order Core都可以不保留。方案的不足之处在于数据冗余，这将造成超级Core现象。比如还有一个product Core（商品信息），order与product是N:M的关系，按这种思路，难道还是聚合吗？考虑一种极限情况：假设user数据有十万，order数据有一千万，product数据有一亿，平均1个用户有100个订单，每个订单有50个商品，这样3表聚合userAndOrderAndProduct Core数据量在50亿左右，平均每个user的信息被重复了50万次！冗余带来的去重问题同样严重，result set进行distinct、groupBy、facet的代价将会极其高昂！</p>
<p>由于user、order、product的关系可以视为单方向，即order一定从属于user，product从属于order。改进的做法是将user的所有order视为一个<a href="https://cwiki.apache.org/confluence/dosearchsite.action?where=solr&spaceSearch=true&queryString=child+document" target="_blank" rel="external">child document set</a>处理，以此类推，有3层递进关系，此时的问题是product会存在冗余问题，相较于full-mix方案冗余度降低！</p>
<h2 id="定位Solr"><a href="#定位Solr" class="headerlink" title="定位Solr"></a>定位Solr</h2><p>产生超级Core问题是错误地定位Solr，Solr在信息检索领域是高效的，如多维度检索、关键词建议、关键词高亮、关键词补全、关键词纠错、更多相关、同义词、近义词、多义词、停用词、保护词、语义转换、空间搜索等等，但在复杂关系型查询以及强事务处理方面不如传统数据库，如MySQL！不应该将复杂join query交给solr来处理，这也侧面验证专业的工具解决特定领域的问题。除了在关键词方面的处理能力，Solr还对分维度权重、部分匹配、相关度等高度复杂需求可以轻松应对，但是搜索领域目前还存在的难题，Solr也无法解决，需要凭借外部辅助系统，例如：自动发现新词、识别流行词、歧义消除、搜索预测等等。目前都是通过NLP辅以海量语料库预处理，结合HDFS等平台海量计算发现。</p>
<p>我们在设计Core时，应理清需求，思考方案与Solr的契合度，保持KISS原则，尽可能的避免超级 Core。</p>
<h1 id="字段设计原则"><a href="#字段设计原则" class="headerlink" title="字段设计原则"></a>字段设计原则</h1><p>字段的含义与MySQL具备类似性，如：name、type、requireed、default。</p>
<p>Solr为了自身的索引高效与内存控制，还加上了很多属性：</p>
<ul>
<li><p>required：是否为必须字段，默认为false，是否必需，对应MySQL NOT NULL</p>
</li>
<li><p>default：index时字段未填，使用这个默认值，常见的应用如：sequence number</p>
</li>
<li><p>multiValued：是否为多值，字段可以是List，但泛型收主type控制，多个值只能是同一种类型，默认值是false</p>
</li>
<li><p>indexed：是否索引，参与到query计算的字段必需设置为true，默认值是true</p>
</li>
<li><p>stored：是否保留原始数据，如果字段需要在query时使用原始值，则需要设置为true，默认值是true</p>
</li>
<li><p>docValues：<a href="https://cwiki.apache.org/confluence/display/solr/DocValues" target="_blank" rel="external">针对大数据量复杂计算加速</a>，使用document-to-value数据结构，例如：需要聚合的字段，包括sort，agg，group，facet等、需要提供函数查询的字段、需要高亮的字段、自定义评分的字段。</p>
</li>
</ul>
<p>这些是常用属性，更多的信息参阅官方wiki：<a href="https://cwiki.apache.org/confluence/display/solr/Defining+Fields" target="_blank" rel="external">https://cwiki.apache.org/confluence/display/solr/Defining+Fields</a></p>
<h2 id="字段的分类"><a href="#字段的分类" class="headerlink" title="字段的分类"></a>字段的分类</h2><p>1.按照结构分类：field、copyField、dynamicField<br><figure class="highlight vim"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">copyField</div><div class="line"></div><div class="line">需要将一个或多个字段的数据汇总到多个字段时用到。典型的作为默认搜索时设计。新闻类<span class="built_in">index</span>应用中，client不指定qf参数时，默认从title、author、catalog、content字段中综合检索。<span class="keyword">source</span>和destination都支持通配符，maxChars，<span class="keyword">int</span>类型参数，用于配置复制字符数的上限。注意：如果dest由多个<span class="keyword">source</span>构成，就需要将其指定为multiValued。</div><div class="line"><copyField <span class="keyword">source</span>=<span class="string">"*t"</span> dest=<span class="string">"content"</span> maxChars=<span class="string">"30000"</span> /></div><div class="line"></div><div class="line"></div><div class="line">dynamicField</div><div class="line"></div><div class="line">动态字段（Dynamic fields）允许 solr 索引没有在 schema 中明确定义的字段。这个在忘记定义一些字段时很有用。动态字段可以让系统更灵活，通用性更强。它主要用作扩展备用，因为添加Field需要重建索引，而这可能是一个漫长的过程（而且没法使用增量更新索引），建议设计之初，多备一些dynamicField。</div><div class="line"></div><div class="line">Solr在Field <span class="keyword">list</span>中没法<span class="keyword">match</span>到query指定的field时，会尝试从dynamicField <span class="keyword">list</span>中<span class="keyword">match</span>，如果没有配置dynamicField则跳过。</div><div class="line"><dynamicField name=<span class="string">"*_i"</span> <span class="built_in">type</span>=<span class="string">"sint"</span> indexed=<span class="string">"true"</span> stored=<span class="string">"true"</span>/></div></pre></td></tr></table></figure></p>
<p>2.按照计算进行分类：keyword、filter、sort、function、join query<br><figure class="highlight stata"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line">1. keyword</div><div class="line">查询字段</div><div class="line"></div><div class="line"><span class="comment">//标题、副标题、区域、商圈、号码、其他号码</span></div><div class="line"><span class="keyword">query</span>.<span class="keyword">set</span>(<span class="string">"qf"</span>, <span class="string">"title"</span>, <span class="string">"slogan"</span>, <span class="string">"expectDistrictName"</span>, <span class="string">"expectTradeAreaName"</span>, <span class="string">"mobile"</span>, <span class="string">"otherContact"</span>);</div><div class="line"></div><div class="line">2. filter</div><div class="line">过滤字段</div><div class="line"></div><div class="line"><span class="keyword">query</span>.addFilterQuery(<span class="string">"maxRent:[* TO 100]"</span>);</div><div class="line"></div><div class="line">3. <span class="keyword">sort</span></div><div class="line">排序字段</div><div class="line"></div><div class="line"><span class="keyword">List</span><SolrQuery.SortClause> sorts = new ArrayList<>();</div><div class="line">sorts.add(SolrQuery.SortClause.<span class="keyword">desc</span>(<span class="string">"hasCharged"</span>));</div><div class="line">sorts.add(SolrQuery.SortClause.<span class="keyword">desc</span>(<span class="string">"updateTime"</span>));</div><div class="line"><span class="keyword">query</span>.setSorts(sorts);</div><div class="line"></div><div class="line">4. function & boost function</div><div class="line">函数、评分、权重</div><div class="line"></div><div class="line"><span class="keyword">query</span>.addSort(<span class="string">"if(isCertificated,1,0)"</span>, SolrQuery.<span class="keyword">ORDER</span>.<span class="keyword">desc</span>);</div><div class="line"><span class="keyword">query</span>.addSort(<span class="string">"hasSalesinProduct"</span>, SolrQuery.<span class="keyword">ORDER</span>.<span class="keyword">desc</span>);</div><div class="line"><span class="keyword">query</span>.addSort(<span class="string">"map(geodist(),0,5,5,map(geodist(),5.00000000001,20,4,map(geodist(),20.000001,50,3,map(geodist(),50.00000000001,100,2,map(geodist(),100.00000000001,100000,1,0)))))"</span>, SolrQuery.<span class="keyword">ORDER</span>.<span class="keyword">desc</span>);</div><div class="line"></div><div class="line"></div><div class="line">5. join <span class="keyword">query</span></div><div class="line">Core 联接查询，field用于联接条件</div><div class="line"></div><div class="line">{!join from=uId to=Id fromIndex=<span class="keyword">order</span> toIndex=user}status:1</div></pre></td></tr></table></figure></p>
<h2 id="字段的命名规范"><a href="#字段的命名规范" class="headerlink" title="字段的命名规范"></a>字段的命名规范</h2><p>一般来说命名规范的目的是：前期降低开发成本，后期减少维护成本。在前期可以省去（在某些场景下甚至不用）编写各式说明文档，keep focus on业务开发、技术攻坚；在后期在维护工作转手时，降低交接成本、理解成本。</p>
<p>与程序开发规范一致，在solr中，我们只需要对命名保持标准统一和易于理解，能做到简短有力就更好了。</p>
<ul>
<li><p>可读性<br>避开毫无意义的命名：i,j</p>
</li>
<li><p>无二义性<br>避免歧义，如：缩写往往会产生多义的问题，一个缩写词可能在不同人理解时，引发语义上不一致(<a href="https://zh.wikipedia.org/wiki/%E9%9F%93%E5%9C%8B%E9%AB%98%E9%80%9F%E9%90%B5%E9%81%93#.E8.BB.BC.E4.BA.8B" target="_blank" rel="external">KTX 09年建设事故</a>)。</p>
</li>
<li><p>统一<br>统一大小写，混合时推荐使用：小驼峰式。统一特殊字符使用规范，如慎用”*”（与dynamicField冲突），保持高效。</p>
</li>
<li><p>简洁性<br>在能说明用途的前提下，保持尽可能短。</p>
</li>
<li><p>使用常用词<br>降低理解成本（思维惯性）。这也是约定大于配置的一种表现吗？</p>
</li>
</ul>
<h2 id="字段的存储结构"><a href="#字段的存储结构" class="headerlink" title="字段的存储结构"></a>字段的存储结构</h2><p>Index document时，solr首先对field(取决于analyzer配置)进行分词，创建index库和document库。所谓的分词是指：将字符文本按照一定的规则分成若干个单词。</p>
<h3 id="Index库"><a href="#Index库" class="headerlink" title="Index库"></a>Index库</h3><p>lucene的倒排索引存储结构为：词项的字符串+词项的文档频率+记录词项的频率信息+记录词项的位置信息+跳跃偏移量。</p>
<h3 id="Document库"><a href="#Document库" class="headerlink" title="Document库"></a>Document库</h3><p>lucene词典中词的顺序是按照英文字母的顺序排列的，这样就可以采用压缩存储：假设有term，termagancy，termagant，termina四个词。每个字母需要1byte的空间，常规存储一共需要35byte。而压缩存储之后为：”term4agancy8t4inal”，一共需要22byte。通过DRY原则，节省大量的空间，这一点与<a href="https://github.com/amao12580/algorithm/blob/master/src/main/java/tree/huffman/HuffmanTree.java" target="_blank" rel="external">Huffman编码</a>思想一致。</p>
<h2 id="Additional-field"><a href="#Additional-field" class="headerlink" title="Additional field"></a>Additional field</h2><p>在实际使用中，为了满足QPS要求，将不参与计算的字段进行数据冗余，以便更快速的响应接口。实际上就是cache一部分信息在solr，避免发起RPC从外部源获取，减少进程上下文切换和网络往返时间。</p>
<p>典型场景是：图片审核，按照用户是否有头像过滤，response头像URL。常见的设计，在user Core中，有字段：id（int型），用户编号、hasHeadPhoto（boolean型），标识用户是否有头像。另一字段headPhotoURL(String型)，头像URI。此时headPhotoURL就是我们说的Additional field。在cache等基础设施不完善时，常这样处理。随着业务的复杂，Additional field不加以限制，会使用的越来越广泛，以至于后续的schema.xml中，参与计算的field反而占少数了。冗余URL这种可预估长度并且低频率变化的field还好理解，我还见过将用户个人说明（simplChinese：Text）进行冗余的，这完全不能接受了，拖累了所有涉及document的RT以及index rebuild。建议在前期基础设施不完善时，可以适量的加入Additional field（技术债务），但请务必在后续完善后，留出重构时间fix，并安排技术沟通会，讨论这种妥协方案的改进办法。</p>
<h2 id="Mix-core"><a href="#Mix-core" class="headerlink" title="Mix core"></a>Mix core</h2><p>单个Core为了兼容多个业务场景使用，将不同类别的业务数据混合塞入。不同的业务场景之间同时存在重叠和交叉的情况（不存在业务重叠是不需要放到一起的）。为了达到数据兼容，程序在index和query两个阶段都需要进行compatible。这样的Core我们称之为Mix core。</p>
<ul>
<li><p>field级别，有字段只对特定业务才会用到（垂直冗余）<br>对不同类型的Client提供不同级别的Query</p>
</li>
<li><p>row级别，有些数据记录对特定业务才会用到（水平冗余）<br>数据聚合类应用中，对所有结果进行计算：top N、sort、groupBy</p>
</li>
</ul>
<p>由于Solr高效检索的前提是将index文件load到内存进行计算，为了说明row数据量以及field分布与index大小的关系（间接分析memory以及disk的占用关系），统计了一份仿真环境的数据分布，供参考：</p>
<figure class="highlight lsl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">一些说明：</div><div class="line"><span class="number">1.</span>未计算dynamicField分布，实际项目中仅用作版本过渡，没有长期使用。</div><div class="line"><span class="number">2.</span>index大小和tlog大小是指相应dir所有文件大小之和。</div><div class="line"><span class="number">3.</span>不存在未commit的数据。</div><div class="line"><span class="number">4.</span><span class="string">"field分布"</span>，这一列中，<span class="string">"6：TrieIntField；"</span>，表示存在<span class="number">6</span>个solr.TrieIntField类型的字段。</div><div class="line"><span class="number">5.</span>仅一个solr节点（singleton node模式），不启用cloud，也意味着没有replica。</div><div class="line"><span class="number">6.</span>所有类型的field的length均不超过<span class="number">256</span>位，多数在<span class="number">11</span>位以内。</div></pre></td></tr></table></figure>
<p><img src="/img/Solr-core-field-analysis.jpg" alt="仿真环境的数据分布"></p>
<p>从中不难发现：最为常用的是整形，1个普通row约占用1 KB的空间，由于需要在数据量增长时，不线性降低solr查询性能。我们需要保持：混合核心重叠部分的field和row处于大比例，仅仅将极少数用作业务兼容！</p>
<h1 id="SolrIndexSearcher"><a href="#SolrIndexSearcher" class="headerlink" title="SolrIndexSearcher"></a>SolrIndexSearcher</h1><p>Solr查询的核心类就是SolrIndexSearcher，在同一时刻只由当前的SolrIndexSearcher供上层的handler使用（当切换SolrIndexSearcher时可能会有两个同时提供服务），而各种Cache是依附于SolrIndexSearcher的，SolrIndexSearcher在则Cache生，SolrIndexSearcher亡则被clear（solrIndexSearcher.close()）。<br>在solr4.0之后，有2种commit，hard commit、soft commit。Hard commit操作会触发SolrIndexSearcher切换，而close操作隐含flush AND commit，也会触发。</p>
<p>AutoCommit，Commit操作也可以按条件自动完成（hard commit），maxDocs：按未commit的最大文档数量，event watch机制；maxTime：按上一次未commit距离现在的时间，定时器。同样提供API，手工commit，这样性能比较差，平均每次commit 600ms；<br>AutoCommit 参考：<a href="http://wiki.apache.org/solr/SolrConfigXml" target="_blank" rel="external">http://wiki.apache.org/solr/SolrConfigXml</a>，solrconfig.xml配置：<br><figure class="highlight dts"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="params"><updateHandler class="solr.DirectUpdateHandler2"></span></div><div class="line"> <span class="params"><updateLog></span></div><div class="line"> <span class="params"><str name="dir"></span>${solr.ulog.dir:}<span class="params"></str></span></div><div class="line"> <span class="params"><int name="numVersionBuckets"></span>${solr.ulog.numVersionBuckets:<span class="number">65536</span>}<span class="params"></int></span></div><div class="line"> <span class="params"></updateLog></span></div><div class="line"> <span class="params"><autoCommit></span></div><div class="line"> <span class="params"><maxTime></span>${solr.autoCommit.maxTime:<span class="number">15000</span>}<span class="params"></maxTime></span></div><div class="line"> <span class="params"><openSearcher></span>false<span class="params"></openSearcher></span></div><div class="line"> <span class="params"></autoCommit></span></div><div class="line"> <span class="params"><autoSoftCommit></span></div><div class="line"> <span class="params"><maxTime></span>${solr.autoSoftCommit.maxTime:<span class="number">1000</span>}<span class="params"></maxTime></span></div><div class="line"> <span class="params"></autoSoftCommit></span></div><div class="line"></div><div class="line"> <span class="params"><!--</span></div><div class="line"><span class="params"> <listener event="postCommit" class="solr.RunExecutableListener"></span></div><div class="line"> <span class="params"><str name="exe"></span>solr<span class="meta-keyword">/bin/</span>snapshooter<span class="params"></str></span></div><div class="line"> <span class="params"><str name="dir"></span>.<span class="params"></str></span></div><div class="line"> <span class="params"><bool name="wait"></span>true<span class="params"></bool></span></div><div class="line"> <span class="params"><arr name="args"></span> <span class="params"><str></span>arg1<span class="params"></str></span> <span class="params"><str></span>arg2<span class="params"></str></span> <span class="params"></arr></span></div><div class="line"> <span class="params"><arr name="env"></span> <span class="params"><str></span>MYVAR=val1<span class="params"></str></span> <span class="params"></arr></span></div><div class="line"> <span class="params"></listener></span></div><div class="line"> --></div><div class="line"><span class="params"></updateHandler></span></div></pre></td></tr></table></figure></p>
<p>Hard commit时，除了向Directory对象（实际是Disk proxy）提交索引变化(new tlog)，SolrIndexSearcher需要重新建立。commit提交后，index文件flush到硬盘（flush：从内存刷回磁盘保存：fsync），并触发listener，建立new SolrIndexSearcher(新的insexReader，从硬盘中load index)，这样后续的Query使用new SolrIndexSearcher。建议不要频繁修改document，特别是要避免大批量reload，影响RT稳定和服务中断。</p>
<p>Hard commit时，如果有配置auto-warm（autowarmCount=？），则会对new SolrIndexSearcher swap一定数量的old SolrIndexSearcher index文件；而soft commit是在NRT（Near Real Time）实时搜索中提出的(如log、analysis)，不会flush到disk，也可以使得document被搜索到，代价比hard commit要小的多；对实时性要求比较高的场景下，可以做soft commit操作，不过还是要定时hard commit，确保索引持久化到disk。遭遇突发性故障时（机器断电），未flush到disk的数据可能会丢失，在updatelog开启时（updatelog是Solr的概念，在Lucene并没有出现）,solr启动时会进行recover check，尽可能的恢复数据。</p>
<h1 id="Cache机制"><a href="#Cache机制" class="headerlink" title="Cache机制"></a>Cache机制</h1><p>如果把Schema定义为Solr的Model的话，那么Solrconfig.xml就是Solr的Configuration，它定义Solr如果处理索引、高亮、搜索等很多请求，同时还配置cache策略。</p>
<p>solrconfig.xml<br><figure class="highlight dust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div></pre></td><td class="code"><pre><div class="line"><span class="xml">1.指定索引文件的存储路径</span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">dataDir</span>></span>$</span><span class="template-variable">{solr.data.dir:./solr/data}</span><span class="xml"><span class="tag"></<span class="name">dataDir</span>></span></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">2.缓存配置</span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">Solr在Lucene之上开发了很多Cache功能，目前提供的Cache类型有：</span></div><div class="line"><span class="xml"><span class="comment"><!-- 1.filterCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">filterCache</span> <span class="attr">class</span>=<span class="string">"solr.FastLRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> initialSize="512"</span></div><div class="line"><span class="xml"> autowarmCount="0"/></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 2.queryResultCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">queryResultCache</span> <span class="attr">class</span>=<span class="string">"solr.LRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> initialSize="512"</span></div><div class="line"><span class="xml"> autowarmCount="0"/></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 3.documentCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">documentCache</span> <span class="attr">class</span>=<span class="string">"solr.LRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> initialSize="512"</span></div><div class="line"><span class="xml"> autowarmCount="0"/></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 4.自定义缓存策略针对数据块交换 --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">cache</span> <span class="attr">name</span>=<span class="string">"perSegFilter"</span></span></span></div><div class="line"><span class="xml"> class="solr.search.LRUCache"</span></div><div class="line"><span class="xml"> size="10"</span></div><div class="line"><span class="xml"> initialSize="0"</span></div><div class="line"><span class="xml"> autowarmCount="10"</span></div><div class="line"><span class="xml"> regenerator="solr.NoOpRegenerator" /></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 5.fieldValueCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">fieldValueCache</span> <span class="attr">class</span>=<span class="string">"solr.FastLRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> autowarmCount="128"</span></div><div class="line"><span class="xml"> showItems="32" /></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">// 是否能使用到filtercache关键配置</span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">useFilterForSortedQuery</span>></span>true<span class="tag"></<span class="name">useFilterForSortedQuery</span>></span></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">// queryresult的结果集控制</span></div><div class="line"><span class="xml"> <span class="tag"><<span class="name">queryResultWindowSize</span>></span>20<span class="tag"></<span class="name">queryResultWindowSize</span>></span></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">// 是否启用懒加载field</span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">enableLazyFieldLoading</span>></span>true<span class="tag"></<span class="name">enableLazyFieldLoading</span>></span></span></div></pre></td></tr></table></figure></p>
<h2 id="LRU-VS-FastLRU"><a href="#LRU-VS-FastLRU" class="headerlink" title="LRU VS FastLRU"></a>LRU VS FastLRU</h2><p>Solr提供了两种SolrCache接口实现类：solr.search.LRUCache和solr.search.FastLRUCache。FastLRUCache是1.4版本中引入的，其速度在普遍意义上要比LRUCache更快些。</p>
<p>LRUCache可配置参数：<br>1）size：cache中可保存的最大的项数，默认是1024。<br>2）initialSize：cache初始化时的大小，默认是1024。</p>
<p>3）autowarmCount：当系统启动时（firstSearcher）或切换SolrIndexSearcher时，可以对新生成的SolrIndexSearcher做autowarm（预热）处理。autowarmCount表示从旧的SolrIndexSearcher中取多少项来在新的SolrIndexSearcher中被重新生成，如何重新生成由CacheRegenerator实现。</p>
<p>查看Solr源码可以发现，在实现上，LRUCache直接使用LinkedHashMap来缓存数据，由initialSize来限定cache的大小，淘汰策略也是使用LinkedHashMap的内置的LRU方式，读写操作都是对map的全局锁，所以并发性效果方面稍差。</p>
<p>filterCache和fieldValueCache使用FastLRUCache实现，FastLRUCache内部使用了ConcurrentLRUCache来缓存数据，它是个加了LRU淘汰策略的ConcurrentHashMap，所以其并发性要好很多，这也是多数Java版Cache的极典型实现。</p>
<h2 id="filterCache"><a href="#filterCache" class="headerlink" title="filterCache"></a>filterCache</h2><p>filterCache中存储了无序的lucene document Id集合，即FilterCache存储了一些无序的文档id，这些Id并不是我们在schema.xml里配置的unique key，而是solr内部的一个文档标识。filterCache存储了filter queries(“fq”参数)得到的document Id集合结果。Solr中的query参数有两种，即q和fq。如果fq存在，Solr是先查询fq（因为fq可以多个，所以多个fq查询是个取结果交集的过程（Map-reduce模型）)，之后将fq结果和q结果取并。在这一过程中，filterCache就是key为单个fq（类型为Query），value为document Id集合（类型为DocSet）的cache。从后面的分析你将会看到对于fq为range query来说，filterCache将表现出其更有价值的一面。</p>
<h2 id="queryResultCache"><a href="#queryResultCache" class="headerlink" title="queryResultCache"></a>queryResultCache</h2><p>顾名思义，queryResultCache是对查询结果的缓存（SolrIndexSearcher中的cache缓存的都是document Id set），这个结果就是针对查询条件的完全有序的结果。因为查询参数是有start和rows的，所以某个QueryResultKey可能命中了cache，但start和rows却不在cache的document Id set范围内。当然，document Id set是越大命中的概率越大，但这也会很浪费内存，这就需要个参数：queryResultWindowSize来指定document Id set的大小。</p>
<h2 id="documentCache"><a href="#documentCache" class="headerlink" title="documentCache"></a>documentCache</h2><p>documentCache用来保存”doc_Id,document”键值对(正排索引)。如果使用documentCache，就尽可能开大些，至少要大过max_results * max_concurrent_queries，否则因为cache的淘汰，一次请求期间还需要重新获取document一次。也要注意document中存储的字段的多少，避免大量的内存消耗。</p>
<h1 id="数据维护原则"><a href="#数据维护原则" class="headerlink" title="数据维护原则"></a>数据维护原则</h1><p>在类ELK准实时日志分析方案中，各节点的logs被聚合到Solr中存储，在内存积累一定批次后，触发hard commit持久化到disk，在disk积累到一定量后进行转移（或直接删除），不涉及到update业务。而在类LBS（Location-Based Service）应用中，经纬度和用户信息被定时上传到database存储，然后sync到Solr，document可能被频繁update。提交document到Solr索引后，那修改documents的策略是什么呢？最简单的，Solr提供full-replace的方式进行更新，如果配置为覆盖旧版本在，则先按照uniqueKey找到old document并标记为删除，让后add new document，这主要是lucene是按照“delete-add”的模式来维护数据。另外，Solr支持document按field局部更新。</p>
<h2 id="Atomic-Updates"><a href="#Atomic-Updates" class="headerlink" title="Atomic Updates"></a>Atomic Updates</h2><p>局部更新允许修改一个或多个field，而不必重新索引整个document。在multiValued类型的field中，它甚至允许按正则表达式局部更新value，以便加速Solr对index的索引处理，减轻QPS波动。</p>
<p><img src="/img/solr-atomic-updates.jpg" alt=""></p>
<h2 id="Optimistic-Concurrency-Control"><a href="#Optimistic-Concurrency-Control" class="headerlink" title="Optimistic Concurrency Control"></a>Optimistic Concurrency Control</h2><p><a href="https://www.google.com.hk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0ahUKEwi3qqHH3qzPAhUJs48KHYRaBPQQFggpMAE&url=https%3A%2F%2Fzh.wikipedia.org%2Fzh-cn%2F%25E4%25B9%2590%25E8%25A7%2582%25E5%25B9%25B6%25E5%258F%2591%25E6%258E%25A7%25E5%2588%25B6&usg=AFQjCNHJ4Y6Fpckt8rQxpRc5NSC_CHQtIA" target="_blank" rel="external">乐观并发控制(OCC)，或称乐观锁</a>。它是一种多数NoSQL数据库具备的特性，它允许client基于version number按条件并发update same document，为了防止ABA问题，Solr使用字段”_version_“来控制document版本，为了可比较，被设置为数值型，默认情况下，这个字段被定义在schema.xml。client在提交document时，不需要提供_version_的值，它在document每次发生变化时，被Solr自动修改，修改的值全局唯一。</p>
<figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><field <span class="attribute">name</span>=<span class="string">"_version_"</span> <span class="attribute">type</span>=<span class="string">"long"</span>/></div></pre></td></tr></table></figure>
<p>在默认配置下，”_version_“配置为”indexed=true”，对于某些操作系统来说，海量的document，FieldCache增加需要消耗太多内存。由于该字段不参与计算，可以关闭index，将”_version_“定义为DocValues。<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><field <span class="attribute">name</span>=<span class="string">"_version_"</span> <span class="attribute">type</span>=<span class="string">"long"</span> <span class="attribute">indexed</span>=<span class="string">"false"</span> <span class="attribute">stored</span>=<span class="string">"true"</span> <span class="attribute">required</span>=<span class="string">"true"</span> <span class="attribute">docValues</span>=<span class="string">"true"</span>/></div></pre></td></tr></table></figure></p>
<h2 id="Field-Storage"><a href="#Field-Storage" class="headerlink" title="Field Storage"></a>Field Storage</h2><p>如果field想要使用atomic update功能，在schema.xml中，除了copyField的destinations需要配置为“stored=’false’”之外，目标field必需配置为“stored=’true’”。在copyField中，Solr会index所有的source fields最新值。如果copyField的destinations配置为“stored=’true’”，如果同时修改source fields和destination field的值，那destination field就会看起来像被source fields修改的值覆盖一样。</p>
<h2 id="小结"><a href="#小结" class="headerlink" title="小结"></a>小结</h2><p>document被修改后相应的documentCache、fieldCache被clear，被删除后filterCache、queryResultCache、fieldCache需要update。应该谨慎对待update操作，尽可能少的发起hard commit操作，减少solrIndexSearcher rebuild。对少量field更新采取atomic update的方式进行，对于大量field更新采取full-replace策略。</p>
<h1 id="索引重建原则"><a href="#索引重建原则" class="headerlink" title="索引重建原则"></a>索引重建原则</h1><p>减少服务中断时间</p>
<p>C端流量控制</p>
<h2 id="单个Solr集群实例"><a href="#单个Solr集群实例" class="headerlink" title="单个Solr集群实例"></a>单个Solr集群实例</h2><p>Core要有版本号<br>opportunity_V1</p>
<p>1.先创建新Core<br>opportunity_V2</p>
<p>2.额外进程构建V2</p>
<p>3.检查V2构建成功</p>
<p>4.流量切换到V2 或删除(unload)V1后 v2-rename-v1</p>
<p>V2 swap V1</p>
<p>5.确认V1Core不再需要后再销毁<br>同时运行多版本时，谨慎</p>
<h2 id="多个Solr集群"><a href="#多个Solr集群" class="headerlink" title="多个Solr集群"></a>多个Solr集群</h2><p>先构建好新集群</p>
<p>流量切换到新集群</p>
<p>旧集群停机</p>
<h1 id="使用总结"><a href="#使用总结" class="headerlink" title="使用总结"></a>使用总结</h1><h2 id="命令行启动"><a href="#命令行启动" class="headerlink" title="命令行启动"></a>命令行启动</h2><figure class="highlight taggerscript"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">F:<span class="symbol">\s</span>olr<span class="symbol">\b</span>in>solr start -m 1g -f</div></pre></td></tr></table></figure>
<h2 id="清空index"><a href="#清空index" class="headerlink" title="清空index"></a>清空index</h2><figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">curl http://<span class="tag"><<span class="name">host</span>></span>:<span class="tag"><<span class="name">port</span>></span>/<span class="tag"><<span class="name">solr_base</span>></span>/update?commit=true -d '<span class="tag"><<span class="name">delete</span>></span><span class="tag"><<span class="name">query</span>></span>*:*<span class="tag"></<span class="name">query</span>></span><span class="tag"></<span class="name">delete</span>></span>'</div></pre></td></tr></table></figure>
<h2 id="Core-别名"><a href="#Core-别名" class="headerlink" title="Core 别名"></a>Core 别名</h2><p>在文件/core.properties中，定义name字段，既可作为别名。同时还可以设置使用的solrconfig.xml路径，以达到多Core共用同一份solrconfig.xml的目的，schema.xml也是类似的设置。</p>
<p>dataDir可以配置Core数据目录，以达到程序与数据分离的效果，方便备份和容灾。<br><figure class="highlight ini"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="comment">#我的Core文件夹名是：order，设置别名为myOrder。在client中就可以使用myOrder来调用我的Core</span></div><div class="line"><span class="attr">name</span>=myOrder</div><div class="line"><span class="attr">config</span>=solrconfig.xml</div><div class="line"><span class="attr">schema</span>=schema.xml</div><div class="line"><span class="attr">dataDir</span>=data</div></pre></td></tr></table></figure></p>
<h2 id="查询技巧"><a href="#查询技巧" class="headerlink" title="查询技巧"></a>查询技巧</h2><p>1.只查询字段orderStatus值在1到3的数据<br>方案1.orderStatus可以是int、string类型<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="attribute">q</span>=*:*&fq={!frange <span class="attribute">l</span>=1 <span class="attribute">u</span>=3}orderStatus</div></pre></td></tr></table></figure></p>
<p>方案2.orderStatus确定是整数<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">q=*<span class="symbol">:*&fq=orderStatus</span><span class="symbol">:</span>[<span class="number">1</span> TO <span class="number">3</span>]</div></pre></td></tr></table></figure></p>
<h1 id="研究Join-query"><a href="#研究Join-query" class="headerlink" title="研究Join query"></a>研究Join query</h1><p>附上研究Join query的实战代码<br><figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div></pre></td><td class="code"><pre><div class="line"><span class="comment">//测试用的Solr环境为：</span></div><div class="line"><span class="comment">// solr 6.1</span></div><div class="line"><span class="comment">// jetty容器</span></div><div class="line"><span class="comment">//启动参数：solr start -m 1g -f</span></div><div class="line"><span class="comment">//windows 7 X64</span></div><div class="line"><span class="comment">//solrj maven: <groupId>org.apache.solr</groupId> <artifactId>solr-solrj</artifactId> <version>5.3.0</version></span></div><div class="line"><span class="comment">//<luceneMatchVersion>6.1.0</luceneMatchVersion></span></div><div class="line"></div><div class="line"><span class="keyword">String</span> solrHostName = <span class="string">"http://127.0.0.1:8983/solr/"</span>;</div><div class="line">SolrClient userClient = <span class="keyword">new</span> HttpSolrClient(solrHostName + <span class="string">"user"</span>);</div><div class="line">SolrClient orderClient = <span class="keyword">new</span> HttpSolrClient(solrHostName + <span class="string">"order"</span>);</div><div class="line"></div><div class="line">List<SolrInputDocument> userDocuments = <span class="keyword">new</span> ArrayList<>();</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"添加用户“张三”"</span>);</div><div class="line">SolrInputDocument userDocument = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">userDocument.addField(<span class="string">"Id"</span>, <span class="number">1000</span>);<span class="comment">//用户唯一编号</span></div><div class="line">userDocument.addField(<span class="string">"type"</span>, <span class="number">0</span>);<span class="comment">//0=普通用户；1=VIP用户</span></div><div class="line">userDocument.addField(<span class="string">"nickname"</span>, <span class="string">"张三"</span>);<span class="comment">//昵称</span></div><div class="line">userDocument.addField(<span class="string">"status"</span>, <span class="number">0</span>);<span class="comment">//0=失效；1=有效</span></div><div class="line">userDocument.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis());<span class="comment">//账户最后一次活动时间</span></div><div class="line">userDocuments.<span class="built_in">add</span>(userDocument);</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"添加用户“李四”"</span>);</div><div class="line">SolrInputDocument userDocument2 = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">userDocument2.addField(<span class="string">"Id"</span>, <span class="number">2000</span>);<span class="comment">//用户唯一编号</span></div><div class="line">userDocument2.addField(<span class="string">"type"</span>, <span class="number">1</span>);<span class="comment">//0=普通用户；1=VIP用户</span></div><div class="line">userDocument2.addField(<span class="string">"nickname"</span>, <span class="string">"李四"</span>);<span class="comment">//昵称</span></div><div class="line">userDocument2.addField(<span class="string">"status"</span>, <span class="number">0</span>);<span class="comment">//0=失效；1=有效</span></div><div class="line">userDocument2.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis() + <span class="number">1</span>);<span class="comment">//账户最后一次活动时间</span></div><div class="line">userDocuments.<span class="built_in">add</span>(userDocument2);</div><div class="line">userClient.<span class="built_in">add</span>(userDocuments);</div><div class="line"></div><div class="line"><span class="comment">//为"张三"加了2条订单信息</span></div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"用户“李四”添加2条订单信息"</span>);</div><div class="line">List<SolrInputDocument> orderDocuments = <span class="keyword">new</span> ArrayList<>();</div><div class="line">SolrInputDocument orderDocument = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">orderDocument.addField(<span class="string">"Id"</span>, <span class="number">3000</span>);<span class="comment">//订单唯一编号</span></div><div class="line">orderDocument.addField(<span class="string">"type"</span>, <span class="number">0</span>);<span class="comment">//0=微信订单；1=唯品会订单</span></div><div class="line">orderDocument.addField(<span class="string">"title"</span>, <span class="string">"张三的订单：商品001、商品002"</span>);<span class="comment">//订单概览</span></div><div class="line">orderDocument.addField(<span class="string">"remark"</span>, <span class="string">"我是备注..."</span>);<span class="comment">//订单备注</span></div><div class="line">orderDocument.addField(<span class="string">"uId"</span>, <span class="number">1000</span>);<span class="comment">//下单用户Id</span></div><div class="line">orderDocument.addField(<span class="string">"status"</span>, <span class="number">0</span>);<span class="comment">//0=创建；1=付款；2=付款超时</span></div><div class="line">orderDocument.addField(<span class="string">"amount"</span>, <span class="number">188</span>);<span class="comment">//金额、以分为单位</span></div><div class="line">orderDocument.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis() + <span class="number">2</span>);<span class="comment">//订单最后一次更新时间</span></div><div class="line">orderDocuments.<span class="built_in">add</span>(orderDocument);</div><div class="line">SolrInputDocument orderDocument2 = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">orderDocument2.addField(<span class="string">"Id"</span>, <span class="number">4000</span>);<span class="comment">//订单唯一编号</span></div><div class="line">orderDocument2.addField(<span class="string">"type"</span>, <span class="number">1</span>);<span class="comment">//0=微信订单；1=唯品会订单</span></div><div class="line">orderDocument2.addField(<span class="string">"title"</span>, <span class="string">"张三的订单：商品041、商品082"</span>);<span class="comment">//订单概览</span></div><div class="line">orderDocument2.addField(<span class="string">"remark"</span>, <span class="string">"我是备注..*******."</span>);<span class="comment">//订单备注</span></div><div class="line">orderDocument2.addField(<span class="string">"uId"</span>, <span class="number">1000</span>);<span class="comment">//下单用户Id</span></div><div class="line">orderDocument2.addField(<span class="string">"status"</span>, <span class="number">1</span>);<span class="comment">//0=创建；1=付款；2=付款超时</span></div><div class="line">orderDocument2.addField(<span class="string">"amount"</span>, <span class="number">9800</span>);<span class="comment">//金额、以分为单位</span></div><div class="line">orderDocument2.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis() + <span class="number">3</span>);<span class="comment">//订单最后一次更新时间</span></div><div class="line">orderDocuments.<span class="built_in">add</span>(orderDocument2);</div><div class="line">orderClient.<span class="built_in">add</span>(orderDocuments);</div><div class="line"></div><div class="line"><span class="comment">//准实时索引大约需要1s time window</span></div><div class="line"><span class="keyword">try</span> {</div><div class="line"> TimeUnit.SECONDS.sleep(<span class="number">2</span>);</div><div class="line">} <span class="keyword">catch</span> (InterruptedException e) {</div><div class="line"> e.printStackTrace();</div><div class="line">}</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"完成“张三”订单add"</span>);</div><div class="line"><span class="comment">//查询有已付款订单的用户昵称和用户Id</span></div><div class="line"><span class="comment">//类似的SQL：SELECT u.Id,u.nickname FROM user u INNER JOIN order o ON u.Id=o.uId WHERE o.status=1;</span></div><div class="line"></div><div class="line">SolrQuery query = <span class="keyword">new</span> SolrQuery();</div><div class="line">query.addField(<span class="string">"Id"</span>);</div><div class="line">query.addField(<span class="string">"nickname"</span>);</div><div class="line">query.setQuery(<span class="string">"*:*"</span>);</div><div class="line"><span class="comment">//status:1作为filter是必需的，不填会报错</span></div><div class="line"><span class="comment">//*Index与当前client相同时可省略</span></div><div class="line"><span class="comment">//query.addFilterQuery("{!join from=uId to=Id fromIndex=order toIndex=user}status:1");</span></div><div class="line">query.addFilterQuery(<span class="string">"{!join from=uId to=Id fromIndex=order}status:1"</span>);</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"query params:"</span> + query.toString());</div><div class="line">SolrDocumentList result = userClient.query(query).getResults();</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"result size:"</span> + result.<span class="built_in">size</span>());</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"查询有已付款订单的用户昵称和用户Id result:"</span> + Utils.objectToJsonWithoutException(result));</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">----------------------------------------------------------------------------</div><div class="line"></div><div class="line">console：</div><div class="line">添加用户“张三”</div><div class="line">添加用户“李四”</div><div class="line">用户“李四”添加<span class="number">2</span>条订单信息</div><div class="line">完成“张三”订单<span class="built_in">add</span></div><div class="line">query params:fl=Id%<span class="number">2</span>Cnickname&q=*%<span class="number">3</span>A*&fq=%<span class="number">7</span>B%<span class="number">21</span><span class="built_in">join</span>+from%<span class="number">3</span>DuId+to%<span class="number">3</span>DId+fromIndex%<span class="number">3</span>Dorder%<span class="number">7</span>Dstatus%<span class="number">3</span>A1</div><div class="line">result <span class="built_in">size</span>:<span class="number">1</span></div><div class="line">查询有已付款订单的用户昵称和用户Id result:[{<span class="string">"Id"</span>:<span class="number">1000</span>,<span class="string">"nickname"</span>:<span class="string">"张三"</span>}]</div></pre></td></tr></table></figure></p>
<h1 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h1><ul>
<li><a href="https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-JoinQueryParser" target="_blank" rel="external">Solr join query wiki</a></li>
<li><a href="http://josh-persistence.iteye.com/blog/2247289" target="_blank" rel="external">Solr cache分析</a></li>
<li><a href="http://blog.csdn.net/yangbutao/article/details/9179347" target="_blank" rel="external">Solr API 分析</a></li>
<li><a href="http://aoyouzi.iteye.com/blog/2291913" target="_blank" rel="external">Solr dataStructure 分析</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents#UpdatingPartsofDocuments-AtomicUpdates" target="_blank" rel="external">Solr update index 策略</a></li>
</ul>
]]></content>
<categories>
<category> Summary </category>
</categories>
<tags>
<tag> Solr </tag>
<tag> Design </tag>
</tags>
</entry>
<entry>
<title><![CDATA[开源贡献 - 升级Solr中文分词器 mmseg4j-solr]]></title>
<url>https://amao12580.github.io/post/2016/07/mmseg4j-solr-compatible-solr-6-1/</url>
<content type="html"><![CDATA[<h2 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h2><p>最近因为团队职责变动，接手了Solr的维护工作。仔细比对各个环境的使用版本才发现，都是使用的5.3，查了一下这个就快发布一年了。为什么不用最新版的6.1呢？有点忍不住技痒了，整一下升级到6.1的方案吧！</p>
<h2 id="简单介绍Solr"><a href="#简单介绍Solr" class="headerlink" title="简单介绍Solr"></a>简单介绍Solr</h2><p>Apache Solr (读音: SOLer) 是一个开源的搜索服务器。Solr 使用 Java 语言开发，主要基于 HTTP 和 Apache Lucene 实现。Apache Solr 中存储的资源是以 Document 为对象进行存储的。每个文档由一系列的 Field 构成，每个 Field 表示资源的一个属性。Solr 中的每个 Document 需要有能唯一标识其自身的属性，默认情况下这个属性的名字是 id，在 Schema 配置文件中使用：id进行描述。 Solr是一个高性能，采用Java开发，基于Lucene的全文搜索服务器。文档通过Http利用XML加到一个搜索集合中。查询该集合也是通过 http收到一个XML/JSON响应来实现。它的主要特性包括：高效、灵活的缓存功能，垂直搜索功能，高亮显示搜索结果，通过索引复制来提高可用性，提 供一套强大Data Schema来定义字段，类型和设置文本分析，提供基于Web的管理界面等。</p>
<h2 id="什么是mmseg4j-solr？"><a href="#什么是mmseg4j-solr？" class="headerlink" title="什么是mmseg4j-solr？"></a>什么是mmseg4j-solr？</h2><p>mmseg4j 用 Chih-Hao Tsai 的 <a href="http://technology.chtsai.org/mmseg/" target="_blank" rel="external">MMSeg 算法</a>实现的中文分词器，并实现 lucene 的 analyzer 和 solr 的TokenizerFactory 以方便在Lucene和Solr中使用。</p>
<p>2、MMSeg 算法有两种分词方法：Simple和Complex，都是基于正向最大匹配。Complex 加了四个规则过虑。官方说：词语的正确识别率达到了 98.41%。mmseg4j 已经实现了这两种分词算法。 <em> 1.5版的分词速度simple算法是 1100kb/s左右、complex算法是 700kb/s左右，（测试机：AMD athlon 64 2800+ 1G内存 xp）。 </em> 1.6版在complex基础上实现了最多分词(max-word)。“很好听” -> “很好|好听”; “中华人民共和国” -> “中华|华人|共和|国”; “中国人民银行” -> “中国|人民|银行”。 <em> 1.7-beta 版, 目前 complex 1200kb/s左右, simple 1900kb/s左右, 但内存开销了50M左右. 上几个版都是在10M左右. </em> 1.8 后,增加 CutLetterDigitFilter过虑器，切分“字母和数”混在一起的过虑器。</p>
<h2 id="升级方案"><a href="#升级方案" class="headerlink" title="升级方案"></a>升级方案</h2><h3 id="尝试强制升级"><a href="#尝试强制升级" class="headerlink" title="尝试强制升级"></a>尝试强制升级</h3><p>首先尝试直接强制升级。下载6.1版的zip，解压后，将5.3版本的核心配置文件、lib包拷贝并覆盖过来，启动时大量报错了：</p>
<p>lib包主要是:</p>
<ul>
<li>mmseg4j-core-1.10.0.jar</li>
<li>mmseg4j-solr-2.3.0.jar</li>
<li>pinyin4j-2.5.jar</li>
<li>pinyinFilter-0.2.jar</li>
</ul>
<p><img src="/img/solr-libs.png" alt=""></p>
<figure class="highlight applescript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">启动solr时报错</div><div class="line"></div><div class="line">cd /solr/bin</div><div class="line">solr start -m <span class="number">1</span>g -f</div><div class="line"></div><div class="line"><span class="number">2016</span><span class="number">-07</span><span class="number">-25</span> <span class="number">05</span>:<span class="number">03</span>:<span class="number">45.776</span> WARN (coreLoadExecutor<span class="number">-6</span>-thread<span class="number">-1</span>) [ ] o.a.s.s.FieldTypePluginLoader TokenFilterFactory <span class="keyword">is</span> using deprecated <span class="number">5.3</span><span class="number">.0</span> emulation. You should <span class="keyword">at</span> <span class="keyword">some</span> point declare <span class="keyword">and</span> reindex <span class="keyword">to</span> <span class="keyword">at</span> least <span class="number">6.0</span>, because <span class="number">5.</span>x emulation <span class="keyword">is</span> deprecated <span class="keyword">and</span> will be removed <span class="keyword">in</span> <span class="number">7.0</span></div><div class="line"><span class="number">2016</span><span class="number">-07</span><span class="number">-25</span> <span class="number">05</span>:<span class="number">03</span>:<span class="number">45.783</span> ERROR (coreLoadExecutor<span class="number">-6</span>-thread<span class="number">-2</span>) [ ] o.a.s.c.CoreContainer Error creating core [opportunity]: org.apache.solr.core.SolrResourceLoader.getInstanceDir()Ljava/lang/String;</div><div class="line">java.lang.NoSuchMethodError: org.apache.solr.core.SolrResourceLoader.getInstanceDir()Ljava/lang/String;</div><div class="line"> <span class="keyword">at</span> com.chenlb.mmseg4j.solr.Utils.getDict(Utils.java:<span class="number">18</span>)</div><div class="line"> <span class="keyword">at</span> com.chenlb.mmseg4j.solr.MMSegTokenizerFactory.inform(MMSegTokenizerFactory.java:<span class="number">65</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:<span class="number">699</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:<span class="number">184</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:<span class="number">56</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:<span class="number">75</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:<span class="number">108</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:<span class="number">79</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.CoreContainer.create(CoreContainer.java:<span class="number">810</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.CoreContainer.lambda$load$<span class="number">0</span>(CoreContainer.java:<span class="number">466</span>)</div><div class="line"> <span class="keyword">at</span> java.util.concurrent.FutureTask.<span class="built_in">run</span>(FutureTask.java:<span class="number">266</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$<span class="number">22</span>(ExecutorUtil.java:<span class="number">229</span>)</div><div class="line"> <span class="keyword">at</span> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:<span class="number">1142</span>)</div><div class="line"> <span class="keyword">at</span> java.util.concurrent.ThreadPoolExecutor$Worker.<span class="built_in">run</span>(ThreadPoolExecutor.java:<span class="number">617</span>)</div><div class="line"> <span class="keyword">at</span> java.lang.Thread.<span class="built_in">run</span>(Thread.java:<span class="number">745</span>)</div></pre></td></tr></table></figure>
<p>看样子是mmseg4j-solr.jar包中，文件com.chenlb.mmseg4j.solr.Utils.java，第18行有方法调用错误，猜测可能是因为API变动，导致版本不兼容。</p>
<p>Google一些关于mmseg4j-solr对solr版本的兼容情况，发现mmseg4j-solr还停留在对5.3版本的支持，没人维护了，那我们自食其力，看能否解决？</p>
<p>好在<a href="https://github.com/chenlb/mmseg4j-solr/blob/master/README.md(https://github.com/chenlb/mmseg4j-solr/blob/master/README.md" target="_blank" rel="external">mmseg4j-solr已经在GitHub开放了源码</a>，开始干吧！</p>
<h3 id="改源码进行兼容"><a href="#改源码进行兼容" class="headerlink" title="改源码进行兼容"></a>改源码进行兼容</h3><p>首先fork源码，clone到本机，对文件com.chenlb.mmseg4j.solr.Utils.java，第18行修改，这一行果然报Compile error。这就完了吗？试试package一下吧！maven install过程中，执行MMSegTokenizerFactoryTest case也报错了。</p>
<figure class="highlight vhdl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">Tests run: <span class="number">1</span>, Failures: <span class="number">0</span>, Errors: <span class="number">1</span>, Skipped: <span class="number">0</span>, <span class="built_in">Time</span> elapsed: <span class="number">10.921</span> sec <<< <span class="literal">FAILURE</span>! - <span class="keyword">in</span> com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest</div><div class="line">com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest <span class="built_in">Time</span> elapsed: <span class="number">10.905</span> sec <<< <span class="literal">ERROR</span>!</div><div class="line">java.lang.RuntimeException: org.apache.solr.common.SolrException: Solr no longer supports forceful unlocking via the <span class="symbol">'unlockOnStartup</span>' option. This <span class="keyword">is</span> no longer necessary <span class="keyword">for</span> the <span class="keyword">default</span> lockType except <span class="keyword">in</span> situations where it would be dangerous <span class="keyword">and</span> should <span class="keyword">not</span> be done. <span class="keyword">For</span> other lockTypes <span class="keyword">and</span>/<span class="keyword">or</span> directoryFactory options it may also be dangerous <span class="keyword">and</span> users must resolve problematic locks manually.</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.beforeClass(MMSegTokenizerFactoryTest.java:<span class="number">33</span>)</div><div class="line">Caused by: org.apache.solr.common.SolrException: Solr no longer supports forceful unlocking via the <span class="symbol">'unlockOnStartup</span>' option. This <span class="keyword">is</span> no longer necessary <span class="keyword">for</span> the <span class="keyword">default</span> lockType except <span class="keyword">in</span> situations where it would be dangerous <span class="keyword">and</span> should <span class="keyword">not</span> be done. <span class="keyword">For</span> other lockTypes <span class="keyword">and</span>/<span class="keyword">or</span> directoryFactory options it may also be dangerous <span class="keyword">and</span> users must resolve problematic locks manually.</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.beforeClass(MMSegTokenizerFactoryTest.java:<span class="number">33</span>)</div><div class="line"></div><div class="line"></div><div class="line">Results :</div><div class="line"></div><div class="line">Tests <span class="keyword">in</span> <span class="literal">error</span>:</div><div class="line"> MMSegTokenizerFactoryTest.beforeClass:<span class="number">33</span>->SolrTestCaseJ4.initCore:<span class="number">443</span>->SolrTestCaseJ4.initCore:<span class="number">436</span>->SolrTestCaseJ4.initCore:<span class="number">594</span>->SolrTestCaseJ4.createCore:<span class="number">601</span> ? Runtime</div><div class="line"></div><div class="line">Tests run: <span class="number">6</span>, Failures: <span class="number">0</span>, Errors: <span class="number">1</span>, Skipped: <span class="number">0</span></div><div class="line"></div><div class="line">[INFO] <span class="comment">------------------------------------------------------------------------</span></div><div class="line">[INFO] BUILD <span class="literal">FAILURE</span></div><div class="line">[INFO] <span class="comment">------------------------------------------------------------------------</span></div><div class="line">[INFO] Total <span class="built_in">time</span>: <span class="number">17.375</span> s</div><div class="line">[INFO] Finished at: <span class="number">2016</span>-<span class="number">07</span>-<span class="number">25</span>T13:<span class="number">18</span>:<span class="number">43</span>+<span class="number">08</span>:<span class="number">00</span></div><div class="line">[INFO] Final Memory: <span class="number">32</span>M/<span class="number">218</span>M</div><div class="line">[INFO] <span class="comment">------------------------------------------------------------------------</span></div><div class="line">[<span class="literal">ERROR</span>] Failed <span class="keyword">to</span> execute goal org.apache.maven.plugins:maven-surefire-plugin:<span class="number">2.19</span>.<span class="number">1</span>:test (<span class="keyword">default</span>-test) <span class="keyword">on</span> project mmseg4j-solr: There are test failures.</div><div class="line">[<span class="literal">ERROR</span>]</div><div class="line">[<span class="literal">ERROR</span>] Please refer <span class="keyword">to</span> D:\GitHub\mmseg4j-solr\target\surefire-reports <span class="keyword">for</span> the individual test results.</div><div class="line">[<span class="literal">ERROR</span>] -> [Help <span class="number">1</span>]</div><div class="line">[<span class="literal">ERROR</span>]</div><div class="line">[<span class="literal">ERROR</span>] <span class="keyword">To</span> see the full stack trace <span class="keyword">of</span> the errors, re-run Maven <span class="keyword">with</span> the -e switch.</div><div class="line">[<span class="literal">ERROR</span>] Re-run Maven using the -X switch <span class="keyword">to</span> enable full debug logging.</div><div class="line">[<span class="literal">ERROR</span>]</div><div class="line">[<span class="literal">ERROR</span>] <span class="keyword">For</span> more information about the errors <span class="keyword">and</span> possible solutions, please read the following articles:</div><div class="line">[<span class="literal">ERROR</span>] [Help <span class="number">1</span>] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException</div></pre></td></tr></table></figure>
<p>看样子是unlockOnStartup选项不再受支持，需要移除，我们移除一下试试吧，找到该选项位于Test Resource包下的文件solrconfig.xml中，删掉了。再尝试install！还是报错！</p>
<figure class="highlight stata"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">Tests <span class="keyword">run</span>: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.586 sec <<< FAILURE! - <span class="keyword">in</span> com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest</div><div class="line">test_mmseg4j(com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest) Time elapsed: 0.012 sec <<< <span class="keyword">ERROR</span>!</div><div class="line">org.apache.solr.common.SolrException: SolrCore 'mmseg4j_core' is not available due to init failure: <span class="keyword">Error</span> loading <span class="keyword">class</span> 'org.apache.solr.handler.admin.AdminHandlers'</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.getDictionaryByFieldType(MMSegTokenizerFactoryTest.java:37)</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.test_mmseg4j(MMSegTokenizerFactoryTest.java:79)</div><div class="line">Caused <span class="keyword">by</span>: org.apache.solr.common.SolrException: <span class="keyword">Error</span> loading <span class="keyword">class</span> 'org.apache.solr.handler.admin.AdminHandlers'</div><div class="line">Caused <span class="keyword">by</span>: org.apache.solr.common.SolrException: <span class="keyword">Error</span> loading <span class="keyword">class</span> 'org.apache.solr.handler.admin.AdminHandlers'</div><div class="line">Caused <span class="keyword">by</span>: java.lang.ClassNotFoundException: org.apache.solr.handler.admin.AdminHandlers</div><div class="line"></div><div class="line"></div><div class="line">Results :</div><div class="line"></div><div class="line">Tests <span class="keyword">in</span> <span class="keyword">error</span>:</div><div class="line"> MMSegTokenizerFactoryTest.test_mmseg4j:79->getDictionaryByFieldType:37 ? Solr ...</div><div class="line"></div><div class="line">Tests <span class="keyword">run</span>: 6, Failures: 0, Errors: 1, Skipped: 0</div><div class="line"></div><div class="line">[INFO] ------------------------------------------------------------------------</div><div class="line">[INFO] BUILD FAILURE</div><div class="line">[INFO] ------------------------------------------------------------------------</div><div class="line">[INFO] <span class="keyword">Total</span> time: 6.131 <span class="built_in">s</span></div><div class="line">[INFO] Finished at: 2016-07-25T13:20:50+08:00</div><div class="line">[INFO] Final <span class="keyword">Memory</span>: 23M/226M</div><div class="line">[INFO] ------------------------------------------------------------------------</div><div class="line">[<span class="keyword">ERROR</span>] Failed to execute goal org.apache.maven.plugins:maven-surefire-<span class="keyword">plugin</span>:2.19.1:<span class="keyword">test</span> (default-<span class="keyword">test</span>) <span class="keyword">on</span> project mmseg4j-solr: There are <span class="keyword">test</span> failures.</div><div class="line">[<span class="keyword">ERROR</span>]</div><div class="line">[<span class="keyword">ERROR</span>] Please refer to <span class="keyword">D</span>:\GitHub\mmseg4j-solr\target\surefire-reports <span class="keyword">for</span> the individual <span class="keyword">test</span> results.</div><div class="line">[<span class="keyword">ERROR</span>] -> [<span class="keyword">Help</span> 1]</div><div class="line">[<span class="keyword">ERROR</span>]</div><div class="line">[<span class="keyword">ERROR</span>] To see the full <span class="keyword">stack</span> trace of the errors, re-<span class="keyword">run</span> Maven with the -<span class="keyword">e</span> switch.</div><div class="line">[<span class="keyword">ERROR</span>] Re-<span class="keyword">run</span> Maven using the -X switch to enable full debug logging.</div><div class="line">[<span class="keyword">ERROR</span>]</div><div class="line">[<span class="keyword">ERROR</span>] <span class="keyword">For</span> <span class="keyword">more</span> information <span class="keyword">about</span> the errors and possible solutions, please <span class="keyword">read</span> the following articles:</div><div class="line">[<span class="keyword">ERROR</span>] [<span class="keyword">Help</span> 1] http:<span class="comment">//cwiki.apache.org/confluence/display/MAVEN/MojoFailureException</span></div></pre></td></tr></table></figure>
<p>看样子是org.apache.solr.handler.admin.AdminHandlers无法加载，该配置位于solrconfig.xml。google发现该api已经换了，与5.3版不再兼容。找了一份6.1的basic solrconfig.xml进行替换，再试着install就成功啦！</p>
<h2 id="PR"><a href="#PR" class="headerlink" title="PR"></a>PR</h2><p>看来改源码，也没有很难嘛！持续几个问题解决后，我们也得到了兼容性的jar包，实验在测试环境可以用！考虑到很多人也需要版本升级，本着取之于开源，反哺开源，则不竭的精神。对源repo提交了pull request。PR地址：<a href="https://github.com/chenlb/mmseg4j-solr/pull/26" target="_blank" rel="external">https://github.com/chenlb/mmseg4j-solr/pull/26</a>。</p>
<p>该PR还没有merger到master，如果等不及请下载我提供的：<a href="/file/mmseg4j-solr-2.3.2.rar">mmseg4j-solr-2.3.2.jar</a></p>
]]></content>
<categories>
<category> Record </category>
</categories>
<tags>
<tag> Solr </tag>
<tag> Mmseg4j-solr </tag>
<tag> Open-Source-Contribution </tag>
</tags>
</entry>
<entry>
<title><![CDATA[MySQL集锦 - IN 真会导致全表扫描吗？]]></title>
<url>https://amao12580.github.io/post/2016/07/MySQL-in-operator-must-lead-to-full-scan/</url>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>项目版本迭代慢了下来，有了段空闲时间，就对近些阶段的代码进行Code review，主要还是保持设计正确性和易维护性吧。</p>
<h1 id="有意思的代码"><a href="#有意思的代码" class="headerlink" title="有意思的代码"></a>有意思的代码</h1><p>由于积累了数个里程碑，review的工作量还挺大，我们分工协作，我这块主要看推荐模块，发现了几段有意思的代码，贴出来看看吧。</p>
<figure class="highlight lasso"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">Result<Record1<<span class="built_in">Integer</span>>> ids = create.<span class="keyword">select</span>(SITING_OPPORTUNITY_DISTRICT.OPPORTUNITY_ID).</div><div class="line"> from(SITING_OPPORTUNITY_DISTRICT).</div><div class="line"> <span class="keyword">where</span>(SITING_OPPORTUNITY_DISTRICT.DISTRICT_ID.like(districtId + <span class="string">"%"</span>)).</div><div class="line"> groupBy(SITING_OPPORTUNITY_DISTRICT.OPPORTUNITY_ID).</div><div class="line"> fetch();</div><div class="line"><span class="built_in">Set</span><<span class="built_in">Integer</span>> opportunityIds = <span class="literal">new</span> HashSet<>(ids.size());</div><div class="line">for (Record1<<span class="built_in">Integer</span>> r : ids) {</div><div class="line"> opportunityIds.add(r.getValue(SITING_OPPORTUNITY_DISTRICT.OPPORTUNITY_ID));</div><div class="line">}</div><div class="line"><span class="keyword">if</span> (opportunityIds.size() == <span class="number">1</span>) {</div><div class="line"> condition = condition.<span class="literal">and</span>(SITING_OPPORTUNITY.OPPORTUNITY_ID.<span class="literal">eq</span>(getSetFirstElement(opportunityIds)));</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"> condition = condition.<span class="literal">and</span>(SITING_OPPORTUNITY.OPPORTUNITY_ID.<span class="keyword">in</span>(opportunityIds));</div><div class="line">}</div></pre></td></tr></table></figure>
<p>这段代码的大意是：在MySQL数据库，表SITING_OPPORTUNITY_DISTRICT中，查询符合DISTRICT_ID字段条件的不重复OPPORTUNITY_ID字段，将这些字段放到一个集合。为描述方便，这个集合下文用ids代替。</p>
<p>1.如果ids的大小等于一，则拼上了一个SQL where条件，使用equal过滤ids中的第一个元素值，我们称这个SQL为A。</p>
<figure class="highlight n1ql"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">SQL A template</div><div class="line"></div><div class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> SITING_OPPORTUNITY <span class="keyword">WHERE</span> OPPORTUNITY_ID=?;</div></pre></td></tr></table></figure>
<p>2.如果ids的大小大于一，则拼上了一个SQL where条件，使用in过滤ids中的所有元素值，我们称这个SQL为B。</p>
<figure class="highlight n1ql"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">SQL B template</div><div class="line"></div><div class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> SITING_OPPORTUNITY <span class="keyword">WHERE</span> OPPORTUNITY_ID <span class="keyword">in</span>(?,?,?,?,?...);</div></pre></td></tr></table></figure>
<h1 id="存疑"><a href="#存疑" class="headerlink" title="存疑"></a>存疑</h1><p>项目中，类似这样的写法大概有10多处，都是按照集合大小来动态build SQL语句。ids的大小是跟用户的个性化数据有关的，统计了一下，超过70%的用户数据，ids大小是小于等于一的，所以这样调整代码的目的是为了降低全表扫描的概率？问了一下，得到确定的答复！但是SITING_OPPORTUNITY表在数据库结构设计评审时，已经加上了UNIQUE索引，为什么还需要这样写代码呢？难道MySQL强大的SQL优化器不能自动来完成？非得要在应用层来做？</p>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line">目标表SITING_OPPORTUNITY结构示意：</div><div class="line"></div><div class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> <span class="string">`siting_opportunity`</span> (</div><div class="line"> <span class="string">`id`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> AUTO_INCREMENT <span class="keyword">COMMENT</span> <span class="string">'ID'</span>,</div><div class="line"> <span class="string">`opportunity_id`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'商机ID'</span>,</div><div class="line"> <span class="string">`industry_id`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'行业id'</span>,</div><div class="line"> <span class="string">`min_area`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最小面积'</span>,</div><div class="line"> <span class="string">`max_area`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最大面积'</span>,</div><div class="line"> <span class="string">`min_rent`</span> <span class="built_in">bigint</span>(<span class="number">15</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最小租金'</span>,</div><div class="line"> <span class="string">`max_rent`</span> <span class="built_in">bigint</span>(<span class="number">15</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最大租金'</span>,</div><div class="line"> <span class="string">`is_deal`</span> tinyint(<span class="number">1</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'是否成交。0=未成交；1=已成交'</span>,</div><div class="line"> <span class="string">`slogan`</span> <span class="built_in">varchar</span>(<span class="number">20</span>) <span class="keyword">COLLATE</span> utf8mb4_unicode_ci <span class="keyword">DEFAULT</span> <span class="literal">NULL</span> <span class="keyword">COMMENT</span> <span class="string">'广告语'</span>,</div><div class="line"> PRIMARY <span class="keyword">KEY</span> (<span class="string">`id`</span>),</div><div class="line"> <span class="keyword">UNIQUE</span> <span class="keyword">KEY</span> <span class="string">`opportunity_id`</span> (<span class="string">`opportunity_id`</span>),</div><div class="line"> <span class="keyword">KEY</span> <span class="string">`fk_sitingOpportunity_industry`</span> (<span class="string">`industry_id`</span>),</div><div class="line"> <span class="keyword">CONSTRAINT</span> <span class="string">`fk_sitingOpportunity_industry`</span> FOREIGN <span class="keyword">KEY</span> (<span class="string">`industry_id`</span>) <span class="keyword">REFERENCES</span> <span class="string">`industry`</span> (<span class="string">`code`</span>) <span class="keyword">ON</span> <span class="keyword">UPDATE</span> <span class="keyword">NO</span> <span class="keyword">ACTION</span></div><div class="line">) <span class="keyword">ENGINE</span>=<span class="keyword">InnoDB</span> <span class="keyword">DEFAULT</span> <span class="keyword">CHARSET</span>=utf8mb4 <span class="keyword">COLLATE</span>=utf8mb4_unicode_ci <span class="keyword">COMMENT</span>=<span class="string">'选址商机表'</span>;</div></pre></td></tr></table></figure>
<h1 id="解谜"><a href="#解谜" class="headerlink" title="解谜"></a>解谜</h1><p>我们绝大多数人，对in操作是否会带来全表扫描开销，还停留在MySQL很古老的版本认识上。网上一篇<a href="http://itindex.net/detail/55421-mysql-sql-%E8%AF%AD%E5%8F%A5" target="_blank" rel="external">30条SQL优化军规</a>流传甚广，大多数人奉此为神道。</p>
<p>其中有一条这样描述：“5.in 和 not in 也要慎用，否则会导致全表扫描”，但没说为什么这样认为，以及面向的MySQL版本和配置也没有说明。</p>
<p>这句话在Google上查询，至少在2004年7月就已经发布到网上了，起始发布的站点出自<a href="http://hovertree.com/h/bjaf/u935eb54.htm" target="_blank" rel="external">《SQL语句优化原则_Sql Server_何问起》</a>，整整12年过去了，还有很多圈内知名IT站点在发布同样的文章。</p>
<ul>
<li>segmentfault：<a href="https://segmentfault.com/a/1190000005008401" target="_blank" rel="external">《mysql语句优化建议-2016年4月26日发布》</a></li>
</ul>
<ul>
<li>红黑联盟：<a href="http://www.2cto.com/database/201606/514022.html" target="_blank" rel="external">《数据库查询优化方法总结-2016年6月1日发布》</a></li>
</ul>
<p>首先，起始发布的文章是针对SQL server数据库的47条优化建议，已经太过久远，这些优化技巧是否还完全适用现今的数据库呢？更别提是否还适用于MySQL数据库了。我猜想是哪位“DBA大牛”应付交差，东抄一段西抄一段放上网，哪想到会有人当了真，更惨的是还完全信了。还真是：尽信书，不如无书。干我们这行，获取信息很容易，但甄别信息很难，希望大家引以为戒。</p>
<h1 id="求是"><a href="#求是" class="headerlink" title="求是"></a>求是</h1><p>怀抱求是精神，我们一起来实践，看看MySQL对于IN操作符的处理。</p>
<p>下文中用的MySQL版本：5.7.13 windows x64 解压安装版.</p>
<figure class="highlight makefile"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div></pre></td><td class="code"><pre><div class="line"><span class="section">MySQL配置文件:</span></div><div class="line"></div><div class="line">[client]</div><div class="line">default-character-set=utf8</div><div class="line">[mysql]</div><div class="line">default-character-set=utf8</div><div class="line">[mysqld]</div><div class="line"><span class="comment">########basic settings########</span></div><div class="line"><span class="comment">#skip-grant-tables,默认注释掉。重置root密码时用得上：http://www.apelearn.com/bbs/thread-9205-1-1.html</span></div><div class="line"><span class="comment">#skip-grant-tables</span></div><div class="line"></div><div class="line">skip-ssl</div><div class="line">secure-file-priv = NULL</div><div class="line"><span class="comment">#服务器唯一ID，默认是1，一般取IP最后一段</span></div><div class="line">log-bin=mysql-bin</div><div class="line">server-id = 100</div><div class="line">port = 3306</div><div class="line">user = mysql</div><div class="line">bind_address = 127.0.0.1</div><div class="line">autocommit = 0</div><div class="line"><span class="comment">#character_set_server=utf8mb4</span></div><div class="line">character_set_server=utf8</div><div class="line">skip_name_resolve = 1</div><div class="line">max_connections = 8000</div><div class="line">max_connect_errors = 10000</div><div class="line">basedir =<span class="string">"F:\mysql/"</span></div><div class="line">datadir =<span class="string">"F:\mysql/data/"</span></div><div class="line">tmpdir =<span class="string">"F:\mysql/temp/"</span></div><div class="line">socket =<span class="string">"F:\mysql/data/mysql.sock"</span></div><div class="line"><span class="section">pid-file="F:\mysql/data/current.pid"</span></div><div class="line">transaction_isolation = READ-COMMITTED</div><div class="line">explicit_defaults_for_timestamp = 1</div><div class="line">join_buffer_size = 134217728</div><div class="line">tmp_table_size = 67108864</div><div class="line">max_allowed_packet = 128MB</div><div class="line">sql_mode = <span class="string">"STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER"</span></div><div class="line">interactive_timeout = 1800</div><div class="line">wait_timeout = 1800</div><div class="line">read_buffer_size = 16777216</div><div class="line">read_rnd_buffer_size = 33554432</div><div class="line">sort_buffer_size = 33554432</div><div class="line"><span class="comment">########log settings########</span></div><div class="line"><span class="section">log-error="F:\mysql/log/error/error.log"</span></div><div class="line">slow_query_log = ON</div><div class="line">slow_query_log_file = <span class="string">"F:\mysql/log/slow/slow.log"</span></div><div class="line">log_queries_not_using_indexes = 1</div><div class="line">log_slow_admin_statements = 1</div><div class="line">log_slow_slave_statements = 1</div><div class="line">log_throttle_queries_not_using_indexes = 10</div><div class="line"><span class="comment">#保留7天的日志</span></div><div class="line">expire_logs_days = 7</div><div class="line"><span class="comment">#记录执行时间超过5秒的慢查询</span></div><div class="line">long_query_time = 5</div><div class="line">min_examined_row_limit = 100</div><div class="line"><span class="comment">########replication settings########</span></div><div class="line">master_info_repository = TABLE</div><div class="line">relay_log_info_repository = TABLE</div><div class="line">log_bin = bin.log</div><div class="line">sync_binlog = 1</div><div class="line">gtid_mode = on</div><div class="line">enforce_gtid_consistency = 1</div><div class="line">log_slave_updates</div><div class="line">binlog_format = row</div><div class="line">relay_log = relay.log</div><div class="line">relay_log_recovery = 1</div><div class="line">binlog_gtid_simple_recovery = 1</div><div class="line">slave_skip_errors = ddl_exist_errors</div><div class="line"><span class="comment">########innodb settings########</span></div><div class="line">innodb_page_size = 16384</div><div class="line"><span class="comment">#innodb_buffer_pool_size = 6G</span></div><div class="line">innodb_buffer_pool_instances = 8</div><div class="line">innodb_buffer_pool_load_at_startup = 1</div><div class="line">innodb_buffer_pool_dump_at_shutdown = 1</div><div class="line">innodb_lru_scan_depth = 2000</div><div class="line">innodb_lock_wait_timeout = 5</div><div class="line">innodb_io_capacity = 4000</div><div class="line">innodb_io_capacity_max = 8000</div><div class="line"><span class="comment">#innodb_flush_method = O_DIRECT</span></div><div class="line">innodb_flush_method=normal</div><div class="line"><span class="comment">#innodb_file_format = Barracuda</span></div><div class="line"><span class="comment">#innodb_file_format_max = Barracuda</span></div><div class="line">innodb_log_group_home_dir = <span class="string">"F:\mysql/log/redolog\"</span></div><div class="line"><span class="string">innodb_undo_directory = "</span>F:\mysql/log/undolog/<span class="string">"</span></div><div class="line"><span class="string">innodb_undo_logs = 128</span></div><div class="line"><span class="string">innodb_undo_tablespaces = 3</span></div><div class="line"><span class="string">innodb_flush_neighbors = 0</span></div><div class="line"><span class="string">#innodb_log_file_size = 4G</span></div><div class="line"><span class="string">innodb_log_file_size = 256MB</span></div><div class="line"><span class="string">innodb_log_buffer_size = 16777216</span></div><div class="line"><span class="string">innodb_purge_threads = 4</span></div><div class="line"><span class="string">innodb_large_prefix = 1</span></div><div class="line"><span class="string">innodb_thread_concurrency = 64</span></div><div class="line"><span class="string">innodb_print_all_deadlocks = 1</span></div><div class="line"><span class="string">innodb_strict_mode = 1</span></div><div class="line"><span class="string">innodb_sort_buffer_size = 67108864</span></div><div class="line"><span class="string">innodb_flush_log_at_trx_commit = 2</span></div><div class="line"><span class="string">########semi sync replication settings########</span></div><div class="line"><span class="string">plugin_dir="</span>F:\mysql/lib/plugin<span class="string">"</span></div><div class="line"><span class="string">plugin_load = "</span>rpl_semi_sync_master=semisync_master.dll;rpl_semi_sync_slave=semisync_slave.dll<span class="string">"</span></div><div class="line"><span class="string">loose_rpl_semi_sync_master_enabled = 1</span></div><div class="line"><span class="string">loose_rpl_semi_sync_slave_enabled = 1</span></div><div class="line"><span class="string">loose_rpl_semi_sync_master_timeout = 5000</span></div><div class="line"><span class="string"></span></div><div class="line"><span class="string">[mysqld-5.7]</span></div><div class="line"><span class="string">innodb_buffer_pool_dump_pct = 40</span></div><div class="line"><span class="string">innodb_page_cleaners = 4</span></div><div class="line"><span class="string">innodb_undo_log_truncate = 1</span></div><div class="line"><span class="string">innodb_max_undo_log_size = 2G</span></div><div class="line"><span class="string">innodb_purge_rseg_truncate_frequency = 128</span></div><div class="line"><span class="string">binlog_gtid_simple_recovery=1</span></div><div class="line"><span class="string">log_timestamps=system</span></div><div class="line"><span class="string">#transaction_write_set_extraction=MURMUR32</span></div><div class="line"><span class="string">show_compatibility_56=on</span></div></pre></td></tr></table></figure>
<p>先用show index from siting_opportunity;语句查看一下索引情况。</p>
<p><img src="/img/showIndexFromSitingOpportunity.png" alt=""><br>由于索引opportunity_id（key_name）是unique类型，可以看到其散列程度（Cardinality）是非常高的，因此该索引是很有效的。相应的该表的记录数是：476772行。</p>
<h2 id="无索引"><a href="#无索引" class="headerlink" title="无索引"></a>无索引</h2><p>字段min_area没有索引，<code>min_area</code> int(11) NOT NULL DEFAULT ‘0’ COMMENT ‘最小面积’。<br><img src="/img/withoutIndex.png" alt=""></p>
<p>很显然，优化器告诉我们，需要进行476772行扫描才能完成计算，也就是全表扫描了。</p>
<h2 id="Unique索引"><a href="#Unique索引" class="headerlink" title="Unique索引"></a>Unique索引</h2><p>先看在表SITING_OPPORTUNITY，OPPORTUNITY_ID字段的处理情况。<br><img src="/img/userUniqueIndex.png" alt=""><br>明显的可以看到，MySQL优化器是选择了索引，没有进行全表扫描，而且rows也为7。</p>
<h2 id="主键索引"><a href="#主键索引" class="headerlink" title="主键索引"></a>主键索引</h2><p>我们知道主键一定是唯一性索引,那情况应该跟上一节差不多？动动手吧！<br><img src="/img/userPrimaryIndex.png" alt=""><br>情况确实跟Unique索引差不多，pass！</p>
<h2 id="外键索引"><a href="#外键索引" class="headerlink" title="外键索引"></a>外键索引</h2><p>在设置外键时，数据库会同时为这个字段设置一个普通索引，所以外键的设置也应该遵循索引的设置策略。<br><img src="/img/useFKIndex.png" alt=""><br>使用外键字段时，使用in操作符，也使用了索引。</p>
<h2 id="普通索引"><a href="#普通索引" class="headerlink" title="普通索引"></a>普通索引</h2><p>我们换一个有普通索引，但该字段不是外键的表来验证。<br><img src="/img/userNormalIndex.png" alt=""></p>
<h1 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h1><p>1.在in操作中，集合体积过大时，索引还会生效吗？</p>
<p>2.在嵌套查询中，内层查询使用in，是否会有效率问题？</p>
<h1 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h1><p>本文中的存储引擎都是innodb，因时间有限，没有对Myisam等其他存储引擎进行类似实验，希望动手能力强的你可以完成！</p>
<p>MySQL在2010年发布5.5版本中，优化器对in操作符可以自动完成优化，针对建立了索引的列可以使用索引，没有索引的列还是会走全表扫描。因此应用层不需要再进行画蛇添足啦，当然优化的精神是可嘉的，但在猜测中进行的优化可能与实际背离甚远！</p>
<p>互联网上的信息太过广泛，但这不应该成为我们掉以轻心的借口，抱着存疑求是的精神进行甄别，任何信息只有在我们实践验证后方可全信，用在生产开发上的知识，不容有失！</p>
]]></content>
<categories>
<category> CodeReview </category>
</categories>
<tags>
<tag> SQL </tag>
<tag> MySQL </tag>
</tags>
</entry>
<entry>
<title><![CDATA[LeetCode刷题 - 求三个和为零的整数组合]]></title>
<url>https://amao12580.github.io/post/2016/06/LeetCode-training-3-elements-sum-zero/</url>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>完整的代码整理好了，请移步GitHub：</p>
<ul>
<li><a href="https://github.com/amao12580/algorithm/tree/master/src/main/java/leetcode/sumEqualZero" target="_blank" rel="external">https://github.com/amao12580/algorithm/tree/master/src/main/java/leetcode/sumEqualZero</a></li>
</ul>
<p>知乎《职人介绍所》中，请到了不论是技术水平还是名气（也许包括薪水）都在业界名列前茅的<a href="https://www.zhihu.com/people/ec03b8e839a6fb763e1b8113455362db" target="_blank" rel="external">@winter</a>和<a href="https://www.zhihu.com/people/78e3b98074a915b222ae1be4ab038a6e" target="_blank" rel="external">@赵劼</a>。链接在这里：<a href="http://daily.zhihu.com/story/8467704" target="_blank" rel="external">《职人介绍所》第 21 期：中国特别有名的两位程序员 winter 和赵劼来了</a></p>
<p>在围观这期剪辑版的视频后，对手写代码的那道题目产生了高昂的兴趣，细想下来这道题还是挺难的，尤其是现场还不准掀桌呢！哈哈！</p>
<h1 id="题目内容"><a href="#题目内容" class="headerlink" title="题目内容"></a>题目内容</h1><p>有一个随机整数数组，从中挑ABC三个整数，让ABC三个整数加起来等于零，看有多少个不重复的组合？</p>
<p>在LeetCode上也有这道题：<a href="https://leetcode.com/problems/3sum/" target="_blank" rel="external">3Sum _ LeetCode OJ</a></p>
<h1 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h1><p>很简单，就是把正负数配对，然后看他们的和的取反是不是存在，存在就说明这是一个合法的结果。</p>
<p>例如发现1和-3这两个正负数，求和为-2，再取反得到2，只需要在剩余数组中找到2即可，若存在则是一个预备结果（还要考虑去重复）。</p>
<p>其实相当暴力，但关键在于正负数配对，导致性能可以很大提升。</p>
<p>因为三个数之和要等于零，必然是++-、–+和000（我们把0归到+或-之一）这三种情况中的一种。</p>
<h2 id="荷兰国旗问题"><a href="#荷兰国旗问题" class="headerlink" title="荷兰国旗问题"></a>荷兰国旗问题</h2><p>刚开始的思路走偏了，想到了荷兰国旗问题。<a href="http://www.cnblogs.com/junyuhuang/p/4390780.html" target="_blank" rel="external">”荷兰国旗难题“</a>是计算机科学中的一个程序难题，它是由Edsger Dijkstra提出，荷兰国旗是由红、白、蓝三色组成的。要求将数值种类固定的乱序数组排列为 “AAA,BBB,CCC” 的形式，其实对两端排好，中间的那部分就自然排好了。</p>
<p>原题还是很不一样的，首先在原题中数组是随机整数，而荷兰国旗问题中，数组是种类固定的数据（不一定是Number类型哦）。其次是相关性，荷兰国旗问题只需要按数据进行分类在找到位置即可（计数排序算法）。</p>
<h1 id="简单实现"><a href="#简单实现" class="headerlink" title="简单实现"></a>简单实现</h1><p>时间复杂度：O(n<sup>3</sup>)</p>
<p>空间复杂度：O(n)</p>
<figure class="highlight nimrod"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">public class <span class="type">SolutionForSimple</span> {</div><div class="line"> public <span class="keyword">static</span> <span class="built_in">void</span> main(<span class="type">String</span>[] args) {</div><div class="line"> <span class="built_in">int</span>[] originArray = {<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, -<span class="number">1</span>, -<span class="number">2</span>, -<span class="number">3</span>, -<span class="number">4</span>, -<span class="number">5</span>};</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println(<span class="string">"input:"</span> + <span class="type">Arrays</span>.toString(originArray));</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println(<span class="string">"------- simple -------"</span>);</div><div class="line"> println(simple(originArray, <span class="number">3</span>));</div><div class="line"> }</div><div class="line"></div><div class="line"> private <span class="keyword">static</span> <span class="built_in">void</span> println(<span class="built_in">int</span>[][] <span class="literal">result</span>) {</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println();</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println(<span class="string">"------- begin -------"</span>);</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println(<span class="string">"------- count:"</span> + <span class="literal">result</span>.length + <span class="string">" -------"</span>);</div><div class="line"> <span class="keyword">for</span> (<span class="built_in">int</span> i = <span class="number">0</span>; i < <span class="literal">result</span>.length; i++) {</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println((i + <span class="number">1</span>) + <span class="string">":"</span> + <span class="type">Arrays</span>.toString(<span class="literal">result</span>[i]));</div><div class="line"> }</div><div class="line"> <span class="type">System</span>.<span class="keyword">out</span>.println(<span class="string">"------- end -------"</span>);</div><div class="line"> }</div><div class="line"></div><div class="line"> /**</div><div class="line"> * 简单实现，<span class="number">3</span>重<span class="keyword">for</span>循环，同时去重</div><div class="line"> *</div><div class="line"> * @param originArray 原始数组</div><div class="line"> * @param number 几个数相加？</div><div class="line"> * @<span class="keyword">return</span> 结果组合</div><div class="line"> */</div><div class="line"> private <span class="keyword">static</span> <span class="built_in">int</span>[][] simple(<span class="built_in">int</span>[] originArray, <span class="built_in">int</span> number) {</div><div class="line"> <span class="built_in">int</span> length = originArray.length;</div><div class="line"> <span class="built_in">int</span>[][] <span class="literal">result</span> = new <span class="built_in">int</span>[length][number];</div><div class="line"> <span class="built_in">int</span> index = <span class="number">0</span>;</div><div class="line"> <span class="keyword">for</span> (<span class="built_in">int</span> i = <span class="number">0</span>; i < length; i++) {</div><div class="line"> <span class="keyword">for</span> (<span class="built_in">int</span> j = i + <span class="number">1</span>; j < length; j++) {</div><div class="line"> <span class="keyword">for</span> (<span class="built_in">int</span> k = j + <span class="number">1</span>; k < length; k++) {</div><div class="line"> <span class="keyword">if</span> ((originArray[i] + originArray[j] + originArray[k]) == <span class="number">0</span>) {</div><div class="line"> <span class="built_in">int</span>[] group = {originArray[i], originArray[j], originArray[k]};</div><div class="line"> <span class="keyword">if</span> (index >= <span class="literal">result</span>.length) {</div><div class="line"> <span class="literal">result</span> = resize(<span class="literal">result</span>);</div><div class="line"> }</div><div class="line"> <span class="literal">result</span>[index] = group;</div><div class="line"> index++;</div><div class="line"> }</div><div class="line"> }</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> ((<span class="literal">result</span>.length - index) > <span class="number">0</span>) {</div><div class="line"> <span class="literal">result</span> = trim(<span class="literal">result</span>, index);</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> <span class="literal">result</span>;</div><div class="line"> }</div><div class="line"></div><div class="line"> private <span class="keyword">static</span> <span class="built_in">int</span>[][] trim(<span class="built_in">int</span>[][] <span class="literal">result</span>, <span class="built_in">int</span> index) {</div><div class="line"> <span class="built_in">int</span>[][] newResult = new <span class="built_in">int</span>[index][<span class="literal">result</span>[<span class="number">0</span>].length];</div><div class="line"> <span class="type">System</span>.arraycopy(<span class="literal">result</span>, <span class="number">0</span>, newResult, <span class="number">0</span>, newResult.length);</div><div class="line"> <span class="keyword">return</span> newResult;</div><div class="line"> }</div><div class="line"></div><div class="line"> private <span class="keyword">static</span> <span class="built_in">int</span>[][] resize(<span class="built_in">int</span>[][] <span class="literal">result</span>) {</div><div class="line"> <span class="built_in">int</span>[][] newResult = new <span class="built_in">int</span>[<span class="literal">result</span>.length + <span class="literal">result</span>[<span class="number">0</span>].length][<span class="literal">result</span>[<span class="number">0</span>].length];</div><div class="line"> <span class="type">System</span>.arraycopy(<span class="literal">result</span>, <span class="number">0</span>, newResult, <span class="number">0</span>, <span class="literal">result</span>.length);</div><div class="line"> <span class="keyword">return</span> newResult;</div><div class="line"> }</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">以下是打印的结果：<span class="type">OK</span></div><div class="line"></div><div class="line">input:[<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, -<span class="number">1</span>, -<span class="number">2</span>, -<span class="number">3</span>, -<span class="number">4</span>, -<span class="number">5</span>]</div><div class="line">------- simple -------</div><div class="line"></div><div class="line">------- begin -------</div><div class="line">------- count:<span class="number">13</span> -------</div><div class="line"><span class="number">1</span>:[<span class="number">0</span>, <span class="number">1</span>, -<span class="number">1</span>]</div><div class="line"><span class="number">2</span>:[<span class="number">0</span>, <span class="number">2</span>, -<span class="number">2</span>]</div><div class="line"><span class="number">3</span>:[<span class="number">0</span>, <span class="number">3</span>, -<span class="number">3</span>]</div><div class="line"><span class="number">4</span>:[<span class="number">0</span>, <span class="number">4</span>, -<span class="number">4</span>]</div><div class="line"><span class="number">5</span>:[<span class="number">0</span>, <span class="number">5</span>, -<span class="number">5</span>]</div><div class="line"><span class="number">6</span>:[<span class="number">1</span>, <span class="number">2</span>, -<span class="number">3</span>]</div><div class="line"><span class="number">7</span>:[<span class="number">1</span>, <span class="number">3</span>, -<span class="number">4</span>]</div><div class="line"><span class="number">8</span>:[<span class="number">1</span>, <span class="number">4</span>, -<span class="number">5</span>]</div><div class="line"><span class="number">9</span>:[<span class="number">2</span>, <span class="number">3</span>, -<span class="number">5</span>]</div><div class="line"><span class="number">10</span>:[<span class="number">3</span>, -<span class="number">1</span>, -<span class="number">2</span>]</div><div class="line"><span class="number">11</span>:[<span class="number">4</span>, -<span class="number">1</span>, -<span class="number">3</span>]</div><div class="line"><span class="number">12</span>:[<span class="number">5</span>, -<span class="number">1</span>, -<span class="number">4</span>]</div><div class="line"><span class="number">13</span>:[<span class="number">5</span>, -<span class="number">2</span>, -<span class="number">3</span>]</div><div class="line">------- <span class="keyword">end</span> -------</div></pre></td></tr></table></figure>
<h1 id="递归实现"><a href="#递归实现" class="headerlink" title="递归实现"></a>递归实现</h1><p>简单实现的代码，虽然清晰易懂，但可复用性很低。例如现在将求三个数的和换为求四个数，基本上代码就得大改了，所以做了可重用的版本（OO思想的潜意识啊！），使用递归实现。</p>
<p>时间复杂度：O(n<sup>3</sup>)</p>
<p>空间复杂度：O(n)</p>
<figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">public</span> class SolutionForRecursion {</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> main(<span class="keyword">String</span>[] args) {</div><div class="line"> <span class="built_in">int</span>[] originArray = {<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">-1</span>, <span class="number">-2</span>, <span class="number">-3</span>, <span class="number">-4</span>, <span class="number">-5</span>};</div><div class="line"> System.out.<span class="built_in">println</span>(<span class="string">"input:"</span> + Arrays.toString(originArray));</div><div class="line"> System.out.<span class="built_in">println</span>(<span class="string">"------- recursion -------"</span>);</div><div class="line"> <span class="built_in">println</span>(recursion(originArray, <span class="number">3</span>));</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="built_in">println</span>(<span class="built_in">int</span>[][] result) {</div><div class="line"> System.out.<span class="built_in">println</span>();</div><div class="line"> System.out.<span class="built_in">println</span>(<span class="string">"------- begin -------"</span>);</div><div class="line"> System.out.<span class="built_in">println</span>(<span class="string">"------- count:"</span> + result.length + <span class="string">" -------"</span>);</div><div class="line"> <span class="keyword">for</span> (<span class="built_in">int</span> i = <span class="number">0</span>; i < result.length; i++) {</div><div class="line"> System.out.<span class="built_in">println</span>((i + <span class="number">1</span>) + <span class="string">":"</span> + Arrays.toString(result[i]));</div><div class="line"> }</div><div class="line"> System.out.<span class="built_in">println</span>(<span class="string">"------- end -------"</span>);</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="comment">/**</span></div><div class="line"><span class="comment"> * 3重for循环的递归版本</span></div><div class="line"><span class="comment"> *</span></div><div class="line"><span class="comment"> * @param originArray 原始数组</span></div><div class="line"><span class="comment"> * @param number 几个数相加？</span></div><div class="line"><span class="comment"> * @return 结果组合</span></div><div class="line"><span class="comment"> */</span></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span>[][] recursion(<span class="built_in">int</span>[] originArray, <span class="built_in">int</span> number) {</div><div class="line"> topLayer = number;</div><div class="line"> resultArray = <span class="keyword">new</span> <span class="built_in">int</span>[number * <span class="number">2</span>][number];</div><div class="line"> <span class="built_in">loop</span>(originArray, <span class="keyword">new</span> <span class="built_in">int</span>[topLayer], <span class="number">0</span>, <span class="number">0</span>, number);</div><div class="line"> <span class="keyword">if</span> ((resultArray.length - currentOffset) > <span class="number">0</span>) {</div><div class="line"> resultArray = <span class="built_in">trim</span>(resultArray, currentOffset);</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> resultArray;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span>[][] resultArray = <span class="keyword">null</span>;</div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span> currentOffset = <span class="number">0</span>;</div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span> topLayer = <span class="number">0</span>;</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> Integer <span class="built_in">loop</span>(<span class="built_in">int</span>[] originArray, <span class="built_in">int</span>[] chain, <span class="built_in">int</span> beginIndex, <span class="built_in">int</span> expectSumValue, <span class="built_in">int</span> currentLayer) {</div><div class="line"> Integer currentLayerLoopResult = <span class="keyword">null</span>;</div><div class="line"> <span class="keyword">for</span> (<span class="built_in">int</span> i = beginIndex; i < originArray.length; i++) {</div><div class="line"> <span class="built_in">int</span> currentValue = originArray[i];</div><div class="line"> <span class="keyword">if</span> (topLayer == currentLayer) {<span class="comment">//最顶层loop时将结果链置空</span></div><div class="line"> chain = <span class="keyword">new</span> <span class="built_in">int</span>[topLayer];</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> chain = <span class="built_in">copy</span>(chain);</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> (currentLayer > <span class="number">0</span>) {</div><div class="line"> Integer nextLayerLoopResult;</div><div class="line"> chain[topLayer - currentLayer] = currentValue;</div><div class="line"> <span class="keyword">if</span> (currentLayer == <span class="number">1</span>) {<span class="comment">//最底层不需要再向下loop</span></div><div class="line"> nextLayerLoopResult = <span class="number">0</span>;</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> nextLayerLoopResult = <span class="built_in">loop</span>(originArray, chain, i + <span class="number">1</span>, expectSumValue - currentValue, currentLayer - <span class="number">1</span>);</div><div class="line"> }</div><div class="line"> <span class="built_in">int</span> currentLayerSumValue = currentValue;</div><div class="line"> <span class="keyword">if</span> (nextLayerLoopResult == <span class="keyword">null</span>) {</div><div class="line"> <span class="keyword">continue</span>;</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> currentLayerSumValue += nextLayerLoopResult;</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> (currentLayerSumValue == expectSumValue) {</div><div class="line"> currentLayerLoopResult = currentValue;</div><div class="line"> <span class="keyword">if</span> (currentLayer == <span class="number">1</span>) {</div><div class="line"> mergeArray(chain);</div><div class="line"> <span class="keyword">return</span> currentLayerLoopResult;</div><div class="line"> }</div><div class="line"> }</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="keyword">return</span> currentLayerLoopResult;</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> currentLayerLoopResult;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span>[] <span class="built_in">copy</span>(<span class="built_in">int</span>[] chain) {</div><div class="line"> <span class="built_in">int</span>[] result = <span class="keyword">new</span> <span class="built_in">int</span>[chain.length];</div><div class="line"> System.arraycopy(chain, <span class="number">0</span>, result, <span class="number">0</span>, chain.length);</div><div class="line"> <span class="keyword">return</span> result;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> mergeArray(<span class="built_in">int</span>[] chain) {</div><div class="line"> <span class="keyword">if</span> (currentOffset >= resultArray.length) {</div><div class="line"> resultArray = resize(resultArray);</div><div class="line"> }</div><div class="line"> resultArray[currentOffset] = chain;</div><div class="line"> currentOffset++;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span>[][] <span class="built_in">trim</span>(<span class="built_in">int</span>[][] result, <span class="built_in">int</span> index) {</div><div class="line"> <span class="built_in">int</span>[][] newResult = <span class="keyword">new</span> <span class="built_in">int</span>[index][result[<span class="number">0</span>].length];</div><div class="line"> System.arraycopy(result, <span class="number">0</span>, newResult, <span class="number">0</span>, newResult.length);</div><div class="line"> <span class="keyword">return</span> newResult;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="built_in">int</span>[][] resize(<span class="built_in">int</span>[][] result) {</div><div class="line"> <span class="built_in">int</span>[][] newResult = <span class="keyword">new</span> <span class="built_in">int</span>[result.length + result[<span class="number">0</span>].length][result[<span class="number">0</span>].length];</div><div class="line"> System.arraycopy(result, <span class="number">0</span>, newResult, <span class="number">0</span>, result.length);</div><div class="line"> <span class="keyword">return</span> newResult;</div><div class="line"> }</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line">以下是打印的结果：OK</div><div class="line"></div><div class="line">input:[<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">-1</span>, <span class="number">-2</span>, <span class="number">-3</span>, <span class="number">-4</span>, <span class="number">-5</span>]</div><div class="line">------- recursion -------</div><div class="line"></div><div class="line">------- begin -------</div><div class="line">------- count:<span class="number">13</span> -------</div><div class="line"><span class="number">1</span>:[<span class="number">0</span>, <span class="number">1</span>, <span class="number">-1</span>]</div><div class="line"><span class="number">2</span>:[<span class="number">0</span>, <span class="number">2</span>, <span class="number">-2</span>]</div><div class="line"><span class="number">3</span>:[<span class="number">0</span>, <span class="number">3</span>, <span class="number">-3</span>]</div><div class="line"><span class="number">4</span>:[<span class="number">0</span>, <span class="number">4</span>, <span class="number">-4</span>]</div><div class="line"><span class="number">5</span>:[<span class="number">0</span>, <span class="number">5</span>, <span class="number">-5</span>]</div><div class="line"><span class="number">6</span>:[<span class="number">1</span>, <span class="number">2</span>, <span class="number">-3</span>]</div><div class="line"><span class="number">7</span>:[<span class="number">1</span>, <span class="number">3</span>, <span class="number">-4</span>]</div><div class="line"><span class="number">8</span>:[<span class="number">1</span>, <span class="number">4</span>, <span class="number">-5</span>]</div><div class="line"><span class="number">9</span>:[<span class="number">2</span>, <span class="number">3</span>, <span class="number">-5</span>]</div><div class="line"><span class="number">10</span>:[<span class="number">3</span>, <span class="number">-1</span>, <span class="number">-2</span>]</div><div class="line"><span class="number">11</span>:[<span class="number">4</span>, <span class="number">-1</span>, <span class="number">-3</span>]</div><div class="line"><span class="number">12</span>:[<span class="number">5</span>, <span class="number">-1</span>, <span class="number">-4</span>]</div><div class="line"><span class="number">13</span>:[<span class="number">5</span>, <span class="number">-2</span>, <span class="number">-3</span>]</div><div class="line">------- end -------</div></pre></td></tr></table></figure>
<h1 id="动态规划思想"><a href="#动态规划思想" class="headerlink" title="动态规划思想"></a>动态规划思想</h1><p>动态规划(Dynamic Programming)的基本思想是：将待求解的问题分解成若干个相互联系的子问题，先求解子问题，然后从这些子问题的解得到原问题的解；对于重复出现的子问题，只在第一次遇到的时候对它进行求解，并把答案保存起来，让以后再次遇到时直接引用答案，不必重新求解。动态规划算法将问题的解决方案视为一系列决策的结果，与贪婪算法不同的是，在贪婪算法中，每采用一次贪婪准则，便做出一个不可撤回的决策；而在动态规划算法中，还要考察每个最优决策序列中是否包含一个最优决策子序列，即问题是否具有最优子结构性质。</p>
<p>下面贴出解法，假设数组长度为N，需要M个数相加为零。则时间复杂度为：<code>N*M</code>，空间复杂度为：<code>N*</code>M<sup>2</sup>。</p>
<figure class="highlight zephir"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">SolutionForDynamicPrograming</span> </span>{</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> void main(String[] args) {</div><div class="line"> <span class="keyword">int</span>[] originArray = {<span class="number">-1</span>, <span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">-1</span>, <span class="number">-4</span>};</div><div class="line"> System.out.println(<span class="string">"input:"</span> + Arrays.toString(originArray));</div><div class="line"> System.out.println(<span class="string">"------- dynamicPrograming -------"</span>);</div><div class="line"> <span class="keyword">int</span> count = <span class="number">3</span>;<span class="comment">//3个数相加</span></div><div class="line"> <span class="keyword">int</span> expectSumValue = <span class="number">0</span>;<span class="comment">//期望累加值为零</span></div><div class="line"> SolutionForDynamicPrograming solutionForDynamicPrograming = <span class="keyword">new</span> SolutionForDynamicPrograming(count);</div><div class="line"> solutionForDynamicPrograming.dynamicPrograming(originArray, <span class="number">0</span>, count, expectSumValue);</div><div class="line"> solutionForDynamicPrograming.println(solutionForDynamicPrograming.toArray(solutionForDynamicPrograming.getSolution()));</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">int</span> count;</div><div class="line"> <span class="keyword">private</span> Set<String> uniques = <span class="keyword">new</span> HashSet<>();</div><div class="line"> <span class="keyword">private</span> Map<Integer, Map<Integer, <span class="keyword">List</span><<span class="keyword">List</span><Integer>>>> existedSolution = <span class="keyword">new</span> HashMap<>();</div><div class="line"> <span class="keyword">private</span> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> solution = <span class="keyword">new</span> ArrayList<>();</div><div class="line"></div><div class="line"> <span class="keyword">public</span> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> getSolution() {</div><div class="line"> <span class="keyword">return</span> solution;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">public</span> SolutionForDynamicPrograming(<span class="keyword">int</span> count) {</div><div class="line"> this.count = count;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">int</span>[][] toArray(<span class="keyword">List</span><<span class="keyword">List</span><Integer>> solution) {</div><div class="line"> <span class="keyword">int</span>[][] result = <span class="keyword">new</span> <span class="keyword">int</span>[solution.size()][count];</div><div class="line"> <span class="keyword">int</span> index = <span class="number">0</span>;</div><div class="line"> <span class="keyword">for</span> (<span class="keyword">List</span><Integer> <span class="keyword">list</span> : solution) {</div><div class="line"> result[index] = fromIntegerList(<span class="keyword">list</span>);</div><div class="line"> index++;</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> result;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">int</span>[] fromIntegerList(<span class="keyword">List</span><Integer> <span class="keyword">list</span>) {</div><div class="line"> Integer[] part = <span class="keyword">new</span> Integer[count];</div><div class="line"> part = <span class="keyword">list</span>.toArray(part);</div><div class="line"> <span class="keyword">int</span>[] result = <span class="keyword">new</span> <span class="keyword">int</span>[count];</div><div class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < part.length; i++) {</div><div class="line"> result[i] = part[i];</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> result;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> void println(<span class="keyword">int</span>[][] result) {</div><div class="line"> System.out.println();</div><div class="line"> System.out.println(<span class="string">"------- begin -------"</span>);</div><div class="line"> System.out.println(<span class="string">"------- count:"</span> + result.length + <span class="string">" -------"</span>);</div><div class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < result.length; i++) {</div><div class="line"> System.out.println((i + <span class="number">1</span>) + <span class="string">":"</span> + Arrays.toString(result[i]));</div><div class="line"> }</div><div class="line"> System.out.println(<span class="string">"------- end -------"</span>);</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="comment">/**</span></div><div class="line"><span class="comment"> * 运用DP思想来降低递归次数</span></div><div class="line"><span class="comment"> */</span></div><div class="line"> <span class="keyword">private</span> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> dynamicPrograming(<span class="keyword">int</span>[] <span class="keyword">array</span>, <span class="keyword">int</span> beginIndex, <span class="keyword">int</span> number, <span class="keyword">int</span> expectValue) {</div><div class="line"> <span class="keyword">if</span> (number == <span class="number">0</span> || beginIndex == <span class="keyword">array</span>.length) {</div><div class="line"> <span class="keyword">return</span> <span class="keyword">null</span>;</div><div class="line"> }</div><div class="line"> <span class="comment">//尝试从已知的答案中获取</span></div><div class="line"> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> existed = getExistedSolution(number, expectValue);</div><div class="line"> <span class="keyword">if</span> (existed != <span class="keyword">null</span> && !existed.isEmpty()) {</div><div class="line"> <span class="comment">//命中了，但有可能不符合beginIndex要求</span></div><div class="line"> <span class="comment">//校验命中项，是否符合要求</span></div><div class="line"> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> vailds = filterVailds(existed, <span class="keyword">array</span>, beginIndex);</div><div class="line"> <span class="keyword">if</span> (vailds != <span class="keyword">null</span> && !vailds.isEmpty()) {</div><div class="line"> <span class="keyword">return</span> vailds;</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> thisPartGroup = <span class="keyword">new</span> ArrayList<>();</div><div class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = beginIndex; i < <span class="keyword">array</span>.length; i++) {</div><div class="line"> <span class="keyword">int</span> currentValue = <span class="keyword">array</span>[i];</div><div class="line"> <span class="keyword">if</span> (number == <span class="number">1</span>) {<span class="comment">//最底层不再需要向下分解</span></div><div class="line"> <span class="keyword">if</span> (currentValue == expectValue) {</div><div class="line"> <span class="keyword">List</span><Integer> underlying = <span class="keyword">new</span> ArrayList<>();</div><div class="line"> underlying.add(currentValue);</div><div class="line"> thisPartGroup.add(underlying);</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"> }</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="comment">//将问题分解为同性质的子问题</span></div><div class="line"> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> nextPartGroup = dynamicPrograming(<span class="keyword">array</span>, i + <span class="number">1</span>, number - <span class="number">1</span>, expectValue - currentValue);</div><div class="line"> <span class="keyword">if</span> (nextPartGroup != <span class="keyword">null</span>) {</div><div class="line"> buildFullGroups(thisPartGroup, nextPartGroup, currentValue);</div><div class="line"> <span class="keyword">if</span> (number == count) {</div><div class="line"> <span class="comment">//找到了多组可行的解法</span></div><div class="line"> solution.addAll(thisPartGroup);</div><div class="line"> thisPartGroup.clear();</div><div class="line"> }</div><div class="line"> }</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> (number != count) {</div><div class="line"> <span class="comment">//如果不是最顶层，就将答案缓存起来</span></div><div class="line"> putExistedSolution(number, expectValue, thisPartGroup);</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> (thisPartGroup.isEmpty()) {</div><div class="line"> thisPartGroup = <span class="keyword">null</span>;</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> thisPartGroup;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> filterVailds(<span class="keyword">List</span><<span class="keyword">List</span><Integer>> existed, <span class="keyword">int</span>[] <span class="keyword">array</span>, <span class="keyword">int</span> beginIndex) {</div><div class="line"> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> noRepeats = <span class="keyword">new</span> ArrayList<>();</div><div class="line"> <span class="keyword">for</span> (<span class="keyword">List</span><Integer> <span class="keyword">list</span> : existed) {</div><div class="line"> <span class="keyword">boolean</span> isOutOfRange = <span class="keyword">false</span>;</div><div class="line"> <span class="keyword">for</span> (Integer key : <span class="keyword">list</span>) {</div><div class="line"> <span class="keyword">int</span> index = Arrays.binarySearch(<span class="keyword">array</span>, beginIndex, <span class="keyword">array</span>.length - <span class="number">1</span>, key);</div><div class="line"> <span class="keyword">if</span> (index < beginIndex) {</div><div class="line"> isOutOfRange = <span class="keyword">true</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">if</span> (!isOutOfRange) {</div><div class="line"> noRepeats.add(<span class="keyword">list</span>);</div><div class="line"> }</div><div class="line"> }</div><div class="line"> <span class="keyword">return</span> noRepeats;</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> void buildFullGroups(<span class="keyword">List</span><<span class="keyword">List</span><Integer>> thisPartGroup, <span class="keyword">List</span><<span class="keyword">List</span><Integer>> nextPartGroup, <span class="keyword">int</span> currentValue) {</div><div class="line"> <span class="keyword">for</span> (<span class="keyword">List</span><Integer> <span class="keyword">list</span> : nextPartGroup) {</div><div class="line"> <span class="keyword">List</span><Integer> resultList = <span class="keyword">new</span> ArrayList<>(<span class="keyword">list</span>);</div><div class="line"> resultList.add(currentValue);</div><div class="line"> Collections.sort(resultList);</div><div class="line"> <span class="keyword">if</span> (uniques.add(resultList.toString())) {</div><div class="line"> thisPartGroup.add(resultList);</div><div class="line"> }</div><div class="line"> }</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> void putExistedSolution(<span class="keyword">int</span> number, <span class="keyword">int</span> expectValue, <span class="keyword">List</span><<span class="keyword">List</span><Integer>> result) {</div><div class="line"> Map<Integer, <span class="keyword">List</span><<span class="keyword">List</span><Integer>>> expects = <span class="keyword">new</span> HashMap<>();</div><div class="line"> <span class="keyword">if</span> (existedSolution.containsKey(number)) {</div><div class="line"> expects = existedSolution.get(number);</div><div class="line"> }</div><div class="line"> expects.put(expectValue, result);</div><div class="line"> existedSolution.put(number, expects);</div><div class="line"> }</div><div class="line"></div><div class="line"> <span class="keyword">private</span> <span class="keyword">List</span><<span class="keyword">List</span><Integer>> getExistedSolution(<span class="keyword">int</span> number, <span class="keyword">int</span> expectValue) {</div><div class="line"> <span class="keyword">if</span> (existedSolution.containsKey(number)) {</div><div class="line"> Map<Integer, <span class="keyword">List</span><<span class="keyword">List</span><Integer>>> expects = existedSolution.get(number);</div><div class="line"> <span class="keyword">if</span> (expects.containsKey(expectValue)) {</div><div class="line"> <span class="keyword">return</span> expects.get(expectValue);</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="keyword">return</span> <span class="keyword">null</span>;</div><div class="line"> }</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="keyword">return</span> <span class="keyword">null</span>;</div><div class="line"> }</div><div class="line"> }</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line">以下是打印的结果：OK</div><div class="line"></div><div class="line">input:[<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">-1</span>, <span class="number">-2</span>, <span class="number">-3</span>, <span class="number">-4</span>, <span class="number">-5</span>]</div><div class="line">------- dynamicPrograming -------</div><div class="line"></div><div class="line">------- begin -------</div><div class="line">------- count:<span class="number">13</span> -------</div><div class="line"><span class="number">1</span>:[<span class="number">-1</span>, <span class="number">1</span>, <span class="number">0</span>]</div><div class="line"><span class="number">2</span>:[<span class="number">-2</span>, <span class="number">2</span>, <span class="number">0</span>]</div><div class="line"><span class="number">3</span>:[<span class="number">-3</span>, <span class="number">3</span>, <span class="number">0</span>]</div><div class="line"><span class="number">4</span>:[<span class="number">-4</span>, <span class="number">4</span>, <span class="number">0</span>]</div><div class="line"><span class="number">5</span>:[<span class="number">-5</span>, <span class="number">5</span>, <span class="number">0</span>]</div><div class="line"><span class="number">6</span>:[<span class="number">-3</span>, <span class="number">2</span>, <span class="number">1</span>]</div><div class="line"><span class="number">7</span>:[<span class="number">-4</span>, <span class="number">3</span>, <span class="number">1</span>]</div><div class="line"><span class="number">8</span>:[<span class="number">-5</span>, <span class="number">4</span>, <span class="number">1</span>]</div><div class="line"><span class="number">9</span>:[<span class="number">-5</span>, <span class="number">3</span>, <span class="number">2</span>]</div><div class="line"><span class="number">10</span>:[<span class="number">-2</span>, <span class="number">-1</span>, <span class="number">3</span>]</div><div class="line"><span class="number">11</span>:[<span class="number">-3</span>, <span class="number">-1</span>, <span class="number">4</span>]</div><div class="line"><span class="number">12</span>:[<span class="number">-4</span>, <span class="number">-1</span>, <span class="number">5</span>]</div><div class="line"><span class="number">13</span>:[<span class="number">-3</span>, <span class="number">-2</span>, <span class="number">5</span>]</div><div class="line">------- end -------</div></pre></td></tr></table></figure>
<p>尝试优化解法的过程中，让我获益良多，尤其是对于动态规划思想理解的更深刻。在对动态规划解法的研究中，遇到了二维数组去重复的问题，例如二维数组：[[1,-1,0],[0,-1,1]]，这是重复的两个组合，需要去除重复，当时还想到了利用先挨个排好序，再用KMP算法（更简单的利用字符串来equals）来做模式匹配，后来想到了一个巧妙的解法，用beginIndex值来进行判定（filterVailds）就可以了。</p>
]]></content>
<categories>
<category> LeetCode </category>
</categories>
<tags>
<tag> LeetCode </tag>
<tag> Training </tag>
</tags>
</entry>
<entry>
<title><![CDATA[如何更开心的写博客？]]></title>
<url>https://amao12580.github.io/post/2016/06/How-to-better-write-the-blog/</url>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>不知不觉中，在GitHub Pages写博客快有半年了，工作一直比较忙，但还是想在闲暇更新。既想锻炼自己的文字能力，也反向促使自己思考更多，博客愈发繁茂时，读者也多了，这更给了我继续下去的动力。Keep Moving！习惯了利用碎片化的时间来更新blog，但是回头想想，每次发布blog的过程是比较繁琐的。如何解放自己，专注于写文章呢？现在就让我们一起好好利用Powershell脚本吧！</p>
<p>最开始主要依靠图形界面进行blog发布，来看看有哪些操作吧！</p>
<figure class="highlight stylus"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div></pre></td><td class="code"><pre><div class="line">在windows DOS下的步骤是：</div><div class="line"></div><div class="line"><span class="number">1</span>.按下“win”+“R”键</div><div class="line"></div><div class="line"><span class="number">2</span>.在打开的对话框中输入“cmd”</div><div class="line"></div><div class="line"><span class="number">3</span>.在打开的黑窗口中输入：</div><div class="line">Microsoft Windows [版本 <span class="number">6.1</span>.<span class="number">7601</span>]</div><div class="line">版权所有 (c) <span class="number">2009</span> Microsoft Corporation。保留所有权利。</div><div class="line"></div><div class="line">C:\Users\Administrator>D:</div><div class="line"></div><div class="line">D:\>cd GitHub</div><div class="line"></div><div class="line">D:\GitHub>cd Hexo</div><div class="line"></div><div class="line">D:\GitHub\Hexo>hexo g</div><div class="line">INFO Start processing</div><div class="line">INFO Files loaded <span class="keyword">in</span> <span class="number">2.14</span> s</div><div class="line">INFO Generated: atom.xml</div><div class="line">INFO Generated: search.xml</div><div class="line">INFO Generated: sitemap.xml</div><div class="line">INFO Generated: about/index.html</div><div class="line">INFO Generated: tags/index.html</div><div class="line">INFO Generated: categories/index.html</div><div class="line">INFO Generated: post/<span class="number">2016</span>/<span class="number">06</span>/What-is-a-transaction/index.html</div><div class="line">.........<span class="comment">//都是一些HTML Generated信息，就不贴出了。</span></div><div class="line"></div><div class="line"></div><div class="line">INFO <span class="number">119</span> files generated <span class="keyword">in</span> <span class="number">4.33</span> s</div><div class="line"></div><div class="line"><span class="number">3</span>.清理不必要的文件：将文件夹D:\GitHub\Hexo\public\vendors删除</div><div class="line"></div><div class="line"><span class="number">4</span>.将D:\GitHub\Hexo\public文件夹拷贝到发布文件夹D:\GitHub\amao12580<span class="selector-class">.github</span><span class="selector-class">.io</span></div><div class="line"></div><div class="line"><span class="number">5</span>.使用GitHub For Windows可视化工具进行提交和同步。</div></pre></td></tr></table></figure>
<p>整个过程大约需要10分钟，因为很多都在图形化界面中操作，感觉效率很低。起初的想法是在DOS下做一个批处理，以下是我之前实际在用的命令。</p>
<figure class="highlight taggerscript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">//一整行实在不美观，折行让大家看得舒服点！</div><div class="line"></div><div class="line">cd D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo & hexo clean & hexo g</div><div class="line">& rd D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo<span class="symbol">\p</span>ublic<span class="symbol">\v</span>endors /S /Q</div><div class="line">& gulp</div><div class="line">& XCOPY D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo<span class="symbol">\p</span>ublic<span class="symbol">\*</span>.* D:<span class="symbol">\G</span>itHub<span class="symbol">\a</span>mao12580.github.io /Q /D /E /C /Y</div><div class="line">& hexo s</div></pre></td></tr></table></figure>
<p>如你所想，还差了最后向github提交呢？并且还不够自动化，需要手动在DOS触发，这明显不符合极客范啊！</p>
<h1 id="gulp"><a href="#gulp" class="headerlink" title="gulp"></a>gulp</h1><p>Gulp是一个构建系统，它能通过自动执行常见任务，比如编译预处理CSS，压缩JavaScript和刷新浏览器，来改进网站开发的过程。我主要使用gulp来压缩CSS、HTML、javascript、images。加速blog的访问速度，效果还是很好的！在这里贴一下我在用的gulpfile.js，放在D:\GitHub\Hexo目录下就可以了。还是很好使的，直接给压缩了20%体积。</p>
<p>目前访问速度还算可以了，想进一步优化，需要自建VPS，用Nginx合并一些请求（<a href="https://github.com/alibaba/nginx-http-concat" target="_blank" rel="external">nginx-http-concat</a> 模块），别忘了Nginx还可以缓存静态资源哦！</p>
<figure class="highlight javascript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> gulp = <span class="built_in">require</span>(<span class="string">'gulp'</span>);</div><div class="line"><span class="comment">//html压缩</span></div><div class="line"><span class="keyword">var</span> htmlmin = <span class="built_in">require</span>(<span class="string">'gulp-htmlmin'</span>);</div><div class="line"><span class="keyword">var</span> htmlclean = <span class="built_in">require</span>(<span class="string">'gulp-htmlclean'</span>);</div><div class="line"><span class="comment">//js压缩</span></div><div class="line"><span class="keyword">var</span> jsmin = <span class="built_in">require</span>(<span class="string">'gulp-jsmin'</span>);</div><div class="line"><span class="keyword">var</span> uglify = <span class="built_in">require</span>(<span class="string">'gulp-uglify'</span>);</div><div class="line"><span class="comment">//文件重命名</span></div><div class="line"><span class="keyword">var</span> rename = <span class="built_in">require</span>(<span class="string">'gulp-rename'</span>);</div><div class="line"><span class="comment">//图片压缩png/jpg/gif</span></div><div class="line"><span class="keyword">var</span> imagemin = <span class="built_in">require</span>(<span class="string">'gulp-imagemin'</span>);</div><div class="line"><span class="comment">//png压缩</span></div><div class="line"><span class="keyword">var</span> pngquant = <span class="built_in">require</span>(<span class="string">'imagemin-pngquant'</span>);</div><div class="line"><span class="keyword">var</span> cache = <span class="built_in">require</span>(<span class="string">'gulp-cache'</span>);</div><div class="line"><span class="comment">//css压缩</span></div><div class="line"><span class="keyword">var</span> csso = <span class="built_in">require</span>(<span class="string">'gulp-csso'</span>);</div><div class="line"></div><div class="line"><span class="keyword">var</span> root = <span class="string">"./public"</span>;</div><div class="line"><span class="keyword">var</span> buildDir = root;</div><div class="line"><span class="keyword">var</span> datas={</div><div class="line"> html:[root+<span class="string">"/**/*.html"</span>],</div><div class="line"> image:[root+<span class="string">"/**/*.{png,jpg,jpeg,gif,ico}"</span>],</div><div class="line"> css:[root+<span class="string">"/**/*.css"</span>],</div><div class="line"> js:[root+<span class="string">"/**/*.js"</span>,<span class="string">'!*min.js'</span>]</div><div class="line">}</div><div class="line"><span class="comment">// 压缩html</span></div><div class="line">gulp.task(<span class="string">'htmlmin'</span>, <span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</div><div class="line"> <span class="keyword">var</span> options = {</div><div class="line"> removeComments: <span class="literal">true</span>,<span class="comment">//清除HTML注释</span></div><div class="line"> collapseWhitespace: <span class="literal">true</span>,<span class="comment">//压缩HTML</span></div><div class="line"> collapseBooleanAttributes: <span class="literal">true</span>,<span class="comment">//省略布尔属性的值 <input checked="true"/> ==> <input /></span></div><div class="line"> removeEmptyAttributes: <span class="literal">true</span>,<span class="comment">//删除所有空格作属性值 <input id="" /> ==> <input /></span></div><div class="line"> removeScriptTypeAttributes: <span class="literal">true</span>,<span class="comment">//删除<script>的type="text/javascript"</span></div><div class="line"> removeStyleLinkTypeAttributes: <span class="literal">true</span>,<span class="comment">//删除<style>和<link>的type="text/css"</span></div><div class="line"> minifyJS: <span class="literal">true</span>,<span class="comment">//压缩页面JS</span></div><div class="line"> minifyCSS: <span class="literal">true</span><span class="comment">//压缩页面CSS</span></div><div class="line"> };</div><div class="line"> gulp.src(datas.html)</div><div class="line"> .pipe(htmlclean())</div><div class="line"> .pipe(htmlmin(options).on(<span class="string">'error'</span>, <span class="function"><span class="keyword">function</span>(<span class="params">e</span>)</span>{</div><div class="line"> <span class="built_in">console</span>.log(e);</div><div class="line"> }))</div><div class="line"> .pipe(gulp.dest(buildDir));</div><div class="line">});</div><div class="line"></div><div class="line"><span class="comment">// png图片压缩</span></div><div class="line">gulp.task(<span class="string">"imagemin"</span>,<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</div><div class="line"> gulp.src(datas.image)</div><div class="line"> .pipe(imagemin({</div><div class="line"> optimizationLevel: <span class="number">7</span>, <span class="comment">//类型：Number 默认：3 取值范围：0-7（优化等级）</span></div><div class="line"> progressive: <span class="literal">true</span>, <span class="comment">//类型：Boolean 默认：false 无损压缩jpg图片</span></div><div class="line"> interlaced: <span class="literal">true</span>, <span class="comment">//类型：Boolean 默认：false 隔行扫描gif进行渲染</span></div><div class="line"> multipass: <span class="literal">true</span>, <span class="comment">//类型：Boolean 默认：false 多次优化svg直到完全优化</span></div><div class="line"> svgoPlugins:[{<span class="attr">removeViewBox</span>:<span class="literal">false</span>}],</div><div class="line"> use:[pngquant()] <span class="comment">//压缩率64%</span></div><div class="line"> }).on(<span class="string">'error'</span>, <span class="function"><span class="keyword">function</span>(<span class="params">e</span>)</span>{</div><div class="line"> <span class="built_in">console</span>.log(e);</div><div class="line"> }))</div><div class="line"> .pipe(gulp.dest(buildDir));</div><div class="line">});</div><div class="line"><span class="comment">// js压缩</span></div><div class="line">gulp.task(<span class="string">"jsmin"</span>,<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</div><div class="line"> gulp.src(datas.js)</div><div class="line"> .pipe(uglify().on(<span class="string">'error'</span>, <span class="function"><span class="keyword">function</span>(<span class="params">e</span>)</span>{</div><div class="line"> <span class="built_in">console</span>.log(e);</div><div class="line"> }))</div><div class="line"> <span class="comment">//.pipe(rename({suffix:'.min'}))</span></div><div class="line"> .pipe(gulp.dest(buildDir));</div><div class="line">});</div><div class="line"><span class="comment">// css压缩</span></div><div class="line">gulp.task(<span class="string">"cssmin"</span>,<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</div><div class="line"> gulp.src(datas.css)</div><div class="line"> .pipe(csso().on(<span class="string">'error'</span>, <span class="function"><span class="keyword">function</span>(<span class="params">e</span>)</span>{</div><div class="line"> <span class="built_in">console</span>.log(e);</div><div class="line"> }))</div><div class="line"> .pipe(gulp.dest(buildDir));</div><div class="line">});</div><div class="line">gulp.task(<span class="string">"default"</span>,[<span class="string">"htmlmin"</span>,<span class="string">"imagemin"</span>,<span class="string">"jsmin"</span>,<span class="string">"cssmin"</span>]);</div></pre></td></tr></table></figure>
<h1 id="Powershell"><a href="#Powershell" class="headerlink" title="Powershell"></a>Powershell</h1><p>想起了windows 7环境已经支持powershell（增强版DOS！），试想如果有一个ps1脚本可以完成发布blog，再配合windows计划任务，不就可以定时或按用户动作触发了吗？YES，这才是我想要的，不啰嗦，开始干吧！</p>
<p>PS：脚本运行之前，别忘了装一个：Git-2.9.0-64-bit.exe，然后配置好自己的github信息。<br><figure class="highlight ada"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">配置完成后的验证：ssh -T git@github.com</div><div class="line">如果提示：Hi *** You<span class="symbol">'ve</span> successfully authenticated, but GitHub does <span class="keyword">not</span> provide shell <span class="keyword">access</span>. 说明你连接成功了</div></pre></td></tr></table></figure></p>
<p>设置PowerShell环境，使能“allow scripts to run”选项，步骤如下：<br>以管理员的身份运行PowerShell<br>执行Set-ExecutionPolicy RemoteSigned命令，在对话框中选择Y，如下<br><figure class="highlight less"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="selector-tag">PS</span> <span class="selector-tag">C</span>:\<span class="selector-tag">Windows</span>\<span class="selector-tag">system32</span>> <span class="selector-tag">Set-ExecutionPolicy</span> <span class="selector-tag">RemoteSigned</span></div><div class="line">执行策略更改</div><div class="line">执行策略可帮助你防止执行不信任的脚本。更改执行策略可能会产生安全风险，如 <span class="selector-tag">http</span>:<span class="comment">//go.microsoft.com/fwlink/?LinkID=135170</span></div><div class="line">中的 <span class="selector-tag">about_Execution_Policies</span> 帮助主题所述。是否要更改执行策略?</div><div class="line"><span class="selector-attr">[Y]</span> 是(Y) <span class="selector-attr">[N]</span> 否(N) <span class="selector-attr">[S]</span> 挂起(S) <span class="selector-attr">[?]</span> 帮助 (默认值为“Y”): <span class="selector-tag">Y</span></div><div class="line"><span class="selector-tag">PS</span> <span class="selector-tag">C</span>:\<span class="selector-tag">Windows</span>\<span class="selector-tag">system32</span>></div></pre></td></tr></table></figure></p>
<p>Finally，一起来看看我的“syncArticle.ps1”，遇到无数坑（语句块是大坑！），不过还是被我搞定了！</p>
<p>‘D:\Program Files\Git\bin\git.exe’是我的Git安装目录。</p>
<figure class="highlight taggerscript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">cd D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo</div><div class="line">hexo g --silent</div><div class="line">Remove-Item D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo<span class="symbol">\p</span>ublic<span class="symbol">\v</span>endors<span class="symbol">\*</span> -recurse</div><div class="line">Remove-Item D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo<span class="symbol">\p</span>ublic<span class="symbol">\v</span>endors -recurse</div><div class="line">gulp --silent</div><div class="line">XCOPY D:<span class="symbol">\G</span>itHub<span class="symbol">\H</span>exo<span class="symbol">\p</span>ublic<span class="symbol">\*</span>.* D:<span class="symbol">\G</span>itHub<span class="symbol">\a</span>mao12580.github.io /Q /D /E /C /Y</div><div class="line">cd D:<span class="symbol">\G</span>itHub<span class="symbol">\a</span>mao12580.github.io</div><div class="line">& 'D:<span class="symbol">\P</span>rogram Files<span class="symbol">\G</span>it<span class="symbol">\b</span>in<span class="symbol">\g</span>it.exe' add .</div><div class="line">& 'D:<span class="symbol">\P</span>rogram Files<span class="symbol">\G</span>it<span class="symbol">\b</span>in<span class="symbol">\g</span>it.exe' commit -m "Auto commit by powershell script."</div><div class="line">& 'D:<span class="symbol">\P</span>rogram Files<span class="symbol">\G</span>it<span class="symbol">\b</span>in<span class="symbol">\g</span>it.exe' push -f origin master</div><div class="line">exit</div></pre></td></tr></table></figure>
<p>在windows计划任务管理，我已经配置好，每晚10点，自动发布有更新的blog啦！来看看导出的windows计划任务XML文件吧！</p>
<p>PC-201402281156\Administrator，这是我的主机名和用户名。</p>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div></pre></td><td class="code"><pre><div class="line"></div><div class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-16"</span><span class="meta">?></span></span></div><div class="line"><span class="tag"><<span class="name">Task</span> <span class="attr">version</span>=<span class="string">"1.2"</span> <span class="attr">xmlns</span>=<span class="string">"http://schemas.microsoft.com/windows/2004/02/mit/task"</span>></span></div><div class="line"> <span class="tag"><<span class="name">RegistrationInfo</span>></span></div><div class="line"> <span class="tag"><<span class="name">Date</span>></span>2016-06-15T17:08:16.1670287<span class="tag"></<span class="name">Date</span>></span></div><div class="line"> <span class="tag"><<span class="name">Author</span>></span>PC-201402281156\Administrator<span class="tag"></<span class="name">Author</span>></span></div><div class="line"> <span class="tag"></<span class="name">RegistrationInfo</span>></span></div><div class="line"> <span class="tag"><<span class="name">Triggers</span>></span></div><div class="line"> <span class="tag"><<span class="name">CalendarTrigger</span>></span></div><div class="line"> <span class="tag"><<span class="name">StartBoundary</span>></span>2016-06-15T22:00:00<span class="tag"></<span class="name">StartBoundary</span>></span></div><div class="line"> <span class="tag"><<span class="name">Enabled</span>></span>true<span class="tag"></<span class="name">Enabled</span>></span></div><div class="line"> <span class="tag"><<span class="name">ScheduleByDay</span>></span></div><div class="line"> <span class="tag"><<span class="name">DaysInterval</span>></span>1<span class="tag"></<span class="name">DaysInterval</span>></span></div><div class="line"> <span class="tag"></<span class="name">ScheduleByDay</span>></span></div><div class="line"> <span class="tag"></<span class="name">CalendarTrigger</span>></span></div><div class="line"> <span class="tag"></<span class="name">Triggers</span>></span></div><div class="line"> <span class="tag"><<span class="name">Principals</span>></span></div><div class="line"> <span class="tag"><<span class="name">Principal</span> <span class="attr">id</span>=<span class="string">"Author"</span>></span></div><div class="line"> <span class="tag"><<span class="name">UserId</span>></span>PC-201402281156\Administrator<span class="tag"></<span class="name">UserId</span>></span></div><div class="line"> <span class="tag"><<span class="name">LogonType</span>></span>InteractiveToken<span class="tag"></<span class="name">LogonType</span>></span></div><div class="line"> <span class="tag"><<span class="name">RunLevel</span>></span>LeastPrivilege<span class="tag"></<span class="name">RunLevel</span>></span></div><div class="line"> <span class="tag"></<span class="name">Principal</span>></span></div><div class="line"> <span class="tag"></<span class="name">Principals</span>></span></div><div class="line"> <span class="tag"><<span class="name">Settings</span>></span></div><div class="line"> <span class="tag"><<span class="name">MultipleInstancesPolicy</span>></span>IgnoreNew<span class="tag"></<span class="name">MultipleInstancesPolicy</span>></span></div><div class="line"> <span class="tag"><<span class="name">DisallowStartIfOnBatteries</span>></span>false<span class="tag"></<span class="name">DisallowStartIfOnBatteries</span>></span></div><div class="line"> <span class="tag"><<span class="name">StopIfGoingOnBatteries</span>></span>true<span class="tag"></<span class="name">StopIfGoingOnBatteries</span>></span></div><div class="line"> <span class="tag"><<span class="name">AllowHardTerminate</span>></span>true<span class="tag"></<span class="name">AllowHardTerminate</span>></span></div><div class="line"> <span class="tag"><<span class="name">StartWhenAvailable</span>></span>false<span class="tag"></<span class="name">StartWhenAvailable</span>></span></div><div class="line"> <span class="tag"><<span class="name">RunOnlyIfNetworkAvailable</span>></span>false<span class="tag"></<span class="name">RunOnlyIfNetworkAvailable</span>></span></div><div class="line"> <span class="tag"><<span class="name">IdleSettings</span>></span></div><div class="line"> <span class="tag"><<span class="name">StopOnIdleEnd</span>></span>true<span class="tag"></<span class="name">StopOnIdleEnd</span>></span></div><div class="line"> <span class="tag"><<span class="name">RestartOnIdle</span>></span>false<span class="tag"></<span class="name">RestartOnIdle</span>></span></div><div class="line"> <span class="tag"></<span class="name">IdleSettings</span>></span></div><div class="line"> <span class="tag"><<span class="name">AllowStartOnDemand</span>></span>true<span class="tag"></<span class="name">AllowStartOnDemand</span>></span></div><div class="line"> <span class="tag"><<span class="name">Enabled</span>></span>true<span class="tag"></<span class="name">Enabled</span>></span></div><div class="line"> <span class="tag"><<span class="name">Hidden</span>></span>false<span class="tag"></<span class="name">Hidden</span>></span></div><div class="line"> <span class="tag"><<span class="name">RunOnlyIfIdle</span>></span>false<span class="tag"></<span class="name">RunOnlyIfIdle</span>></span></div><div class="line"> <span class="tag"><<span class="name">WakeToRun</span>></span>false<span class="tag"></<span class="name">WakeToRun</span>></span></div><div class="line"> <span class="tag"><<span class="name">ExecutionTimeLimit</span>></span>PT10M<span class="tag"></<span class="name">ExecutionTimeLimit</span>></span></div><div class="line"> <span class="tag"><<span class="name">Priority</span>></span>7<span class="tag"></<span class="name">Priority</span>></span></div><div class="line"> <span class="tag"></<span class="name">Settings</span>></span></div><div class="line"> <span class="tag"><<span class="name">Actions</span> <span class="attr">Context</span>=<span class="string">"Author"</span>></span></div><div class="line"> <span class="tag"><<span class="name">Exec</span>></span></div><div class="line"> <span class="tag"><<span class="name">Command</span>></span>powershell<span class="tag"></<span class="name">Command</span>></span></div><div class="line"> <span class="tag"><<span class="name">Arguments</span>></span>D:\GitHub\syncArticle.ps1<span class="tag"></<span class="name">Arguments</span>></span></div><div class="line"> <span class="tag"><<span class="name">WorkingDirectory</span>></span>D:\GitHub<span class="tag"></<span class="name">WorkingDirectory</span>></span></div><div class="line"> <span class="tag"></<span class="name">Exec</span>></span></div><div class="line"> <span class="tag"></<span class="name">Actions</span>></span></div><div class="line"><span class="tag"></<span class="name">Task</span>></span></div></pre></td></tr></table></figure>
<p>如下图所示，用户可以手动触发计划任务，来看看运行结果吧！点击历史记录还可以查看每次任务计划的运行情况呢！</p>
<p><img src="/img/windowsPlanTask.png" alt=""></p>
<p>现在我的blog就可以每晚10点自动发布到github了，终于可以专注于写文章了！</p>
]]></content>
<categories>
<category> Tutorial </category>
</categories>
<tags>
<tag> GitHub </tag>
<tag> Git </tag>
<tag> Windows </tag>
<tag> Powershell </tag>
<tag> Gulp </tag>
</tags>
</entry>
<entry>
<title><![CDATA[消息系统的架构演进之路]]></title>
<url>https://amao12580.github.io/post/2016/06/Message-system-architecture-evolution/</url>
<content type="html"><![CDATA[<p>暂时未完成</p>
<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><h1 id="1-0版架构的问题"><a href="#1-0版架构的问题" class="headerlink" title="1.0版架构的问题"></a>1.0版架构的问题</h1><h2 id="详细说明"><a href="#详细说明" class="headerlink" title="详细说明"></a>详细说明</h2><h2 id="缺陷"><a href="#缺陷" class="headerlink" title="缺陷"></a>缺陷</h2><h2 id="优势"><a href="#优势" class="headerlink" title="优势"></a>优势</h2><h1 id="2-0版如何改进？"><a href="#2-0版如何改进？" class="headerlink" title="2.0版如何改进？"></a>2.0版如何改进？</h1><h2 id="详细说明-1"><a href="#详细说明-1" class="headerlink" title="详细说明"></a>详细说明</h2><h2 id="缺陷-1"><a href="#缺陷-1" class="headerlink" title="缺陷"></a>缺陷</h2><h2 id="优势-1"><a href="#优势-1" class="headerlink" title="优势"></a>优势</h2><h1 id="展望3-0版"><a href="#展望3-0版" class="headerlink" title="展望3.0版"></a>展望3.0版</h1><h2 id="详细说明-2"><a href="#详细说明-2" class="headerlink" title="详细说明"></a>详细说明</h2><h2 id="缺陷-2"><a href="#缺陷-2" class="headerlink" title="缺陷"></a>缺陷</h2><h2 id="优势-2"><a href="#优势-2" class="headerlink" title="优势"></a>优势</h2><p><img src="/img/03-消息系统改进1阶段的总体结构.jpg" alt=""></p>
]]></content>
<categories>
<category> Summary </category>
</categories>
<tags>
<tag> MessageSystem </tag>
<tag> architecture </tag>
</tags>
</entry>
<entry>
<title><![CDATA[当我们谈事务时，我们在谈什么？]]></title>
<url>https://amao12580.github.io/post/2016/06/What-is-a-transaction/</url>
<content type="html"><![CDATA[<h2 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h2><p>写这篇文章的初衷是记录和总结关于事务的方方面面，还记得以前在面试高工时，被问到数据库事务的实现原理是什么？答了读写锁和表锁，以及myisam和innodb引擎在事务隔离级别的差异。当时面试官不是很满意，自然那家公司最终也没有谈成。面试完了，这个问题却一直萦绕在脑海中挥之不去，经过自己对存储引擎知识的补全，才知道原来事务是这样的妙不可言。如果读者需要对数据库事务进行系统的学习，我推荐一本书<a href="https://www.amazon.cn/MySQL%E6%8A%80%E6%9C%AF%E5%86%85%E5%B9%95-InnoDB%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E-%E5%A7%9C%E6%89%BF%E5%B0%A7/dp/B00ETOV48K/ref=sr_1_1?s=digital-text&ie=UTF8&qid=1465707176&sr=1-1&keywords=MySQL%E6%8A%80%E6%9C%AF%E5%86%85%E5%B9%95%EF%BC%9AInnoDB%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E" target="_blank" rel="external">《MySQL技术内幕 InnoDB存储引擎 第2版》</a>。</p>
<p>讨论传统关系型数据库内的事务，以及如何与NoSQL领域结合进行事务化。</p>
<h2 id="什么是事务？"><a href="#什么是事务？" class="headerlink" title="什么是事务？"></a>什么是事务？</h2><p>数据库事务（简称：事务）是数据库管理系统执行过程中的一个逻辑单位，由一个有限的数据库操作序列构成。这是使用Google Search以“事务”为关键字查找时，第一条是wikipedia给出的描述。通俗地讲，事务就是通过一系列操作来完成一件事情，在进行这些操作的过程中，要么这些操作完全执行，要么这些操作全不执行，不存在中间状态，事务分为事务执行阶段和事务提交阶段。</p>
<h3 id="与文件系统的差异"><a href="#与文件系统的差异" class="headerlink" title="与文件系统的差异"></a>与文件系统的差异</h3><p>以MySQL数据库系统为例，数据被按页（16K）存储在磁盘中，如何组织数据的存储格式，由存储引擎来定义。在数据被插入或修改时，为了提高TPS，规避磁盘写入缓慢的问题，MySQL会将新数据页缓冲到内存（Insert Buffer），达到一定时间，或者积累的数据达到一定量级后，会由Master Thread刷入磁盘，同时清空缓冲区（CheckPoint机制）。在数据需要读取时，先从磁盘按页读取，如果有配置缓冲池，这些取出的数据还会在内存中留一个副本（FIX），只要数据不进行修改，下次读取时就会冲缓冲池命中了，大大加快了访问速度。</p>
<p>一般存储引擎都是采用 btree 或者 lsm tree 来实现索引，但是索引的最小单位不是 K/V 记录对象，而是数据页，数据页的组织关系实现就是存储引擎的数据组织方式。</p>
<p>传统数据库引擎大都是设计一个磁盘和内存完全一样的数据组织方式,这个结构是固定的空间大小（innodb 的 page 是 16KB）,访问它必须遵守严格的 The FIX Rules 规则：1.<br>修改一个 page 需要获得该页的 x-latch lock，2.访问一个 page 需要获得该页的 s-latch lock 或者 x-latch lock。3.持有该 page 的 latch 直到修改或者访问该页的操作完成 latch unlock。</p>
<p>看到这里，我们不经有疑问，在内存与磁盘上，读、写、组织数据，这不是文件系统在干的事情吗？的确，在NTFS、Ext4等文件系统中，这些功能都是被支持的。数据库系统是基于文件系统（Base-on-disk）的，按照不同文件系统(包括不同的操作系统内存分配算法)，在底层调用了不同的API进行数据的读、写、组织。但它最重要的特性是事务的支持，允许有限个的数据库操作进行逻辑单元化，解决文件系统访问无状态的问题。同时也是由于事务一致性的支持，数据库系统相较于文件系统的性能普遍是下降的，隔离级别的支持越高（更强的一致性），性能下降越是厉害。</p>
<h2 id="本地事务"><a href="#本地事务" class="headerlink" title="本地事务"></a>本地事务</h2><p>不同的数据库系统有不同的事务行为。有些数据库系统根本不支持事务。有些数据库系统支持事务，但是不支持双向提交（2PC）协议。这类事务被称为支持本地事务。有些数据库系统既支持本地事务，又支持 2PC。这类事务被称为支持分布式事务，或者全局事务。全局事务也被称为 XA 事务，因为它们包含 XAResource接口。</p>
<h3 id="ACID"><a href="#ACID" class="headerlink" title="ACID"></a>ACID</h3><p>一般说到事务，就会想到它的特性— ACID，那么什么是 ACID 呢？我们先用一个现实中的例子来说明：AB 两同学在同一家银行ZSBANK的账号都有 1,000 块钱，A 通过ZSBANK银行转账向 B 转了 100 块钱，这个事务分为两个操作,即从 A 同学账号扣除 100，向 B 同学账号增加 100。</p>
<p>对于应用层程序的同一个线程X来说，逻辑伪码如下：</p>
<figure class="highlight cs"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div></pre></td><td class="code"><pre><div class="line"><span class="comment">//定义转账金额</span></div><div class="line"><span class="keyword">double</span> transferAmount=<span class="number">100.00</span>;</div><div class="line"></div><div class="line"><span class="comment">//1.开始事务</span></div><div class="line"></div><div class="line"><span class="comment">//2.检查A同学账户余额是否大于100元</span></div><div class="line"><span class="keyword">double</span> balanceForA=<span class="keyword">select</span> account.balance <span class="keyword">from</span> account <span class="keyword">where</span> uid=A;</div><div class="line"><span class="keyword">if</span>(transferAmount>balanceForA){</div><div class="line"> <span class="keyword">return</span> <span class="keyword">new</span> ErrorResult(<span class="number">100</span>,<span class="string">"账户余额不足."</span>);</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//3.将A同学账户扣款100元</span></div><div class="line"><span class="keyword">int</span> deductRet=update account <span class="keyword">set</span> balance=balance-transferAmount <span class="keyword">where</span> uid=A and balance >=transferAmount;</div><div class="line"><span class="keyword">if</span>(deductRet==<span class="number">1</span>){</div><div class="line"> LogHelper.info(<span class="string">"account A deduct "</span>+transferAmount+<span class="string">" success."</span>);</div><div class="line">}<span class="keyword">else</span>{</div><div class="line"> <span class="keyword">return</span> <span class="keyword">new</span> ErrorResult(<span class="number">200</span>,<span class="string">"账户扣款失败."</span>);</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//4.将B同学账户增加100元</span></div><div class="line"><span class="keyword">int</span> deductRet=update account <span class="keyword">set</span> balance=balance+transferAmount <span class="keyword">where</span> uid=B;</div><div class="line"><span class="keyword">if</span>(deductRet==<span class="number">1</span>){</div><div class="line"> LogHelper.info(<span class="string">"account B add "</span>+transferAmount+<span class="string">" success."</span>);</div><div class="line">}<span class="keyword">else</span>{</div><div class="line"> <span class="keyword">return</span> <span class="keyword">new</span> ErrorResult(<span class="number">300</span>,<span class="string">"账户增加余额失败."</span>);</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">//5.提交事务</span></div><div class="line"></div><div class="line"><span class="comment">//6.如果遭遇数据库异常(SQLException)，回滚事务</span></div><div class="line"></div><div class="line"><span class="comment">//7.返回处理结果</span></div></pre></td></tr></table></figure>
<p>那在没有事务的情况下会发生什么呢？在步骤1中，同时有2个线程X、Y，线程X先执行查询，发现余额充足，可以扣款，还没有进行步骤2时，Y此时将款项先扣除了，导致X线程的步骤2失败。其他的，在执行的任一阶段，都有可能遭遇不可抗力因素，比如，执行完步骤2，还未执行步骤3时，适逢操作系统crash或断电、存储介质失败等情况，此时没有事务的一致性保证，在系统恢复时将无法回滚，导致数据不一致。</p>
<p>Lost update：<br>两个事务都同时更新一行数据，但是第二个事务却中途失败退出，导致对数据的两个修改都失效了。</p>
<p>Dirty Reads：<br>一个事务开始读取了某行数据，但是另外一个事务已经更新了此数据但没有能够及时提交。这是相当危险的，因为很可能所有的操作都被回滚。</p>
<p>Non-repeatable Reads：<br>一个事务对同一行数据重复读取两次，但是却得到了不同的结果。</p>
<p>Second lost updates problem：<br>无法重复读取的特例。有两个并发事务同时读取同一行数据，然后其中一个对它进行修改提交，而另一个也进行了修改提交。这就会造成第一次写操作失效。</p>
<p>Phantom Reads：<br>事务在操作过程中进行两次查询，第二次查询的结果包含了第一次查询中未出现的数据（这里并不要求两次查询的SQL语句相同）。这是因为在两次查询过程中有另外一个事务插入数据造成的。</p>
<h4 id="原子性（Atomicity）"><a href="#原子性（Atomicity）" class="headerlink" title="原子性（Atomicity）"></a>原子性（Atomicity）</h4><p>不可拆分，组成事务的系列操作是一个整体，要么全执行，要么不执行，不允许部分执行。通过上面例子就是从 A 同学扣除钱和向 B 同学增加 100 是一起发生的，不可能出现扣除了 A 的钱，但没增加 B 的钱的情况。</p>
<h4 id="一致性（Consistency）"><a href="#一致性（Consistency）" class="headerlink" title="一致性（Consistency）"></a>一致性（Consistency）</h4><p>一致性指的是语义上的一致性，即业务逻辑层面的一致。在事务开始之前和事务结束以后，数据库的完整性和状态没有被破坏，而在事务执行阶段，一致性是会被破坏的。这个怎么理解呢？就是 A、B 两人在转账钱的总和是 2,000，转账后两人的总和也必须是 2,000。不会因为这次转账事务破坏这个状态。如果帐户A上的钱减少了，而帐户B上的钱却没有增加(如在执行步骤3时失败)，那么我们认为此时数据处于不一致的状态。</p>
<p>在事务处理的ACID属性中，一致性是最基本的属性，其它的三个属性都为了保证一致性而存在的。MySQL数据库innodb的事务，是通过redo log（innodb log)，undo log，锁机制，来维护这个一致性的。</p>
<h4 id="隔离性（Isolation）"><a href="#隔离性（Isolation）" class="headerlink" title="隔离性（Isolation）"></a>隔离性（Isolation）</h4><p>多个事务在并发执行时，事务执行的中间状态是其他事务不可访问的。A 转出 100 但事务没有确认提交，这时候银行人员对其账号查询时，看到的应该还是 1,000，不是 900。</p>
<h4 id="持久性（Durability）"><a href="#持久性（Durability）" class="headerlink" title="持久性（Durability）"></a>持久性（Durability）</h4><p>事务一旦提交生效，其结果将永久保存，不受任何故障影响。A 转账一但完成，那么 A 就是 900，B 就是 1,100，这个结果将永远保存在银行的数据库中，直到他们下次交易事务的发生。</p>
<h3 id="4种Log"><a href="#4种Log" class="headerlink" title="4种Log"></a>4种Log</h3><p>4种Log指的是：redo log、undo log、bin log 、relay log</p>
<p>日志在内存里也是有缓存的，这里将其叫做log buffer。磁盘上的日志文件称为log file。log file一般是追加内容，可以认为是顺序写，顺序写的磁盘IO开销要小于随机写。</p>
<p>Undo log记录某数据被修改前的值，可以用来在事务失败时进行rollback；redo log记录某数据块被修改后的值（按页记录），可以用来恢复未写入data file的已成功事务更新的数据。undo log和redo log 都有持久化的要求，两者的共同协作实现了事务持久化要求，不论是否遭遇宕机，数据库实例在启动时都会检查是否存在数据不一致，这与redis启动时检查AOF文件是类似的。</p>
<p>Undo log除了在rollback事务时用到，另外还在MVCC机制中使用，它按页记录了每一个事务版本在开始时需要用到的数据版本，因此同样的数据页，在undo log中可能存在多个版本，因此undo log需要支持随机读（redo log不需要）。在事务提交后，undo log对应的数据页不再有用了，但不会被立即清除，MySQL内部线程架构中，有Purge Thread专门对它进行定时清理。具体触发清理的机制是比较复杂的。</p>
<p>MySQL Innodb引擎采用 WAL（Write-Ahead Log）方式写入redo log，WAL通俗点说就是说在事务所有数据修改提交前，需要先将其对应的操作日志追加写入磁盘文件（AOF：append only file），以便出现意外可以恢复（所以redo log也叫做重放日志）,这样就达到了持久性的要求。此外鼎鼎大名的MongoDB（WiredTiger存储引擎）、SQLite也采用了WAL机制。</p>
<p>Bin log的提出主要是面向于MySQL的Replication架构，在多个MySQL节点间，需要一种机制对节点间的数据进行同步。例如在Master-Slave模式下，Master节点每次对数据库进行一次修改操作（DML SQL），就记录这些SQL语句到Master节点本机的bin log文件中，Slave节点会定期访问Master节点的bin log，并按照上次最后一次读取的position值，读取后续的字节，读取后的字节先放在Slave节点的本地relay log中（AOF），由Slave本机的SQL线程重放到Slave实例，重放时Slave也会记录bin log，至此就完成了同步。Redis也采用了类似的多节点数据增量同步方案，详情请查阅<a href="http://www.distorage.com/%E5%88%86%E5%B8%83%E5%BC%8F%E7%B3%BB%E7%BB%9F%E6%8A%80%E6%9C%AF%E7%B3%BB%E5%88%97-gossip%E7%AE%97%E6%B3%95/?utm_source=tuicool&utm_medium=referral" target="_blank" rel="external">Gossip算法</a>。</p>
<p>Master节点也可以触发强制Slave进行一次同步，如果Slave节点下面不再挂其他MySQL节点，Slave节点可以关闭bin log，这依据Master-Slave的配置来决定。</p>
<p>Bin log与redo log的主要不同体现在，redo log只记录由Innodb存储引擎产生的事务重放日志，它是按数据页进行存储的。而bin log记录了所有类型的存储引擎执行的DML SQL语句。最后一点是，redo log总是在事务进行中持续顺序写入磁盘，而bin log只在DML SQL提交后写入一次。如果没有使用事务，执行DML SQL时，bin log file会产生日志，而redo log file无变化。</p>
<p>为了减少日志刷盘造成写IO压力，Innodb对redo log和bin log的刷盘操作做了大量优化，使用组提交（Group commit）的刷盘方式来提高性能，同是使用prepare lock保证redo log和bin log的顺序一致性（prepare_commit_mutex 配置项）。</p>
<h3 id="Checkpoint机制"><a href="#Checkpoint机制" class="headerlink" title="Checkpoint机制"></a>Checkpoint机制</h3><p>Checkpoint是为了定期将db buffer的内容刷新到data file。当遇到内存不足、db buffer已满等情况时，需要将db buffer中的内容/部分内容（特别是脏数据）转储到data file中。在转储时，会记录checkpoint发生的”时刻“。在故障回复时候，只需要redo/undo最近的一次checkpoint之后的操作。</p>
<h3 id="逻辑事务-VS-物理事务"><a href="#逻辑事务-VS-物理事务" class="headerlink" title="逻辑事务 VS 物理事务"></a>逻辑事务 VS 物理事务</h3><p>在谈逻辑与物理时，经常说到“逻辑删除”/“物理删除”、“逻辑地址”/“物理地址”，那逻辑事务与物理事务又是什么呢？</p>
<p>一般而言，所谓的数据库事务都是针对单个数据库的事务，即单库事务。而跨库事务，顾名思义，是指涉及多个数据库的事务，理论上也必须满足ACID属性。两者最核心的区别在于，单库事务一般是由数据库保证的，俗称物理事务，而跨库事务一般是由应用保证的，俗称逻辑事务。与单库事务相比，跨库事务执行成本高，稳定性差，管理也更复杂，但在某些场景下，尤其是分布式应用环境下，又是不得不使用的技术。</p>
<p>再举个栗子，单库事务好比你从北京飞上海，到东航官网买张票就搞定了，而跨库事务好比北京飞纽约，到上海转机，就得买东航转上航的联票，出票就转由携程保证了。</p>
<p>而在Spring中，事务分为物理事务和逻辑事务；<br>物理事务：就是底层数据库提供的事务支持，如JDBC或JTA提供的事务；<br>逻辑事务：是Spring管理的事务，不同于物理事务，逻辑事务提供更丰富的控制，而且如果想得到Spring事务管理的好处，必须使用逻辑事务，因此在Spring中如果没特别强调一般就是逻辑事务；</p>
<p>物理事务和逻辑事务最大差别就在于事务传播行为，事务传播行为用于指定在多个事务方法间调用时，事务是如何在这些方法间传播的</p>
<p>逻辑事务即支持非常低级别的控制，也有高级别解决方案：</p>
<h4 id="低级别解决方案"><a href="#低级别解决方案" class="headerlink" title="低级别解决方案"></a>低级别解决方案</h4><p>工具类：使用工具类获取连接（会话）和释放连接（会话），如使用org.springframework.jdbc.datasource包中的ConnectionUtils类来获取和释放具有逻辑事务功能的连接。当然对集成第三方ORM框架也提供了类似的工具类，如对Hibernate提供了SessionFactoryUtils工具类，JPA的EntityManagerFactoryUtils等，其他工具类都是使用类似<em>*</em>Utils命名；</p>
<h4 id="高级别解决方案"><a href="#高级别解决方案" class="headerlink" title="高级别解决方案"></a>高级别解决方案</h4><p>模板类：使用Spring提供的模板类，如JdbcTemplate、HibernateTemplate和JpaTemplate模板类等，而这些模板类内部其实是使用了低级别解决方案中的工具类来管理连接或会话；</p>
<h3 id="编程式事务-VS-声明式事务"><a href="#编程式事务-VS-声明式事务" class="headerlink" title="编程式事务 VS 声明式事务"></a>编程式事务 VS 声明式事务</h3><p>Spring提供两种编程式事务支持：直接使用PlatformTransactionManager实现和使用TransactionTemplate模板类，用于支持逻辑事务管理。<br>如果采用编程式事务推荐使用TransactionTemplate模板类和高级别解决方案。</p>
<h3 id="事务隔离级别"><a href="#事务隔离级别" class="headerlink" title="事务隔离级别"></a>事务隔离级别</h3><h4 id="默认隔离级别"><a href="#默认隔离级别" class="headerlink" title="默认隔离级别"></a>默认隔离级别</h4><h5 id="不同ORM框架的默认隔离级别"><a href="#不同ORM框架的默认隔离级别" class="headerlink" title="不同ORM框架的默认隔离级别"></a>不同ORM框架的默认隔离级别</h5><h5 id="不同DB-Proxy的默认隔离级别"><a href="#不同DB-Proxy的默认隔离级别" class="headerlink" title="不同DB Proxy的默认隔离级别"></a>不同DB Proxy的默认隔离级别</h5><h5 id="不同RDBMS的默认隔离级别"><a href="#不同RDBMS的默认隔离级别" class="headerlink" title="不同RDBMS的默认隔离级别"></a>不同RDBMS的默认隔离级别</h5><p>MySQL InnoDB Default：可重复读</p>
<p>Oracle Default：读已提交</p>
<p>不同存储引擎的默认隔离级别</p>
<h3 id="事务传播行为"><a href="#事务传播行为" class="headerlink" title="事务传播行为"></a>事务传播行为</h3><p>Spring 事务传播行为</p>
<h4 id="默认传播行为"><a href="#默认传播行为" class="headerlink" title="默认传播行为"></a>默认传播行为</h4><h3 id="RDBMS如何权衡事务隔离与高并发读写？"><a href="#RDBMS如何权衡事务隔离与高并发读写？" class="headerlink" title="RDBMS如何权衡事务隔离与高并发读写？"></a>RDBMS如何权衡事务隔离与高并发读写？</h3><h4 id="MVCC与Free-Lock"><a href="#MVCC与Free-Lock" class="headerlink" title="MVCC与Free Lock"></a>MVCC与Free Lock</h4><p>多线程环境下各种数据结构的实现有了很大的变化，每当我们更新某个数据的时候，我们都要考虑其它线程是否对其进行了修改。最简单的一种方法就是加锁，不过加锁会导致性能低下，而且可能阻塞其他线程。因此，我们引入了非阻塞(non-blocking)的算法 —— 通过CAS（Compare & Set，或是Compare & Swap）操作保证操作的原子性，同时我们还引入了 lock-free 的概念，它指的是一个线程出现问题（如阻塞，失败）但不影响其他线程（从总体看程序仍然是在运行的）</p>
<p>CPU：CAS_ADD</p>
<h5 id="共享锁"><a href="#共享锁" class="headerlink" title="共享锁"></a>共享锁</h5><h5 id="排它锁"><a href="#排它锁" class="headerlink" title="排它锁"></a>排它锁</h5><h5 id="间隙锁"><a href="#间隙锁" class="headerlink" title="间隙锁"></a>间隙锁</h5><h3 id="在主从架构下RDBMS如何保证事务？"><a href="#在主从架构下RDBMS如何保证事务？" class="headerlink" title="在主从架构下RDBMS如何保证事务？"></a>在主从架构下RDBMS如何保证事务？</h3><h3 id="在双主架构下RDBMS如何保证事务？"><a href="#在双主架构下RDBMS如何保证事务？" class="headerlink" title="在双主架构下RDBMS如何保证事务？"></a>在双主架构下RDBMS如何保证事务？</h3><h3 id="双活数据中心的事务管理"><a href="#双活数据中心的事务管理" class="headerlink" title="双活数据中心的事务管理"></a>双活数据中心的事务管理</h3><h2 id="分布式事务"><a href="#分布式事务" class="headerlink" title="分布式事务"></a>分布式事务</h2><h3 id="2PC"><a href="#2PC" class="headerlink" title="2PC"></a>2PC</h3><h3 id="3PC"><a href="#3PC" class="headerlink" title="3PC"></a>3PC</h3><h3 id="拜占庭将军问题-两军问题"><a href="#拜占庭将军问题-两军问题" class="headerlink" title="拜占庭将军问题/两军问题"></a>拜占庭将军问题/两军问题</h3><h3 id="事务解耦"><a href="#事务解耦" class="headerlink" title="事务解耦"></a>事务解耦</h3><h3 id="事务补偿"><a href="#事务补偿" class="headerlink" title="事务补偿"></a>事务补偿</h3><h3 id="事务回滚"><a href="#事务回滚" class="headerlink" title="事务回滚"></a>事务回滚</h3><h3 id="最终资源一致性"><a href="#最终资源一致性" class="headerlink" title="最终资源一致性"></a>最终资源一致性</h3><h3 id="与nosql"><a href="#与nosql" class="headerlink" title="与nosql"></a>与nosql</h3><h4 id="MongoDB如何实现事务？"><a href="#MongoDB如何实现事务？" class="headerlink" title="MongoDB如何实现事务？"></a>MongoDB如何实现事务？</h4><h5 id="表锁与行锁"><a href="#表锁与行锁" class="headerlink" title="表锁与行锁"></a>表锁与行锁</h5><h4 id="Redis如何实现事务？"><a href="#Redis如何实现事务？" class="headerlink" title="Redis如何实现事务？"></a>Redis如何实现事务？</h4><h5 id="串行化"><a href="#串行化" class="headerlink" title="串行化"></a>串行化</h5><h6 id="Redis与MVCC实现Free-Lock"><a href="#Redis与MVCC实现Free-Lock" class="headerlink" title="Redis与MVCC实现Free Lock"></a>Redis与MVCC实现Free Lock</h6><p>HBase，couchBase，leveldb</p>
<h3 id="阿里DTS"><a href="#阿里DTS" class="headerlink" title="阿里DTS"></a>阿里DTS</h3><h2 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h2><ul>
<li><p>《MySQL技术内幕：InnoDB存储引擎（第2版）》</p>
</li>
<li><p><a href="https://zh.wikipedia.org/wiki/%E6%95%B0%E6%8D%AE%E5%BA%93%E4%BA%8B%E5%8A%A1" target="_blank" rel="external">数据库事务-维基百科</a></p>
</li>
<li><a href="https://zh.wikipedia.org/wiki/%E6%8B%9C%E5%8D%A0%E5%BA%AD%E5%B0%86%E5%86%9B%E9%97%AE%E9%A2%98" target="_blank" rel="external">拜占庭将军问题</a></li>
<li><a href="http://weibo.com/ttarticle/p/show?id=2309403992797932856430" target="_blank" rel="external">7-10倍写入性能提升:剖析WiredTiger数据页无锁及压缩黑科技</a></li>
</ul>
]]></content>
<categories>
<category> Summary </category>
</categories>
<tags>
<tag> Redis </tag>
<tag> Transaction </tag>
<tag> Spring </tag>
<tag> MVCC </tag>
<tag> WAL </tag>
<tag> RedoLog </tag>
<tag> UndoLog </tag>
<tag> RelayLog </tag>
<tag> Checkpoint </tag>
<tag> Gossip </tag>
<tag> Replication </tag>
</tags>
</entry>
<entry>
<title><![CDATA[分布式软件理论总结]]></title>
<url>https://amao12580.github.io/post/2016/05/Summary-of-distributed-theory/</url>
<content type="html"><![CDATA[<p><img src="/img/thinking.jpg" alt=""></p>
<h1 id="软件系统的本质"><a href="#软件系统的本质" class="headerlink" title="软件系统的本质"></a>软件系统的本质</h1><p>软件是现实的抽象，讨论抽象的本质是困难的，让我们一起在现实中思考。小到一支笔，大到一部汽车，这些产品的出现，使我们在现实生活中得到了方便。这里说的方便，是指因为使用了这些产品，降低了成本，提高了效率。在没有笔的时代，想要记录信息需要结绳、岩画或篆刻，这是高成本的，效率也很低。同样的，没有汽车的时代，想要到达另一个城市，会因为恶劣天气而被迫中途搁置，相应的通行时间也会比现在长得多。进一步的思考，笔与汽车的共同点又在哪里呢？</p>
<p>笔与汽车，这些产品是为了满足人们的各类生活需求而发明的。即，人们因为现实需要，而发明或改进了产品，产品是为了帮助人们更好的生活而存在。同样的，软件系统也是如此，人们出于远距离沟通交流需要，发明电话拨号系统；因为更随时随地的信息多样化交流，发明了各种互联网IM产品。</p>
<p>软件系统是依托于人们现实需求而存在，用以辅助人们更好的生活，它是现实世界在虚拟世界的抽象。同样的软件系统也存在着兴衰淘汰，一款软件刚问世时，受欢迎程度总是处于低谷，存在现实世界对其接受适应的时间窗。突破时间窗，进而逐渐闻名天下，此时依托此软件又会催生出许多子软件链（如chromium的众多壳浏览器、SVN的众多客户端）。随着技术革新，软件产业也会更新换代，跟不上时代的软件系统会逐渐退出市场（如vista操作系统的短命）。但市场总是存在需要的，类似的软件在倒下后总会有后继者，如浏览器，这几乎在国内，每家互联网公司都想做或者已经在做。</p>
<h1 id="不良设计的局限性"><a href="#不良设计的局限性" class="headerlink" title="不良设计的局限性"></a>不良设计的局限性</h1><p>在软件系统的发展初期，适用面狭窄，功能单一、用户少、价值不够高，造成对软件系统研发的不重视，成本投入低。相应的，软件所带来的经济收益也是有限的。此时的软件系统往往谈不上架构，往往是一个软件系统承载了所有功能，单个功能模块没有边界，逻辑分散，而各个功能模块之间的耦合关系是散乱的。</p>
<p><img src="/img/chaos.jpg" alt=""></p>
<p>在软件系统有机会成长起来后，带来了一定的经济收益。我们会思考软件系统的功能增多，更适应于用户（市场需要），或者考虑增强原有用户黏性，此时会对软件系统提出更多要求。而初期功能模块散乱的堆叠，对迫切需要的可扩展性带来了的麻烦，紧耦合的架构造成牵一发而动全身的问题，新功能的增加，往往意味着巨大地改造成本。勉强加上一些功能后，整个软件系统简直是乱作一团，就像是一堆红豆绿豆，这样的软件系统，我们称之为集中式软件系统。</p>
<p>集中式的软件系统还对重构带来挑战，随着功能需求越来越频繁的提出，往往意味着需要更短的版本迭代周期。现有系统无法灵活扩展，意味着需要对整个软件系统进行重构，而等到软件散乱性膨胀到一定程度，重构几乎变成不可完成的任务，就算是系统初始负责人，也无法评估重构带来的风险与收益究竟谁更多？</p>
<h2 id="停不下来的运动员"><a href="#停不下来的运动员" class="headerlink" title="停不下来的运动员"></a>停不下来的运动员</h2><p>集中式的软件系统在濒危阶段，开发团队往往面临两难，团队任何人都知道急需重构，但因为没有决策权，最高决策领导也顾虑着市场利益，而不可能给出重构所需的资源(时间、人力、软硬件)。在愈发杂乱的系统上进行维护和新功能叠加，这让团队内的每个人都需要指数级的投入成本，往往还达不到市场需要的质量。</p>
<p>就像田径400米接力比赛中，只能跑第一棒的运动员，本应在交棒后停下来休息，自我调整。而由于接手下一棒的运动员迟迟不能到位，运动员只能咬牙坚持跑第二棒，但迈出的每一步都越来越累，没法再像刚开始保持加速度。更令人担忧的是，此时裁判还吹响了冲刺哨，要求运动员进行冲锋。这样，运动员只有倒下了。相应的，软件系统到此时，开发团队成员会加速流失，新招进来的人也很难理解整套系统的协作，经常顾此失彼，从而导致最终的系统失败。</p>
<p>这也就是为什么CTO必需对技术团队保持深入理解，同时还应有对产品、开发的最高决策权，以支持技术团队在适时进行系统级别的重构，让失去体力的软件系统下线休息，换上更适合的新软件系统上线，从而更好地满足公司的战略需求，这是非常有必要的。</p>
<h1 id="向敏捷开发过渡"><a href="#向敏捷开发过渡" class="headerlink" title="向敏捷开发过渡"></a>向敏捷开发过渡</h1><p><img src="/img/building-blocks.jpg" alt=""></p>
<p>在软件系统有机会从集中式软件模式，过渡到积木型的软件模式后。系统原有是100%的功能堆叠到一个系统，现在变成了多个积木，也就是软件子系统，每个积木只负责其中的部分功能，积木间的协作，从而拼装出完整的软件系统。</p>
<h2 id="快速交付"><a href="#快速交付" class="headerlink" title="快速交付"></a>快速交付</h2><p>集中式的软件往往灵活性低下，这体现在很多方面。</p>
<p>1.设计的灵活性，想要以好的设计替换掉不再适应设计方案，会遭遇无法评估改造成本。</p>
<p>2.构建的灵活性，软件功能过于集中，导致一个小的编译问题，都会影响整个软件的构建，就像走钢丝一样，必需每一步都完美无缺才能安全走到终点。</p>
<p>3.测试的灵活性，测试任务因为功能互相紧耦合而变得繁重，无法评估已经做好的功能是否会受到影响。</p>
<p>4.部署的灵活性，运维无法根据系统的流量特性来优化软硬件配置，如JVM调优，Load Balance等。</p>
<p>5.修复的灵活性，这一点往往在集中式软件中是一个优势，在线上出现问题，只需要简单的分析少量的运行日志即可定位问题，这在积木性系统中是一个软肋。</p>
<p>敏捷开发的主要优势就在于，合理的利用分治思想，将复杂的功能的系统模块化处理，拆分成由只有简单功能的子系统而拼凑出来的完整系统，每一个子系统可以交付到小团队来维护，甚至与有单独的运维人员来进行保障。</p>
<p><img src="/img/MircoService.png" alt=""></p>
<h2 id="可维护性"><a href="#可维护性" class="headerlink" title="可维护性"></a>可维护性</h2><p>这是一个巨大的转变，首当其冲的就是可维护性，只需要定义一套灵活的通信协议，子系统之间通过这套协议进行通讯，我们在对子系统内部的功能进行维护时，不必担忧其他子系统受到干扰。这也存在一个前提，能够封装在一个子系统的功能模块集合，必须是内聚型的，它们几乎很少对外层子系统产生耦合，或者通过统一的出口进行耦合关联。</p>
<h2 id="可扩展性"><a href="#可扩展性" class="headerlink" title="可扩展性"></a>可扩展性</h2><p>其次是软件的可扩展性，新功能的开发，只需要新加一个子系统，扩展通讯协议，就可以满足要求，这对团队内的任何一个人都是令人振奋的。</p>
<h2 id="隔离性"><a href="#隔离性" class="headerlink" title="隔离性"></a>隔离性</h2><p>积木型的软件构建，还给了我们保持专注的可能。现在我们可以为软件进行动静分离设计，以适应不同的最佳部署方式。可以按照系统流量特性，应用CQRS架构，提供更好的局部性能。进一步的还可以为强共用性的功能进行独立剥离出来，譬如单独的文件系统负责所有的文件服务，以在避免各个积木内部的重复造轮子，独立的服务还为以后的：可维护性、伸缩性带来了诸多方便。</p>
<h2 id="质量可控"><a href="#质量可控" class="headerlink" title="质量可控"></a>质量可控</h2><p>同时，软件的质量可以得到保证，测试部门可以明确的针对某个软件子系统进行测试，而不需要每次系统上线都进行整体测试，工作量的降低，在相等的时间内，有了更多的时间进行问题修复，这往往意味着质量的提高。</p>
<p>积木型的软件，是对积木之间通讯协议的高要求，实际上，我们需要引入一整套消息通讯中间件来解决这个问题，在SOA和EDA架构中，最为关键的就是MQ协议的设计了。</p>
<p>松耦合的模式还带来局部失败和最终一致性的问题，可能会给用户造成短时间迷惑，但不影响最终的数据状态统一，这在传统软件领域往往是不可接受的，但进入到互联网后，在系统层面往往需要进行权衡，是市场机遇重要？还是少部分的用户利益重要？这个问题需要每一个互联网人的思考，产品是不是要满足每一个人的要求呢？</p>
<h1 id="分布式系统的利与弊"><a href="#分布式系统的利与弊" class="headerlink" title="分布式系统的利与弊"></a>分布式系统的利与弊</h1><h2 id="软件的构成"><a href="#软件的构成" class="headerlink" title="软件的构成"></a>软件的构成</h2><p>概括的讲，软件普遍是由三部分组成：接入、逻辑、存储。其中接入是可能没有的，比如定时任务型软件，由操作系统时间来触发，不需要外界的主动调度。而逻辑和存储，很难有软件是不需要的。而比如典型的在线社交服务，常规的构成可以分为：聊天服务、长连接服务、推送服务、好友信息服务、文件服务、推荐服务、广告服务。这些按业务功能相关性而划分的服务，没有哪一个可以脱离接入、逻辑和存储的。这些服务从本质上来说大部分都是OLTP(<a href="http://blog.csdn.net/tianlesoftware/article/details/5794844" target="_blank" rel="external">Online Transaction Processing</a>)，就是将现实生活中的沟通交流需求转移到了互联网进行。</p>
<h2 id="SOA-VS-EDA"><a href="#SOA-VS-EDA" class="headerlink" title="SOA VS EDA"></a>SOA VS EDA</h2><h3 id="什么是SOA？"><a href="#什么是SOA？" class="headerlink" title="什么是SOA？"></a>什么是SOA？</h3><h4 id="SOA-VS-Mirco-Service"><a href="#SOA-VS-Mirco-Service" class="headerlink" title="SOA VS Mirco Service"></a>SOA VS Mirco Service</h4><h3 id="什么是EDA？"><a href="#什么是EDA？" class="headerlink" title="什么是EDA？"></a>什么是EDA？</h3><h3 id="分场景使用"><a href="#分场景使用" class="headerlink" title="分场景使用"></a>分场景使用</h3><h2 id="Design-for-failure"><a href="#Design-for-failure" class="headerlink" title="Design for failure"></a>Design for failure</h2><h3 id="Fast-fail-VS-Retry"><a href="#Fast-fail-VS-Retry" class="headerlink" title="Fast fail VS Retry"></a>Fast fail VS Retry</h3><h2 id="无状态"><a href="#无状态" class="headerlink" title="无状态"></a>无状态</h2><h2 id="应用中心和任务中心"><a href="#应用中心和任务中心" class="headerlink" title="应用中心和任务中心"></a>应用中心和任务中心</h2><p>单点服务向多点服务的转变</p>
<p>原因：<br>性能要求，多节点同时参与服务(按流量权重分发，负载均衡；按读写流量分发，读写分离)。<br>允许少部分节点失败，更可靠的运行。</p>
<p>数据存储跨机分片，同时每片在多机存在复制集。计算资源无状态，本身就可以动态管理。便捷的扩容缩容(伸缩性)。</p>
<p>因业务分拆，降低耦合粒度，降低系统扩展成本(可扩展性)，更高的可维护性。</p>
<p>带来的问题。<br>系统内部改造，支持多节点转变。</p>
<p>分布式事务，事务补偿。</p>
<p>分布式存储</p>
<p>分布式访问</p>
<p>容忍最终一致性。</p>
<h1 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h1><ul>
<li><p><a href="https://blog.eood.cn/exception" target="_blank" rel="external">互联网系统可靠性基础：正确的异常处理</a></p>
</li>
<li><p><a href="http://www.sczyh30.com/posts/Microservice/circuit-breaker-pattern/" target="_blank" rel="external">微服务设计：熔断器模式</a></p>
</li>
<li><p><a href="http://ju.outofmemory.cn/entry/195996" target="_blank" rel="external">采用断路器设计模式来保护软件</a></p>
</li>
</ul>
]]></content>
<categories>
<category> Summary </category>
</categories>
<tags>
<tag> Distributed </tag>
</tags>
</entry>
<entry>
<title><![CDATA[软件重构经验总结]]></title>
<url>https://amao12580.github.io/post/2016/05/Software-refactoring-experience-summary/</url>
<content type="html"><![CDATA[<p>结合小旺项目的重构经历来分析</p>
<h1 id="小旺项目：5月份重构方案"><a href="#小旺项目：5月份重构方案" class="headerlink" title="小旺项目：5月份重构方案"></a>小旺项目：5月份重构方案</h1><h2 id="转店-找店-拆表"><a href="#转店-找店-拆表" class="headerlink" title="转店/找店 拆表"></a>转店/找店 拆表</h2><p>参考：转店需求：transfer_requirement，字段requirement_id。转化为Service表的id。</p>
<h2 id="地理位置相关业务重构"><a href="#地理位置相关业务重构" class="headerlink" title="地理位置相关业务重构"></a>地理位置相关业务重构</h2><p>读 >> 写</p>
<p>涉及到以下业务字段的，需要重构。</p>
<ul>
<li>经纬度</li>
<li>区域</li>
<li>详细地址</li>
</ul>
<p>Table：geographical_location</p>
<p>Fileds：</p>
<table>
<thead>
<tr>
<th>主键</th>
<th style="text-align:center">业务模块编号</th>
<th style="text-align:right">业务模块类型</th>
<th style="text-align:right">纬度</th>
<th style="text-align:right">经度</th>
<th style="text-align:right">区域编号</th>
<th style="text-align:right">详细地址</th>
</tr>
</thead>
<tbody>
<tr>
<td>id</td>
<td style="text-align:center">source_id</td>
<td style="text-align:right">type</td>
<td style="text-align:right">latitude</td>
<td style="text-align:right">longitude</td>
<td style="text-align:right">aeraId</td>
<td style="text-align:right">address</td>
</tr>
</tbody>
</table>
<p>source_id：关联的业务模块的主键</p>
<p>type：标识到业务模块的类型，枚举。</p>
<p>type字段主要是考虑这张表数据会膨胀的很快，以后可以以这个字段为依据做：mysql表分区、redis分库</p>
<h3 id="问题"><a href="#问题" class="headerlink" title="问题"></a>问题</h3><p>冗余：不是所有的业务都有这些字段，特别是经纬度</p>
<h2 id="数据的记录与状态分离"><a href="#数据的记录与状态分离" class="headerlink" title="数据的记录与状态分离"></a>数据的记录与状态分离</h2><p>（业务与图片的关联性，也可以参照）</p>
<p>读 ~= 写</p>
<h3 id="数据状态"><a href="#数据状态" class="headerlink" title="数据状态"></a>数据状态</h3><p>Table：status</p>
<p>Fileds：</p>
<table>
<thead>
<tr>
<th>主键</th>
<th style="text-align:center">业务模块编号</th>
<th style="text-align:right">业务模块分类状态</th>
<th style="text-align:right">状态值(tinyint)</th>
</tr>
</thead>
<tbody>
<tr>
<td>id</td>
<td style="text-align:center">source_id</td>
<td style="text-align:right">category</td>
<td style="text-align:right">value</td>
</tr>
</tbody>
</table>
<p>source_id：关联的业务模块的主键</p>
<p>category：区分单条数据记录内的多个状态属性值。一条数据记录可能同时存在很多状态属性，例如：订单，订单状态、支付状态。</p>
<h3 id="数据状态依赖"><a href="#数据状态依赖" class="headerlink" title="数据状态依赖"></a>数据状态依赖</h3><p>Table：status_dependence</p>
<p>Fileds：</p>
<table>
<thead>
<tr>
<th>主键</th>
<th style="text-align:center">状态编号</th>
<th style="text-align:right">状态依赖编号</th>
<th style="text-align:right">状态值(tinyint)</th>
</tr>
</thead>
<tbody>
<tr>
<td>id</td>
<td style="text-align:center">s_id</td>
<td style="text-align:right">dependence_id</td>
<td style="text-align:right">value</td>
</tr>
</tbody>
</table>
<p>s_id与dependence_id都来源于status表的id字段</p>
<h3 id="问题边界"><a href="#问题边界" class="headerlink" title="问题边界"></a>问题边界</h3><h4 id="状态依赖的边界"><a href="#状态依赖的边界" class="headerlink" title="状态依赖的边界"></a>状态依赖的边界</h4><ul>
<li>数据状态的依赖，需要区分于：数据记录的存在性或不存在性。对数据记录的计算，需要在应用层解决。</li>
</ul>
<h3 id="问题-1"><a href="#问题-1" class="headerlink" title="问题"></a>问题</h3><h4 id="如何解决集中化的读写？"><a href="#如何解决集中化的读写？" class="headerlink" title="如何解决集中化的读写？"></a>如何解决集中化的读写？</h4><h4 id="数据记录删除的处理"><a href="#数据记录删除的处理" class="headerlink" title="数据记录删除的处理"></a>数据记录删除的处理</h4><p>数据在主表进行物理删除或逻辑删除(isDeleted=1)时，如何处理关联的状态？</p>
<h4 id="状态依赖树形结构过深"><a href="#状态依赖树形结构过深" class="headerlink" title="状态依赖树形结构过深"></a>状态依赖树形结构过深</h4><p>这会带来递归更新、删除的问题，需要在应用层避免</p>
<h2 id="Solr异步写入-低优先级"><a href="#Solr异步写入-低优先级" class="headerlink" title="Solr异步写入 [低优先级]"></a>Solr异步写入 [低优先级]</h2><p>提高吞吐量：去除JOOQ Listener模式，减少在Solr进行非关键性操作时，对主流程的阻塞。</p>
<p>可控性：独立出数据同步组件，负责在mysql数据有变化后，按规则将数据同步到Solr、Redis等</p>
<h2 id="支付与合同独立出来"><a href="#支付与合同独立出来" class="headerlink" title="支付与合同独立出来"></a>支付与合同独立出来</h2><h2 id="与Notify系统的通讯方式"><a href="#与Notify系统的通讯方式" class="headerlink" title="与Notify系统的通讯方式"></a>与Notify系统的通讯方式</h2><h3 id="ActiveMQ"><a href="#ActiveMQ" class="headerlink" title="ActiveMQ"></a>ActiveMQ</h3><p>引入MQ。<br>将上层应用与Notify交互时依赖于Socket，转变为MQ代理。</p>
<h4 id="优势"><a href="#优势" class="headerlink" title="优势"></a>优势</h4><h5 id="消息传递可靠"><a href="#消息传递可靠" class="headerlink" title="消息传递可靠"></a>消息传递可靠</h5><h5 id="消息的优先级"><a href="#消息的优先级" class="headerlink" title="消息的优先级"></a>消息的优先级</h5><h5 id="消息重试机制"><a href="#消息重试机制" class="headerlink" title="消息重试机制"></a>消息重试机制</h5><h5 id="Notify的伸缩性"><a href="#Notify的伸缩性" class="headerlink" title="Notify的伸缩性"></a>Notify的伸缩性</h5><p>Notify节点的上下线，MQ是感知到的。</p>
<h5 id="Notify的吞吐量"><a href="#Notify的吞吐量" class="headerlink" title="Notify的吞吐量"></a>Notify的吞吐量</h5><p>P端的写入是长连接机制，省去大多数Socket链接创建、断开的开销。</p>
<p>Notify与APP通讯时，依靠JPUSH提供的HTTP接口，该接口存在调用频率限制</p>
<h5 id="MQ的可控性"><a href="#MQ的可控性" class="headerlink" title="MQ的可控性"></a>MQ的可控性</h5><p>支持自我监控</p>
<h5 id="MQ的可靠性"><a href="#MQ的可靠性" class="headerlink" title="MQ的可靠性"></a>MQ的可靠性</h5><h5 id="MQ的伸缩性"><a href="#MQ的伸缩性" class="headerlink" title="MQ的伸缩性"></a>MQ的伸缩性</h5><h5 id="MQ的HA方案"><a href="#MQ的HA方案" class="headerlink" title="MQ的HA方案"></a>MQ的HA方案</h5><h1 id="小旺项目重构方案"><a href="#小旺项目重构方案" class="headerlink" title="小旺项目重构方案"></a>小旺项目重构方案</h1><h2 id="串行化的幂等性操作"><a href="#串行化的幂等性操作" class="headerlink" title="串行化的幂等性操作"></a>串行化的幂等性操作</h2><p>钱包业务：WalletModelImpl.java getWalletSummary(int uid)</p>
<p>实现“callable”接口，提高性能。</p>
<h2 id="可以合并的请求"><a href="#可以合并的请求" class="headerlink" title="可以合并的请求"></a>可以合并的请求</h2><p>典型 1：for循环访问redis获取多个图片信息</p>
<p>改为：批量获取，访问一次redis</p>
<p>典型 2: 转店插件保存图片信息到mysql，循环访问mysql，插入数据</p>
<p>改为：batchInsert接口</p>
<h2 id="不必要的字段查询"><a href="#不必要的字段查询" class="headerlink" title="不必要的字段查询"></a>不必要的字段查询</h2><p>典型：ScopedContext.java getUserRecord()</p>
<p>基础接口，查询user表中，单条记录的所有字段</p>
<p>但是后续使用中，绝大多数只使用了其中的1个字段：id</p>
<h2 id="读-gt-gt-写"><a href="#读-gt-gt-写" class="headerlink" title="读 >> 写"></a>读 >> 写</h2><p>钱包业务：WalletModelImpl.java getWalletSummary(int uid)</p>
<p>请将读操作的结果保存下来，或者在写入时，后台线程计算出来并存储</p>
<h2 id="多表join"><a href="#多表join" class="headerlink" title="多表join"></a>多表join</h2><p>简单业务需要多表join</p>
<p>1.适度冗余，修改业务逻辑，重建数据库模型<br>2.利用好缓存：数据库级别的查询缓存、redis缓存行记录</p>
<h2 id="接口业务过于复杂"><a href="#接口业务过于复杂" class="headerlink" title="接口业务过于复杂"></a>接口业务过于复杂</h2><h2 id="去除jooq-listener同步调用"><a href="#去除jooq-listener同步调用" class="headerlink" title="去除jooq listener同步调用"></a>去除jooq listener同步调用</h2><p>异步方式重构<br>优点：<br>1.主流程可以尽快结束，提高TPS，降低RT<br>2.出错可以有重试机制</p>
<p>缺点：<br>1.业务需要容忍最终一致性<br>2.新系统的加入，加大了研发成本、运维成本、监控成本</p>
<p>1.调用notif</p>
<p>ActiveMQ、Redis</p>
<p>2.调用solr</p>
<p>另做系统，实时解析mysql binlog。参考阿里方案：otter</p>
<h2 id="系统监控"><a href="#系统监控" class="headerlink" title="系统监控"></a>系统监控</h2><p>1.日志流分析<br>2.系统业务指标<br>3.报警系统</p>
<h2 id="外键的大量使用"><a href="#外键的大量使用" class="headerlink" title="外键的大量使用"></a>外键的大量使用</h2><p>对没有强一致性要求的业务，去除外键。</p>
<h2 id="HA、Scalable-out"><a href="#HA、Scalable-out" class="headerlink" title="HA、Scalable out"></a>HA、Scalable out</h2><h2 id="系统容错性"><a href="#系统容错性" class="headerlink" title="系统容错性"></a>系统容错性</h2><h1 id="先了解"><a href="#先了解" class="headerlink" title="先了解"></a>先了解</h1><h2 id="了解中发现问题症结"><a href="#了解中发现问题症结" class="headerlink" title="了解中发现问题症结"></a>了解中发现问题症结</h2><h2 id="了解中思考解决方案"><a href="#了解中思考解决方案" class="headerlink" title="了解中思考解决方案"></a>了解中思考解决方案</h2><h2 id="选取合适的解决方案"><a href="#选取合适的解决方案" class="headerlink" title="选取合适的解决方案"></a>选取合适的解决方案</h2><h2 id="保留可扩展性"><a href="#保留可扩展性" class="headerlink" title="保留可扩展性"></a>保留可扩展性</h2><h2 id="保留可维护性"><a href="#保留可维护性" class="headerlink" title="保留可维护性"></a>保留可维护性</h2><h1 id="再动手"><a href="#再动手" class="headerlink" title="再动手"></a>再动手</h1><h2 id="重构方案的评审"><a href="#重构方案的评审" class="headerlink" title="重构方案的评审"></a>重构方案的评审</h2><h2 id="理清方案的优势与不足"><a href="#理清方案的优势与不足" class="headerlink" title="理清方案的优势与不足"></a>理清方案的优势与不足</h2><h2 id="资源预先到位"><a href="#资源预先到位" class="headerlink" title="资源预先到位"></a>资源预先到位</h2><p>包括重构的时间争取，参与人员配比，对其他软硬件资源、中间件的预研。</p>
<h2 id="重构前后的效果量化"><a href="#重构前后的效果量化" class="headerlink" title="重构前后的效果量化"></a>重构前后的效果量化</h2><h2 id="充分的测试"><a href="#充分的测试" class="headerlink" title="充分的测试"></a>充分的测试</h2><p>基础方面的重构，需要充分的测试</p>
<h1 id="方案上线"><a href="#方案上线" class="headerlink" title="方案上线"></a>方案上线</h1>]]></content>
<categories>
<category> Summary </category>
</categories>
<tags>
<tag> Experience </tag>
<tag> Refactor </tag>
</tags>
</entry>
<entry>
<title><![CDATA[如何更好地管理技术团队？]]></title>
<url>https://amao12580.github.io/post/2016/05/How-to-better-manage-the-technical-team/</url>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><h2 id="PM与TL的区别"><a href="#PM与TL的区别" class="headerlink" title="PM与TL的区别"></a>PM与TL的区别</h2><p>其实一个Team Leader的职责与Project Manager相像，但Team Leader更着重于技术开发方面，通常一个大型项目都会有一两个开发团队由Team Leader带领，负责开发核心部分，而其它部分分派给不同开发小组或者分派给外包公司。在网上常看到几句话，贴切地形容了PM与TL的区别：“技术人员乐于被领导；但他们不喜欢被管理，不喜欢像牛一样被驱赶或指挥。管理者强迫人们服从他们的命令，而领导者则会带领他们一起工作。管理是客观的，没有个人感情因素，它假定被管理者没有思想和感受，被告知要做什么和该如何做。领导是引领、引导，它激励人们达成目标。领导力是带有强烈个人感情色彩的，它不是你能命令的，也不是你能测量评估和测试的。”</p>
<p>TL对于团队内的member，必须在技术上能够进行胜任领导，他就像黑夜里的灯塔，引导和修正member前进的航向。因此TL也必需保持照亮团队，保持对member的充分了解，并在技术领域投入持续的学习热情，向团队成员传道，补齐短板，让大家的核心战斗力一起提高。</p>
<p>无论是PM与TL，对业务与技术都要有深入的了解，只是PM更侧重于业务的管理，盈利的多少，风险的大小等等，而TL则侧重于项目的成本，开发的难度，软件的架构等技术方面的问题。在某些人眼中，技术与管理就像鱼与熊掌，不可兼得，但依在下看来，两者却是秤不离砣，密不可分。只要及时提升自己对技术与管理的认识，不断地向深一层发展，要从程序员提升到技术管理人员只是时间的问题。</p>
<p>团队规模有限时，分工很可能没这么清晰，很多时候TL和PM是同一个人在兼任，角色的模糊，可能带来职责和权利的混乱，而且个人精力也有限，仅适合任务量较小时实行。</p>
<h2 id="技术管理-VS-团队管理"><a href="#技术管理-VS-团队管理" class="headerlink" title="技术管理 VS 团队管理"></a>技术管理 VS 团队管理</h2><p>技术团队与其它的团队的区别，可能在于技术人员的管理难度。一方面因为领域之间的差异性，譬如前端技术链与后端技术链的深与广，导致熟悉前端的同时，很难对后端保持足够的了解。另一方面在领域内的差异性，典型的是多年争论C语言与Java语言的优劣，忽视了技术都有局限性，脱离了使用需求的比较没有意义。如果技术人员比较狭隘，往往无法理解对方的差异性，从而造成不必要的争议。这一点在年轻的团队中，是很突出的问题，大家普遍经验欠缺，对上下游的技术无法做到足够认知，同时因为年轻而无法接受建议，行成内耗。</p>
<h3 id="凤栖梧桐"><a href="#凤栖梧桐" class="headerlink" title="凤栖梧桐"></a>凤栖梧桐</h3><p>技术团队管理需要划分为正切的两个方向：技术管理和团队管理。管理的目标是：提高协作时的执行力，执行效率，并约束进行规范执行。技术管理的目标是：保持并引导团队成员之间的技术协作一致性，如前端技术团队和后端技术团队。由技术负责人制定日常规范，推动规范的落地，以公有的强制约定来避免不必要的内耗。规范的含义是比较广泛的：设计规范、编码规范、安全规范、接口规范、部署规范。团队管理解决的问题是：如何持久降低团队之间或团队成员之间的合作成本？如产品团队和技术团队。除了leader的影响，往往需要合理的行政制度来作为基础。好的行政制度是公司最重要的部分，它就像生态系统的土壤，有了好的土壤，才可能生长出好的团队。在互联网时代，站在巨人肩上而诞生的伟大产品屈指可数了，很难有产品在一开始就100%贴合市场，更多的需要快速试错，进行小步快走的方式进行迭代，而在持续打磨的过程中，产品也就自我完善，可以产生商业价值了。我们很难想象在二流的团队中如何诞生一流的产品呢？管理者也需要深思，在渴求的一流产品的同时，是否给予了一流产品诞生所需要的环境呢？</p>
<h3 id="能力与影响力同样重要"><a href="#能力与影响力同样重要" class="headerlink" title="能力与影响力同样重要"></a>能力与影响力同样重要</h3><p>从纯粹的技术工作者转向管理者难点在于是否具备了管理能力？对与团队内的成员，是否一致的认为在你的带领下可以得到更好的发展？可以是经济利益上的或职业发展上的。对于上层决策领导是否认为你可担此大任？而如果太过于沉迷技术，缺少与决策层的互动交流，导致上层无法相信你已经具备管理能力，或确实缺少部分能力(可能是软实力，语言沟通能力)，这在国外很常见，管理层大多是美国人或印度人，中国人由于文化上的本位差异，很难上升到管理层。</p>
<h1 id="技术管理"><a href="#技术管理" class="headerlink" title="技术管理"></a>技术管理</h1><h2 id="设计规范"><a href="#设计规范" class="headerlink" title="设计规范"></a>设计规范</h2><p>如何在低复杂度、可维护性、可理解性之间进行平衡？需要在需求分析和系统设计时制定一些规范来约束，帮组我们低维护成本的延长软件生命周期。没有人可以保证业务在长远未来的走向，因此软件设计也只需要在可预见的范围内进行权衡考虑，超出范围的要交给升级版的设计来解决，这可能与传统思维不同。典型的，以系统容量进行设计，比如在电商系统中，按照统计数据分析预测在未来5年内，系统用户数会逐步增加到五千万，日订单量约在五百万左右。那我们在做系统设计时，就可以参照这些指标来做，只考虑5年内系统如何满足要求即可，但是请保留5年后设计进行升级的可能性，毕竟推倒重来的代价在系统愈发庞大后会让人无法接受。</p>
<h3 id="DRY"><a href="#DRY" class="headerlink" title="DRY"></a>DRY</h3><p>DRY(Don’t Repeat Yourself)的核心是面向可维护性，将性质重复的逻辑进行封装，抽象为一组对外调用的功能接口。</p>
<p>DRY 是一个最简单的法则，也是最容易被理解的。但它也可能是最难被应用的（因为要做到这样，我们需要在泛型设计上做相当的努力，这并不是一件容易的事）。它意味着，当我们在两个或多个地方的时候发现一些相似的代码的时候，我们需要把他们的共性抽象出来形一个唯一的新方法，并且改变现有的地方的代码让他们以一些合适的参数调用这个新的方法。</p>
<h4 id="vs-WET"><a href="#vs-WET" class="headerlink" title="vs WET"></a>vs WET</h4><p>违反DRY原则的解决方案通常被称为WET，指代“write everything twice”。</p>
<h3 id="KISS"><a href="#KISS" class="headerlink" title="KISS"></a>KISS</h3><p>KISS(Keep It Simple, Stupid)原则，是指在设计当中应当注重简约的原则。不只是软件系统设计，甚至在用户体验，需求分析时也可以应用。产品设计中，堆叠功能是容易的，考虑如何做减法才能体现产品经理的水准。</p>
<p>KISS原则在设计上可能最被推崇的，在家装设计，界面设计 ，操作设计上，复杂的东西越来越被众人所BS了，而简单的东西越来越被人所认可，比如这些UI的设计和我们中国网页（尤其是新浪的网页）者是负面的例子。“宜家”（IKEA）简约、效率的家居设计、生产思路；“微软”（Microsoft）“所见即所得”的理念；“谷歌”（Google)简约、直接的商业风格，无一例外的遵循了“kiss”原则，也正是“kiss”原则，成就了这些看似神奇的商业经典。而苹果公司的iPhone/iPad将这个原则实践到了极至。</p>
<p>把一个事情搞复杂是一件简单的事，但要把一个复杂的事变简单，这是一件复杂的事。</p>
<h3 id="SOLID"><a href="#SOLID" class="headerlink" title="SOLID"></a>SOLID</h3><p>在程序设计领域， SOLID（单一功能、开闭原则、里氏替换、接口隔离以及依赖反转）是由罗伯特·C·马丁在21世纪早期引入的记忆术首字母缩略字，指代了面向对象编程和面向对象设计的五个基本原则。当这些原则被一起应用时，它们使得一个程序员开发一个容易进行软件维护和扩展的系统变得更加可能。SOLID所包含的原则是通过引发编程者进行软件源代码的代码重构进行软件的代码异味清扫，从而使得软件清晰可读以及可扩展时可以应用的指南。SOLID被典型的应用在测试驱动开发上，并且是敏捷开发以及自适应软件开发的基本原则的重要组成部分。</p>
<h3 id="YAGNI"><a href="#YAGNI" class="headerlink" title="YAGNI"></a>YAGNI</h3><p>YAGNI(You Ain’t Gonna Need It)：适可而止，只考虑和设计必须的功能，避免过度设计。只实现目前需要的功能，在以后您需要更多功能时，可以再进行添加。（You Ain’t Gonna Need It，YAGNI 原则）</p>
<ul>
<li>如无必要，勿增复杂性。</li>
<li>软件开发先是一场沟通博弈。</li>
</ul>
<p>WebSphere的设计者就表示过他过度设计了这个产品。我们的程序员或是架构师在设计系统的时候，会考虑很多扩展性的东西，导致在架构与设计方面使用了大量折衷，最后导致项目失败。这是个令人感到讽刺的教训，因为本来希望尽可能延长项目的生命周期，结果反而缩短了生命周期。</p>
<h2 id="编码规范"><a href="#编码规范" class="headerlink" title="编码规范"></a>编码规范</h2><h3 id="命名规范"><a href="#命名规范" class="headerlink" title="命名规范"></a>命名规范</h3><p>我们在编写任何程序之前，第一件事要做的就是命名。</p>
<p>形如：需求文档名称，概要设计文档名称，接口文档名称，软件系统名称，功能模块名称，类名称，方法名称，参数名称…</p>
<p>在认识陌生人时，第一次接触时互递名片，熟络后的沟通交流，都以名字来起始了。名字是否好记，影响会有多大呢？前公司有一同事丁某某，父母取名讨巧用了生僻字，导致每次登机之前，需要到机场值班经理盖章，白白浪费很多时间。而恰好他是销售经理，每周有2~3次的商务飞行。你在想他为什么不去换名字？想想30来岁的人生积累，需要多少成本改名字吧？想象不出来，问问自己更换使用5年以上的手机号码，成本是多大吧？</p>
<h4 id="重要性"><a href="#重要性" class="headerlink" title="重要性"></a>重要性</h4><p>命名规范竟如此重要，但大多数开发者的命名习惯往往没有你想象的那么好。除了上面提到的生僻字问题，还有很多人喜欢用拼音去命名类、函数，甚至是变量。我不知道这是英文词汇量的问题还是个人风格，但我个人非常不提倡这样做。如何优雅地为程序中的变量和函数命名？很简单，把你的变量名拎出来，问别人，你看到这个名字会想到什么，他说的和你想的一致，就用，否则就改。改到基本上不懂程序的人都能大概看懂你写的是什么，就优雅了。</p>
<p>为什么重要？提高可维护性。不加约束的开发者写出来的代码，就像是大家临时拼凑出来的一桌菜，你上粤菜，他上川菜。虽然都好吃，但没有人全想吃，造成的局面是，爱吃粤菜的人总是夹粤菜，爱吃川菜的就不去吃粤菜了。在代码上来讲，各自的代码风格迥异，导致理解成本加大，你负责的代码需要修改时，就只有你能改了，团队其他人很难帮得上忙，如果修改工作量比较大，这岂不是把自己逼死吗？如果结合好的代码注释和设计文档，会不会理解成本下降呢？但就算是文档写得再详细，我们也要去读代码，所以文档主要是体现思路和反映需求和设计。在程序上，我们的命令应当和文档中的术语保持一致，而程序中的命名也应该是用和文档相同的风格，这样，我们可以少很多理解上的成本。好的代码在命名上是完全可读的，代码即文档，做到这点，可以极大的提高团队内协作效率。</p>
<p>比较常见的有下面三种命名方式：</p>
<ul>
<li>驼峰(Camel)命名法:又称小驼峰命名法，除首单词外，其余所有单词的第一个字母大写。GetOrderList</li>
<li>帕斯卡(Pascal)命名法:又称大驼峰命名法，所有单词的第一个字母大写。gtOrderList</li>
<li>蛇形(Snake)命名法:单词与单词间用下划线做间隔，看起来就像是上下扭动的蛇。get_order_list</li>
</ul>
<p>一般在命名函数和一般变量时，多使用小驼峰，命名常量和枚举变量时，使用大写的蛇形；命名类名多使用大驼峰，不建议使用蛇形命名。如在命名数据库表名时，最好使用小驼峰以一致性的对应到pojo。</p>
<p>好的命名，还给代码审计、代码排错工作降低成本，在做code review时，如果缺失详细注释(在敏捷开发中很常见)，规范化的命名使大家的理解成本降低，从而让审计工作更容易完成。我们回过头来想，很多公司很难推动代码审计的落地，到底是什么原因在阻碍呢？还有在线上代码深层排错时，我们可以采用SystemTap火焰图技术来做，由于函数的名称通常会包含语义上的信息，在输出的图表中就可以轻易的按照函数名来推断问题所在，这是非常了不起的。</p>
<p><img src="/img/flameGraph.png" alt=""></p>
<h4 id="如何做？"><a href="#如何做？" class="headerlink" title="如何做？"></a>如何做？</h4><p>具体的施行，可以参考：<a href="http://blog.csdn.net/yzmyyff/article/details/45243261" target="_blank" rel="external">Google Java Style</a></p>
<h4 id="约定优于配置"><a href="#约定优于配置" class="headerlink" title="约定优于配置"></a>约定优于配置</h4><p>约定优于配置的目标是，团队内各成员参照约定的策略来进行开发，减少无意义的说明项，从而降低沟通成本。为降低持续学习成本和保证团队新成员快速融入，约定项不宜过多，且应该放到团队成员随意备查的地方(media wiki?)。</p>
<h5 id="过渡配置"><a href="#过渡配置" class="headerlink" title="过渡配置"></a>过渡配置</h5><p>Hibernate的早期版本中，将类及其属性映射到数据库上需要是在XML文件中的描述，其中大部分信息都应能够按照约定得到，如将类映射到同名的数据库表，将属性分别映射到表上的同名字段。这样的做法不仅繁琐易出错，而且可维护性低，后续的版本抛弃了XML配置文件，而是使用这些恰当的约定，对于不符合这些约定的情形，可以使用Java注解来说明。</p>
<p>Spring由于其繁琐的配置，一度被人成为“配置地狱”，各种XML、Annotation配置，让人眼花缭乱，而且如果出错了也很难找出原因。Spring Boot项目就是为了解决配置繁琐的问题，最大化的实现convention over configuration(约定大于配置)。熟悉Ruby On Rails（ROR框架的程序员都知道，借助于ROR的脚手架工具只需简单的几步即可建立起一个Web应用程序。而Spring Boot就相当于Java平台上的ROR</p>
<p>构建管理工具链中，有很多体现了这个原则的正确性，如从Ant到Maven的过渡，Grunt到Gulp的过渡。其他的在中间件产品中，redis和mongodb的配置文件也有体现，只需要很少的个性化配置就可以正常运行。</p>
<h5 id="改进"><a href="#改进" class="headerlink" title="改进"></a>改进</h5><p>约定优于配置（convention over configuration），也称作按约定编程，是一种软件设计范式，旨在减少软件开发人员需做决定的数量，获得简单的好处，而又不失灵活性。<br>本质是说，开发人员仅需规定应用中不符约定的部分。例如，如果模型中有个名为Sale的类，那么数据库中对应的表就会默认命名为sales。只有在偏离这一约定时，例如将该表命名为”products_sold”，才需写有关这个名字的配置。</p>
<h3 id="合理的设计"><a href="#合理的设计" class="headerlink" title="合理的设计"></a>合理的设计</h3><p>合理的设计，主要是面向软件可维护性、可扩展性提出，如果能兼顾性能和安全就更好了。设计是有成本的，如果软件的生命周期在预期内就不会太长，例如很多一次性的产品：企业内网IM系统，售卖后，几乎不再需要任何的维护。此时设计上就可以从简，没有必要为不会来临的维护去打基础，因此设计也是需要有一定的前瞻性。</p>
<h4 id="适时分层"><a href="#适时分层" class="headerlink" title="适时分层"></a>适时分层</h4><p>分层设计的好处主要是面向软件可维护性。合理的分层设计，往往将逻辑拆分为内聚的多层次，层之间的边界和职责是清晰的。例如典型的三层设计，展示层、逻辑层、数据层。带来的好处是显而易见的，对于数据层来说，只需要对逻辑层提供访问数据的接口，具体如何实现访问数据，是访问文件还是RDBMS、NoSQL，与逻辑层无关。相应的重构起来也是局部的，如更换搜索引擎：Solr –> Elastic Search，只需要重构数据层的具体实现即可，上层任何代码不需要改动，这对于高复杂度的软件无疑是友好的。</p>
<p><img src="/img/splitLayer.png" alt=""></p>
<p>那分层对与软件可扩展性的影响呢？带来了一定程度的坏影响，新加的业务逻辑功能，可能涉及到很多层的改造，而在Maven mutilModule模式下，很可能层之间的代码是隔离的。这意味着，新加功能的代码量会有增多，相应的整体构建需要分布进行后融合。反过来想，这不失为一个好的过滤器，帮助我们分辨开发者的技能是否满足团队要求？</p>
<p>一般来说分层设计适用于大型软件的持续构建，它依靠团队内的其它高效率手段来弥补稍低的可扩展性。它对安全和性能，并没有什么影响。</p>
<h4 id="再谈过度设计"><a href="#再谈过度设计" class="headerlink" title="再谈过度设计"></a>再谈过度设计</h4><p>简单来说，过度设计就是进行了过多的面向未来的设计，进行了不必要的抽象封装，为系统增加了不必要的复杂度。<br>举个例子，你要做一个功能模块，但你考虑到到这个系统里还有几个未完成的模块和你要做的东西类似，所以你决定为此额外做一些抽象和封装，以便将来复用。然而到后来你开发那些相似的模块时你才发现，可能是由于抽象不足或抽象错误，你不得不重新修改之前的封装才能完成复用，导致最终成本实际上还不如不做；或者你发现复用的部分所降低的成本实际上还不如包装花费的成本。 这些都是最常见的过度设计的例子。<br>程序员在掌握了一些基本的设计能力之后，最常见也是最难克服的设计问题往往就是过度设计。上面的错误我相信大多数人都一而再，再而三的的犯过。</p>
<p>与过度设计相对的就是设计不足。<br>虽然是两个相对的概念，但设计不足和过度设计绝大多数时候都是一起出现的。都是最常见的设计问题。设计不足不仅常见于新手，老手也常犯。甚至我还见过有一类老程序员在经历过多次过度设计的打击之后，转向另一个极端，否定抽象封装的作用，走上“反设计”的道路。</p>
<p>过度设计和设计不足的平衡问题没有很好的解决办法，只有依靠经验的积累和不断的总结思考。如何把握这个度是最能考验程序员的经验和价值的问题之一。</p>
<p>我所尝试过的软件方法中，有一种方法的思维方式对于解决这个问题帮助最大，就是TDD（测试驱动开发），这里简单说下为什么TDD能解决这个问题：<br>TDD的一个核心思想是小步增量，不断重构。具体说来就是TDD有两个状态（常见的说法是两顶帽子）：<br>状态A：用test case描绘需求，并使用最简单的方式满足这个test case。注意要用最简单的方式满足这个需求，不能为任何test case之外的需求做任何设计。 test case通过之后进入状态B；<br>状态B：重构代码，让现有的代码在尽量保持简单性的同时足够优雅清晰。注意此时你只能对现有的实现代码进行重构，不能增加任何新的功能和test case。<br>整个TDD的过程就是在这两个状态间不断转换的过程。在状态A增加功能，在状态B优化设计。</p>
<p>TDD的这种思维方式走的稍微极端一点。它直接排斥任何对未来的设计，转而以优雅简洁的设计和test case来为未来需求的重构降低成本。 可以说严格遵循TDD做出来的设计必然在过度设计和设计不足方面都不会有太大的问题。</p>
<p>我严重推荐TDD。不管你最终会不会接受TDD这种开发方式，它独特的思维方式都必然会给你的设计观念带来很大影响。</p>
<p>过度设计的典型案例：<a href="http://coolshell.cn/articles/3005.html" target="_blank" rel="external">代码重构的一个示例</a></p>
<h4 id="模块化设计"><a href="#模块化设计" class="headerlink" title="模块化设计"></a>模块化设计</h4><p>模块化设计在重构时，可以将影响降到最低。达到低风险、低成本。它符合CCP原则（Common Closure Principle（CCP）– 共同封闭原则）。它对系统快速构建是一个阻碍，要求将可预见的性质相同代码，内聚到一个模块中。</p>
<p>例如读取和更新配置文件的功能，在软件中很多逻辑中需要调用。例如：读取定时任务的触发条件、读取接口调用频次限制。无论是从XML或JSON甚至在DB中读取，其实逻辑差别甚少，无非是：链接配置源、读取所需参数、解析正确性，返回并应用。如果重复这样的代码，将给我们带来麻烦，多类配置项产生依赖关系，读取和更新的交织将变得非常复杂。采用模块化设计，也是DRY原则的体现，对同性质的代码进行内聚为一个模块，对外部调用提供多个组合接口即可。</p>
<h3 id="前轻后重"><a href="#前轻后重" class="headerlink" title="前轻后重"></a>前轻后重</h3><p>系统之间的调用链，前置系统轻逻辑。</p>
<h4 id="上轻下重，核心逻辑内敛"><a href="#上轻下重，核心逻辑内敛" class="headerlink" title="上轻下重，核心逻辑内敛"></a>上轻下重，核心逻辑内敛</h4><p>模块之间的调用链，上层模块轻逻辑</p>
<h3 id="基础优先"><a href="#基础优先" class="headerlink" title="基础优先"></a>基础优先</h3><p>在构建软件时，保持基础组件优先稳定下来，定义好对外接口，并保留可扩展性。<br>例如安全性设计，在后期加比在前期加的成本高很多。</p>
<h3 id="最小冗余"><a href="#最小冗余" class="headerlink" title="最小冗余"></a>最小冗余</h3><h4 id="代码保持最少行数"><a href="#代码保持最少行数" class="headerlink" title="代码保持最少行数"></a>代码保持最少行数</h4><p>1.少一行代码，就少了一个潜在的bug</p>
<p>2.行数少了，往往意味着重用性高了</p>
<p>3.不要过度牺牲可读性</p>
<h3 id="保持简单"><a href="#保持简单" class="headerlink" title="保持简单"></a>保持简单</h3><h4 id="简单意味着快速"><a href="#简单意味着快速" class="headerlink" title="简单意味着快速"></a>简单意味着快速</h4><h4 id="简单意味着灵活"><a href="#简单意味着灵活" class="headerlink" title="简单意味着灵活"></a>简单意味着灵活</h4><h4 id="简单意味着易扩展"><a href="#简单意味着易扩展" class="headerlink" title="简单意味着易扩展"></a>简单意味着易扩展</h4><h4 id="简单意味着易重构"><a href="#简单意味着易重构" class="headerlink" title="简单意味着易重构"></a>简单意味着易重构</h4><h3 id="Fast-fail"><a href="#Fast-fail" class="headerlink" title="Fast fail"></a>Fast fail</h3><h3 id="代码即文档"><a href="#代码即文档" class="headerlink" title="代码即文档"></a>代码即文档</h3><h4 id="合理的注释"><a href="#合理的注释" class="headerlink" title="合理的注释"></a>合理的注释</h4><h3 id="关键业务设计宣讲"><a href="#关键业务设计宣讲" class="headerlink" title="关键业务设计宣讲"></a>关键业务设计宣讲</h3><h4 id="保持一致的看法"><a href="#保持一致的看法" class="headerlink" title="保持一致的看法"></a>保持一致的看法</h4><h3 id="小步快走"><a href="#小步快走" class="headerlink" title="小步快走"></a>小步快走</h3><h4 id="意味着版本迭代快速"><a href="#意味着版本迭代快速" class="headerlink" title="意味着版本迭代快速"></a>意味着版本迭代快速</h4><h4 id="意味着版本迭代稳定"><a href="#意味着版本迭代稳定" class="headerlink" title="意味着版本迭代稳定"></a>意味着版本迭代稳定</h4><h4 id="意味着部署回滚成本低"><a href="#意味着部署回滚成本低" class="headerlink" title="意味着部署回滚成本低"></a>意味着部署回滚成本低</h4><h2 id="重构与优化"><a href="#重构与优化" class="headerlink" title="重构与优化"></a>重构与优化</h2><h3 id="紧急优先"><a href="#紧急优先" class="headerlink" title="紧急优先"></a>紧急优先</h3><p>优先对紧急需要改进的部分进行重构</p>
<h3 id="计划性的重构"><a href="#计划性的重构" class="headerlink" title="计划性的重构"></a>计划性的重构</h3><p>在敏捷开发时，往往时间不够，有些模块只能以非优雅的方式构建。但是需要在这些模块加上TODO以免重构时遗忘。</p>
<h3 id="时机"><a href="#时机" class="headerlink" title="时机"></a>时机</h3><p>团队一致认为模块维护成本高昂时，果断重构。</p>
<h3 id="基于成本的优化"><a href="#基于成本的优化" class="headerlink" title="基于成本的优化"></a>基于成本的优化</h3><p>CBO(Cost base optimize)</p>
<p>引用数据库查询分析计划来说明：<a href="http://blog.jobbole.com/100349" target="_blank" rel="external">如果有人问你数据库的原理，叫他看这篇文章#查询优化器</a></p>
<h3 id="数据支撑"><a href="#数据支撑" class="headerlink" title="数据支撑"></a>数据支撑</h3><p>给出重构前后的成本与收益报告</p>
<h3 id="持续性"><a href="#持续性" class="headerlink" title="持续性"></a>持续性</h3><p>在时间不够或时机不成熟时，分阶段的进行</p>
<h2 id="技术选型"><a href="#技术选型" class="headerlink" title="技术选型"></a>技术选型</h2><h3 id="开源优先"><a href="#开源优先" class="headerlink" title="开源优先"></a>开源优先</h3><p>Facebook在推广MyRock持有非常开放的态度，所有的源码开发都是在外部的github上进行的，社区的大量使用可以带来软件的进一步成熟，进而反哺其本身的业务。</p>
<p>开源意味着安全，源码的公开维护，使得热门源码的安全漏洞被很大几率发现，进而修复。</p>
<p>开源不意味着任何人都可以修改代码，事实上这将带来隐藏bug问题。在github上，你可以对public权限的代码进行PR（pull request），而是否采纳你的PR，由维护团队来决定。</p>
<h3 id="稳定优先"><a href="#稳定优先" class="headerlink" title="稳定优先"></a>稳定优先</h3><h3 id="掌控度高优先"><a href="#掌控度高优先" class="headerlink" title="掌控度高优先"></a>掌控度高优先</h3><h3 id="保留备选项"><a href="#保留备选项" class="headerlink" title="保留备选项"></a>保留备选项</h3><h3 id="只有合适的技术，没有最好的技术。"><a href="#只有合适的技术，没有最好的技术。" class="headerlink" title="只有合适的技术，没有最好的技术。"></a>只有合适的技术，没有最好的技术。</h3><h3 id="避免一招鲜吃遍天"><a href="#避免一招鲜吃遍天" class="headerlink" title="避免一招鲜吃遍天"></a>避免一招鲜吃遍天</h3><p>手里有锤子，看到什么都觉得像钉子</p>
<h3 id="技术升级"><a href="#技术升级" class="headerlink" title="技术升级"></a>技术升级</h3><p>系统容量与性能指标的要求，在业务发展的各个阶段是不固定的，技术需要保留一定前瞻性，不做被动跟随。</p>
<h2 id="接口规范"><a href="#接口规范" class="headerlink" title="接口规范"></a>接口规范</h2><h3 id="内部接口"><a href="#内部接口" class="headerlink" title="内部接口"></a>内部接口</h3><h4 id="版本控制"><a href="#版本控制" class="headerlink" title="版本控制"></a>版本控制</h4><h4 id="访问控制"><a href="#访问控制" class="headerlink" title="访问控制"></a>访问控制</h4><p>访问权限的验证<br>访问频率</p>
<h4 id="流量控制"><a href="#流量控制" class="headerlink" title="流量控制"></a>流量控制</h4><p>拒绝恶意访问</p>
<p>暴力破解</p>
<h4 id="审查与监控"><a href="#审查与监控" class="headerlink" title="审查与监控"></a>审查与监控</h4><h4 id="灰度发布"><a href="#灰度发布" class="headerlink" title="灰度发布"></a>灰度发布</h4><h4 id="无状态"><a href="#无状态" class="headerlink" title="无状态"></a>无状态</h4><h4 id="原子化"><a href="#原子化" class="headerlink" title="原子化"></a>原子化</h4><h4 id="CQRS"><a href="#CQRS" class="headerlink" title="CQRS"></a>CQRS</h4><p>命令查询职责分离模式，读不强依赖写。</p>
<h3 id="APP"><a href="#APP" class="headerlink" title="APP"></a>APP</h3><h4 id="远程控制权"><a href="#远程控制权" class="headerlink" title="远程控制权"></a>远程控制权</h4><h4 id="展示为主"><a href="#展示为主" class="headerlink" title="展示为主"></a>展示为主</h4><h4 id="少的逻辑"><a href="#少的逻辑" class="headerlink" title="少的逻辑"></a>少的逻辑</h4><h4 id="适度安全"><a href="#适度安全" class="headerlink" title="适度安全"></a>适度安全</h4><h3 id="外部接口"><a href="#外部接口" class="headerlink" title="外部接口"></a>外部接口</h3><h4 id="合理调用"><a href="#合理调用" class="headerlink" title="合理调用"></a>合理调用</h4><p>同步VS异步<br>批量VS单个 批量：局部失败</p>
<h4 id="保持怀疑"><a href="#保持怀疑" class="headerlink" title="保持怀疑"></a>保持怀疑</h4><h5 id="兜底方案"><a href="#兜底方案" class="headerlink" title="兜底方案"></a>兜底方案</h5><p>如果有可能，在外部接口调用失败时，读取本地上一次调用成功的缓存。例如：加载和更新广告，CDN访问失败进行回源。</p>
<p>进行兜底时，注意将失败的信息进行记录以便告警</p>
<h2 id="部署规范"><a href="#部署规范" class="headerlink" title="部署规范"></a>部署规范</h2><h3 id="监控"><a href="#监控" class="headerlink" title="监控"></a>监控</h3><h3 id="降低升级时间窗"><a href="#降低升级时间窗" class="headerlink" title="降低升级时间窗"></a>降低升级时间窗</h3><h3 id="没有人需要等待"><a href="#没有人需要等待" class="headerlink" title="没有人需要等待"></a>没有人需要等待</h3><p>正式发布后，对于任何用户都是公平使用的。不需要牺牲部分用户利益。<br>常见于缓存autowarm时需要阻塞第一批到达的用户。</p>
<h3 id="提前消除隐患"><a href="#提前消除隐患" class="headerlink" title="提前消除隐患"></a>提前消除隐患</h3><h1 id="团队管理"><a href="#团队管理" class="headerlink" title="团队管理"></a>团队管理</h1><p>识人、用人、管人、留人。(华为)</p>
<h2 id="精细化管理"><a href="#精细化管理" class="headerlink" title="精细化管理"></a>精细化管理</h2><h2 id="keep-watch"><a href="#keep-watch" class="headerlink" title="keep watch"></a>keep watch</h2><p>工作与心理的交流分享</p>
<h2 id="保持信任"><a href="#保持信任" class="headerlink" title="保持信任"></a>保持信任</h2><h2 id="保持怀疑-1"><a href="#保持怀疑-1" class="headerlink" title="保持怀疑"></a>保持怀疑</h2><h2 id="持续考评"><a href="#持续考评" class="headerlink" title="持续考评"></a>持续考评</h2><h3 id="KPI-VS-OKR"><a href="#KPI-VS-OKR" class="headerlink" title="KPI VS OKR"></a>KPI VS OKR</h3><h2 id="分享中提高"><a href="#分享中提高" class="headerlink" title="分享中提高"></a>分享中提高</h2><p>有一个著名的理论，叫木桶理论。意思是木桶能装多少水，是由木桶最短的那个板来决定的。团队也是如此。团队的力量有多大，很多时候也是由能力最差的那个成员来决定的。为了提高团队整体的实力，我们必须提高每个人的能力。我们的改善必须得是可度量的，所以我们也要数字化能力的标准。</p>
<h2 id="提高工作效率"><a href="#提高工作效率" class="headerlink" title="提高工作效率"></a>提高工作效率</h2><p>其实很多团队都在进行着这个工作。最常见的就是会做一些小工具，来节省我们的时间。比如代码自动生成工具，自动打包工具，自动的比较工具等等。我们应该制作尽可能多的自动化工具，来解放我们的时间。</p>
<p>为了让大家投入更多的热情来制作各种工具，团队可以制定一定的奖励规则，对制作工具的人给予奖励。</p>
<p>公司要把最好的人才放到工具开发那一块，因为工具做好了，可以达到事半功倍的效果，所有人的效率都可以得到提高，而不仅仅是工程师。</p>
<h2 id="互相备份"><a href="#互相备份" class="headerlink" title="互相备份"></a>互相备份</h2><h2 id="互相支撑"><a href="#互相支撑" class="headerlink" title="互相支撑"></a>互相支撑</h2><h2 id="向上管理"><a href="#向上管理" class="headerlink" title="向上管理"></a>向上管理</h2><h2 id="向下管理"><a href="#向下管理" class="headerlink" title="向下管理"></a>向下管理</h2><h3 id="了解队员的擅长与不足"><a href="#了解队员的擅长与不足" class="headerlink" title="了解队员的擅长与不足"></a>了解队员的擅长与不足</h3><h3 id="合理的任务分配"><a href="#合理的任务分配" class="headerlink" title="合理的任务分配"></a>合理的任务分配</h3><h3 id="适度挑战性的任务"><a href="#适度挑战性的任务" class="headerlink" title="适度挑战性的任务"></a>适度挑战性的任务</h3><h3 id="给出完成任务需要的资源"><a href="#给出完成任务需要的资源" class="headerlink" title="给出完成任务需要的资源"></a>给出完成任务需要的资源</h3><h3 id="激励中成长"><a href="#激励中成长" class="headerlink" title="激励中成长"></a>激励中成长</h3><p>眼前利益与发展前景</p>
<h3 id="给出职业发展的机会"><a href="#给出职业发展的机会" class="headerlink" title="给出职业发展的机会"></a>给出职业发展的机会</h3><h3 id="公开与公平中考评"><a href="#公开与公平中考评" class="headerlink" title="公开与公平中考评"></a>公开与公平中考评</h3><h3 id="保持团队稳定"><a href="#保持团队稳定" class="headerlink" title="保持团队稳定"></a>保持团队稳定</h3><p>团队成员的诉求：团队，产品，薪资，发展。</p>