-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
509 lines (438 loc) · 327 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Cat's Blog</title>
<subtitle>一饮一啄，莫非前定.</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="https://amao12580.github.io/"/>
<updated>2017-01-16T02:01:53.964Z</updated>
<id>https://amao12580.github.io/</id>
<author>
<name>Steven Cheng</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>深入浅出Android Chrome浏览器源代码 - 书签功能</title>
<link href="https://amao12580.github.io/post/2017/01/Propose-a-perfect-solution-to-crack-android-chrome-browser's-bookmarks-function/"/>
<id>https://amao12580.github.io/post/2017/01/Propose-a-perfect-solution-to-crack-android-chrome-browser's-bookmarks-function/</id>
<published>2017-01-10T06:43:48.340Z</published>
<updated>2017-01-16T02:01:53.964Z</updated>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>很久没更博了，自从16年国庆节后开始着手<a href="http://www.coolapk.com/apk/pro.kisscat.www.bookmarkhelper" target="_blank" rel="external">书签助手APP</a>的开发，更新了二十余个版本后，正逢年底，工作一下子紧张起来，暂时将APP的开发工作搁置了近一个月。</p>
<h1 id="问题描述"><a href="#问题描述" class="headerlink" title="问题描述"></a>问题描述</h1><p>最近几天有了时间来继续开发，计划17年的第一版更新，完成23款浏览器的书签合并到Android Chrome浏览器，尽最大努力，保持合并后的Chrome浏览器不出现FC、ANR等问题，最好能支持使用Android Google Accout同步到云端。而书签功能作为浏览器的底层基础，很多上层功能需要以此为基点，什么样的方案才能避免蝴蝶效应引起牵一发而动全身呢？</p>
<h1 id="可行性分析"><a href="#可行性分析" class="headerlink" title="可行性分析"></a>可行性分析</h1><p>前提说明：以下内容，大量使用<a href="https://github.com/alibaba/fastjson" target="_blank" rel="external">fastjson</a> api，请提前温习。</p>
<h2 id="Bookmarks文件"><a href="#Bookmarks文件" class="headerlink" title="Bookmarks文件"></a>Bookmarks文件</h2><p>我们都知道，Android Chrome浏览器的书签数据，存储在“/data/data/com.android.chrome/app_chrome/Default/Bookmarks”文件中，下文简称“Bookmarks文件”。文件大小视书签数据量而定，300条书签，接近250KB，将这个文件拷贝出来，简单的使用EditPlus软件打开。一眼看出是经 pretty formatter 处理的json文件，序列化时，为方便阅读，适当加上换行和空格，以体现视觉层次感。</p>
<p><img src="/img/chrome/Bookmarks-Overview.jpg" alt=""></p>
<p>稍加留意，可以知道Bookmarks文件中，书签与文件夹的关系被处理为一棵树形结构。事实上，我在开发Chrome浏览器提取书签，合并到Via浏览器时，就是读取了Bookmarks文件，使用递归将每条书签的文件夹完整路径提取出来。但这毕竟是read-only操作，不涉及到写入。</p>
<p>那么是不是将书签数据，按照既定的json格式组织起来，写入到Bookmarks文件就可以了呢？先暂且认为是可以的。那么如何安全的写入呢？我考虑到有如下几个关键点：</p>
<p>1.写入时，确保Chrome浏览器安全退出，避免多进程同时写，结果相互覆盖。由于GAPPS很有可能同Chrome一起安装，需要写入时监测GMS，避免唤醒Chrome浏览器，比如定期唤醒Chrome同步浏览器数据到云端。</p>
<p><img src="/img/chrome/Google-Account-Sync.jpg" alt=""></p>
<p>2.读取时，确保读取完整已有书签数据，避免在回写时数据丢失。</p>
<p>3.写入时，将已有Bookmarks文件做一次backup，避免程序bug或崩溃（OS Crash）时，可以回退到写入前（restore）。</p>
<p>针对第1点，分为两种情况。</p>
<p>1.如果没有安装其他的GAPPS套件，我们可以在书签合并之前，提示用户保存好浏览数据，然后手动退出（可以是强制停止），也可以由书签助手APP利用Root权限强行杀进程。</p>
<p>2.如果有安装其他的GAPPS套件，这种情况就复杂了，在不卸载的前提下，是否可以调用相关API对GAPPS套件进行完整启动和停止，以避免唤醒Chrome浏览器，可行性是未知的，理论上Google不会提供这样的API。以借用Root权限杀进程(kill -9 ?)的方式强制退出，会引发数据丢失的问题，何况完成书签合并后，还需要将GAPPS启动回来，那将有一段时间的运行中断，例如GMS通知等重要业务将没办法使用。强制退出的GAPPS会在书签合并中途重新启动吗？broadcast receiver 接收到系统event而启动主进程，这是可能的。这个方案目前问题太多，几乎不可行。也许有其他方式可以解决，先搁置这个问题，思考在可以解决的前提下，第2点和第3点该如何应对？</p>
<p>针对第2点，在下一小节里说明。</p>
<p>针对第3点，目前的程序逻辑，已经有了backup策略，但是restore是缺失的，这个作为健壮性设计在下一版本迭代中完成。</p>
<h3 id="读取完整书签数据"><a href="#读取完整书签数据" class="headerlink" title="读取完整书签数据"></a>读取完整书签数据</h3><p>针对上文中的第2点，确保读取完整已有书签数据。Bookmarks文件作为json格式的文件，完整读取，是需要将所有key-value对 都转化为java object，并保持好key-value对 之间的逻辑依赖关系，比如名为folderA的文件夹，共有5个field，包含了2条书签，分别是bookmarkB和bookmarkC，而bookmarkB有12个field，bookmarkC有6个field。别惊讶，下文中我将说明为何field数量出现不一致，这是真实存在的。想要完整读取，就需要这种描述信息由json格式，转化为java object。</p>
<p>在之前的读取时，只需要针对基本的4个field进行解析就可以：name，url，type，children。而且这些field的排列规则是非常固定的。我们可以抽象出来，组成Node对象，更加方便后续的程序逻辑解析。</p>
<p><img src="/img/chrome/Node-old.jpg" alt=""></p>
<p>那为了读取完整的书签数据，这一点也需要调整，整理出全新的Node对象，包含9个field。可以留意到，为了保持序列化的写入顺序，加入了注解：“@JSONField(ordinal = 5)”，这很重要，最小化影响Chrome浏览器功能，经观察Bookmarks文件中的key是以当前深度下的key字母正向顺序写入的。</p>
<p><img src="/img/chrome/Node-new.jpg" alt=""></p>
<h4 id="参差不齐的JSON"><a href="#参差不齐的JSON" class="headerlink" title="参差不齐的JSON"></a>参差不齐的JSON</h4><p>参差不齐的JSON是指统一深度下的节点对象，出现field数量变化。这一点体现在Node对象的meta_info field。</p>
<figure class="highlight less"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="variable">@Setter</span></div><div class="line"><span class="variable">@Getter</span></div><div class="line"><span class="variable">@JSONField</span>(ordinal = <span class="number">5</span>)</div><div class="line">private MetaInfo meta_info;</div></pre></td></tr></table></figure>
<p><img src="/img/chrome/Node-meta_info-noRules.jpg" alt=""></p>
<p>经过仔细观察，遍历一个比较大和复杂的Bookmarks文件，我们发现field数量变化也出现规律性。1.MetaInfo的自身field交叉，field数量不相同的MetaInfo对象，存在交叉关系，部分或完全重叠。2.有的Node对象不存在meta_info field，例如新添加的书签数据，这点在下文中说明。</p>
<p>MetaInfo也是一个对象，整理出可能的field后，得到以下pojo。</p>
<p><img src="/img/chrome/Node-meta_info-pojo.jpg" alt=""></p>
<p>你可以看出来，又有了新的注解：“@JSONField(ordinal = 10, name = synonymsPrefix + “userEdit”)”，由于java对于field的定义规范，不允许“.”。我们将key与field name的映射关系，做了特别处理。</p>
<p>仅MetaInfo对象就有11个field，数量不是问题，关键是无法确定，是否完整罗列了所有field ？</p>
<p>不能接受数据丢失，差之毫厘谬以千里。</p>
<h3 id="JSON-parseObject"><a href="#JSON-parseObject" class="headerlink" title="JSON.parseObject"></a>JSON.parseObject</h3><p>在上一小节，除了GAPPS的问题外，还遇到了meta_info对象不确定的问题，接下来solve it。</p>
<p>有没有可能在不定义pojo或不完整定义pojo的前提下，解析json string呢？在此基础上经过修改，还能在序列化为string不丢失数据呢？</p>
<p>这有点像在原有的json string上，再合适的位置，打开一道口子，放进新的书签数据段，然后重新缝合起来，变成一个全新的json string，就像是给计算机多加一个内存条，增加的不影响已有的，并且能协作一起运行。</p>
<p>多方尝试发现是可以的，整理出如下代码段，可以完成在“bookmark_bar”节点下，添加一个书签，进行序列化时，没有数据丢失，原有的格式被完整保留。</p>
<figure class="highlight haxe"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div></pre></td><td class="code"><pre><div class="line"><span class="comment">//将Bookmarks文件，完整读取到内存</span></div><div class="line"><span class="keyword">String</span> content = FileUtil.read(tmpFilePath);</div><div class="line"><span class="keyword">if</span> (content == <span class="literal">null</span> || content.isEmpty()) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:content is null or empty."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//解析为JSONObject，保持原有key排列顺序</span></div><div class="line">JSONObject top = JSON.parseObject(content, Feature.OrderedField);</div><div class="line"><span class="keyword">if</span> (top == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:top is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//获取名为roots的子节点</span></div><div class="line">JSONObject roots = top.getJSONObject(<span class="string">"roots"</span>);</div><div class="line"><span class="keyword">if</span> (roots == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:roots is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//获取名为bookmark_bar的子节点</span></div><div class="line">JSONObject bookmarkBar = roots.getJSONObject(<span class="string">"bookmark_bar"</span>);</div><div class="line"><span class="keyword">if</span> (bookmarkBar == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:bookmarkBar is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line"><span class="comment">//获取名为children的子节点</span></div><div class="line">JSONArray children = bookmarkBar.getJSONArray(<span class="string">"children"</span>);</div><div class="line"><span class="keyword">if</span> (children == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:children is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div><div class="line">LogHelper.v(<span class="string">"children origin size:"</span> + children.size());</div><div class="line"></div><div class="line">Node node = <span class="keyword">new</span> <span class="type">Node</span>();</div><div class="line">node.setName(<span class="string">"test01"</span>);</div><div class="line">node.setUrl(<span class="string">"http://www.github.com"</span>);</div><div class="line">node.setType(MetaData.FOLDER_TYPE_DEFAULT_NAME);</div><div class="line">node.setId(<span class="string">"99999999999999999"</span>);</div><div class="line"><span class="comment">//添加一个书签，名为：“test01”</span></div><div class="line">children.add(node);</div><div class="line"><span class="comment">//覆盖原有的children节点</span></div><div class="line">JsonUtil.replace(bookmarkBar, <span class="string">"children"</span>, children);</div><div class="line"><span class="comment">//关于checksum的处理下文说明</span></div><div class="line">JsonUtil.update(top, <span class="string">"checksum"</span>, <span class="string">""</span>);</div><div class="line"><span class="comment">//获取添加书签后的完整json数据</span></div><div class="line"><span class="keyword">String</span> finalContent = JsonUtil.toJson(top, <span class="literal">true</span>);</div><div class="line"><span class="keyword">if</span> (finalContent == <span class="literal">null</span>) {</div><div class="line"> LogHelper.e(TAG + <span class="string">".appendBookmark:finalContent is null."</span>);</div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">ConverterException</span>(ContextUtil.buildAppendBookmarksErrorMessage(<span class="built_in">this</span>.getName()));</div><div class="line">}</div></pre></td></tr></table></figure>
<p>效果图：<br><img src="/img/chrome/Bookmarks-addOne-fake.jpg" alt=""></p>
<h3 id="字段的分析"><a href="#字段的分析" class="headerlink" title="字段的分析"></a>字段的分析</h3><p>按上一小节的代码段逻辑，我们对原有的书签json string，是不做修改的，这样是否可以被chrome浏览器识别为正常完整的新Bookmarks文件呢？经模拟器（Genymotion：V2.8.1）环境测试（完整安装GAPPS），是不可行的。chrome浏览器的书签页面，不显示新加的书签“test01”，但在Bookmarks文件中确实是存在的，这说明chrome浏览器不认可这一数据段。</p>
<p>经查证，修改完Bookmarks文件后，恢复启动chrome浏览器，新书签“test01”数据段被删除，这说明chrome浏览器在启动时，有verify机制和rollback机制，检测到不正常的书签数据hack，自动恢复到上一个savepoint，那这些机制的运作细节是怎样的呢？如何应对？</p>
<p>同时发现，chrome浏览器页面中，无法再添加新书签了，提示：“无法添加书签。”，这很可能是在异常修改后，chrome浏览器的一种自我防御机制，还只是猜想。</p>
<p>也许完整分析Bookmarks文件的字段会有一些灵感？在字面上来分析业务含义，很容易知道大部分字段的功能性和算法。</p>
<figure class="highlight delphi"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">String</span> id;<span class="comment">//【猜测】唯一主键，数据来源不明，经观察，存在自增特点，步长不确定。同级路径从上到下，存在顺序性，值大于内层目录id值</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> <span class="keyword">name</span>;<span class="comment">//书签或者文件夹名称</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> url;<span class="comment">//书签的url，文件夹无此field</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> <span class="keyword">type</span>;<span class="comment">//类型，区分是书签还是文件夹，固定枚举值</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> date_added;<span class="comment">//数据被添加时的时间戳，固定存在</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> date_modified;<span class="comment">//数据被修改时的时间戳，不一定存在。如果修改，则该书签或文件夹新增该field，同时上级路径也会有该field</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> checksum;<span class="comment">//【猜测】roots节点才有，书签完整性和有效性校验值，具体算法不明。</span></div><div class="line"></div><div class="line">Integer version;<span class="comment">//【猜测】roots节点才有，版本？具体产生算法不明。</span></div><div class="line"></div><div class="line"><span class="keyword">String</span> sync_transaction_version;<span class="comment">//【猜测】chrome浏览器的书签想要同步到云端，就需要这个字段控制版本</span></div><div class="line"></div><div class="line"><span class="keyword">Object</span> meta_info;<span class="comment">//【猜测】观察仅书签才有，可能记录了书签的元数据。</span></div><div class="line"></div><div class="line"><span class="keyword">Object</span>[] children;<span class="comment">//文件夹才有，描述该文件夹所包含的书签数据</span></div></pre></td></tr></table></figure>
<p>未知的字段，大概占了50%左右，仅仅从字面含义观察，已经不能满足分析要求了。</p>
<p>可以看出来，chrome浏览器对于运行稳定性以及健壮性，有很多的考虑，如何不破坏这些机制，完成书签添加呢？</p>
<h2 id="黑盒测试验证"><a href="#黑盒测试验证" class="headerlink" title="黑盒测试验证"></a>黑盒测试验证</h2><p>在静态分析meta_info field 阶段时，意外的留意到，不是每个字段都会出现，呈现不整齐的现象。那在实际的chrome浏览器中，新增书签或文件夹时是否也存在呢？如果存在，意味着有些字段的产生规则和业务含义可以不必理会，这需要在模拟器环境来验证猜想。</p>
<h3 id="新书签"><a href="#新书签" class="headerlink" title="新书签"></a>新书签</h3><p>在模拟器环境，完整GAPPS套件安装，新安装最新版chrome浏览器，完成Google account 同步【FQ】，包括浏览器信息同步。此时正常完整退出chrome浏览器，保存一个Bookmarks文件副本。打开chrome浏览器，在“书签栏”路径下，新加一个书签，名为“test02”（新书签，之前是不存在的），正常完整的退出chrome浏览器，回过头来再看Bookmarks文件的变化。</p>
<p>添加书签：<br><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-bookmark-one.jpg" alt=""></p>
<p>Bookmarks文件变化：</p>
<p>第1部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-bookmark-one-after-change.jpg" alt=""></p>
<p>第2部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-bookmark-one-after-change-checksum.jpg" alt=""></p>
<p>我们可以发现，新的书签，不仅仅影响局部数据，像原有的date_modified、checksum等值也会跟随变化。好消息是，观察新书签数据结构，发现新书签数据段，字段结构并不复杂，大部分字段可以由外部程序提供，但还是存在疑点：“id”、“sync_transaction_version”、“checksum”等产生规则是什么呢？</p>
<h3 id="新文件夹"><a href="#新文件夹" class="headerlink" title="新文件夹"></a>新文件夹</h3><p>已经知道新书签的添加，对Bookmarks文件的影响，继续来观察新文件夹对Bookmarks文件的影响。按照上一节的流程添加，只是添加时改为添加文件夹，名为“新建文件夹10”,同时为这个新文件夹，加入一个书签，名为：“test03”。</p>
<p>添加文件夹：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-one.jpg" alt=""></p>
<p>Bookmarks文件变化：</p>
<p>第1部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-bookmark-one-after-change.jpg" alt=""></p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-bookmark-one-after-change2.jpg" alt=""></p>
<p>第2部分：</p>
<p><img src="/img/chrome/genymotion-chrome-bookmark_bar-add-folder-and-bookmark-one-after-change-checksum.jpg" alt=""></p>
<p>同上一小节，“id”、“sync_transaction_version”、“checksum”等字段的产生规则还不明确。</p>
<h3 id="SyncData-sqlite3-不确定问题"><a href="#SyncData-sqlite3-不确定问题" class="headerlink" title="SyncData.sqlite3 不确定问题"></a>SyncData.sqlite3 不确定问题</h3><p>在观察新加书签和新加文件夹的时候，观察的重点对象是Bookmarks文件。经过上文的观察结果，有一些疑问不能仅仅通过Bookmarks文件来得到答案。在观察Bookmarks文件的同时，我们使用文件系统自身的watch机制，来感知其他数据文件的变化，得到了另一重点对象：SyncData.sqlite3，它的路径位于：“/data/data/com.android.chrome/app_chrome/Default/Sync Data/SyncData.sqlite3”，经分析，这是一个典型的sqlite3数据库。</p>
<p>将这个文件拷贝出来，用Navicat Premium（11.1.12）打开，果然是不需要密码，可以直接打开查看。</p>
<p><img src="/img/chrome/SyncData-sqlite3-overview.jpg" alt=""></p>
<p>5张表的结构也出来了<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">CREATE TABLE deleted_metas (metahandle bigint primary key ON CONFLICT FAIL,base_version bigint<span class="built_in"> default </span>-1,server_version bigint<span class="built_in"> default </span>0,local_external_id bigint<span class="built_in"> default </span>0,transaction_version bigint<span class="built_in"> default </span>0,mtime bigint<span class="built_in"> default </span>0,server_mtime bigint<span class="built_in"> default </span>0,ctime bigint<span class="built_in"> default </span>0,server_ctime bigint<span class="built_in"> default </span>0,id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,server_parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,is_unsynced bit<span class="built_in"> default </span>0,is_unapplied_update bit<span class="built_in"> default </span>0,is_del bit<span class="built_in"> default </span>0,is_dir bit<span class="built_in"> default </span>0,server_is_dir bit<span class="built_in"> default </span>0,server_is_del bit<span class="built_in"> default </span>0,non_unique_name varchar,server_non_unique_name varchar(255),unique_server_tag varchar,unique_client_tag varchar,unique_bookmark_tag varchar,specifics blob,server_specifics blob,base_server_specifics blob,server_unique_position blob,unique_position blob,attachment_metadata blob,server_attachment_metadata blob);</div><div class="line"></div><div class="line">CREATE TABLE metas(metahandle bigint primary key ON CONFLICT FAIL,base_version bigint<span class="built_in"> default </span>-1,server_version bigint<span class="built_in"> default </span>0,local_external_id bigint<span class="built_in"> default </span>0,transaction_version bigint<span class="built_in"> default </span>0,mtime bigint<span class="built_in"> default </span>0,server_mtime bigint<span class="built_in"> default </span>0,ctime bigint<span class="built_in"> default </span>0,server_ctime bigint<span class="built_in"> default </span>0,id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,server_parent_id varchar(255)<span class="built_in"> default </span><span class="string">"r"</span>,is_unsynced bit<span class="built_in"> default </span>0,is_unapplied_update bit<span class="built_in"> default </span>0,is_del bit<span class="built_in"> default </span>0,is_dir bit<span class="built_in"> default </span>0,server_is_dir bit<span class="built_in"> default </span>0,server_is_del bit<span class="built_in"> default </span>0,non_unique_name varchar,server_non_unique_name varchar(255),unique_server_tag varchar,unique_client_tag varchar,unique_bookmark_tag varchar,specifics blob,server_specifics blob,base_server_specifics blob,server_unique_position blob,unique_position blob,attachment_metadata blob,server_attachment_metadata blob);</div><div class="line"></div><div class="line">CREATE TABLE models (model_id BLOB primary key, progress_marker BLOB, transaction_version BIGINT<span class="built_in"> default </span>0,context BLOB);</div><div class="line"></div><div class="line">CREATE TABLE share_info (id TEXT primary key, name TEXT, store_birthday TEXT, cache_guid TEXT, bag_of_chips BLOB);</div><div class="line"></div><div class="line">CREATE TABLE share_version (id VARCHAR(128) primary key, data INT);</div></pre></td></tr></table></figure></p>
<p>经过对比分析，metas表，是主要观察对象，这个表的数据量1000+，保存了多处登录的所有设备上：浏览历史记录、当前打开的标签页、书签信息。当数据<br>新加书签“test02”的对应记录：</p>
<p><img src="/img/chrome/SyncData-sqlite3-metas-bookmarks-newBookmark-test02.jpg" alt=""></p>
<p>新加文件夹“test03”的对应记录：</p>
<p><img src="/img/chrome/SyncData-sqlite3-metas-folder-newFolder-test03.jpg" alt=""></p>
<p>很明显的，metas.local_external_id 字段，对应着Bookmarks文件中，id字段；metas.transaction_version 字段，对应着Bookmarks文件中，sync_transaction_version字段，checksum字段暂时没有找到映射关系。</p>
<p>还发现，metas表中，有10多个字段尚有疑问，</p>
<figure class="highlight applescript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">“metahandle”，“local_external_id”，“transaction_version”，</div><div class="line">“unique_bookmark_tag”，“specifics”，“server_specifics”，</div><div class="line">“server_unique_position”，“unique_position”，“<span class="built_in">id</span>”，“parent_id”，</div><div class="line">“server_parent_id”</div></pre></td></tr></table></figure>
<p>很容易从字面上猜测到业务含义，但这些字段的产生算法是怎么样的呢？</p>
<p>还好在上一小节中的：“id”、“sync_transaction_version”这两个字段在SyncData.sqlite3文件中得到印证。总算没有白费功夫，但是“checksum”是什么呢？猜测的是否有误呢？</p>
<p>checksum字段的算法，经google找到一小部分非官方说明：<a href="http://stackoverflow.com/questions/11308603/logic-behind-creating-bookmark-checksum-in-google-chrome" target="_blank" rel="external">http://stackoverflow.com/questions/11308603/logic-behind-creating-bookmark-checksum-in-google-chrome</a>。关键的代码段在chromium开源项目中：<a href="https://chromium.googlesource.com/chromium/chromium/+/20f8aa123f98b2bcb0d346af0d78ad7a8ddea5d0/chrome/browser/bookmarks/bookmark_codec.cc" target="_blank" rel="external">https://chromium.googlesource.com/chromium/chromium/+/20f8aa123f98b2bcb0d346af0d78ad7a8ddea5d0/chrome/browser/bookmarks/bookmark_codec.cc</a></p>
<figure class="highlight xl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div></pre></td><td class="code"><pre><div class="line">Value* BookmarkCodec::Encode(BookmarkModel* model) {</div><div class="line"> <span class="function"><span class="title">return</span> Encode(model-></span>bookmark_bar_node(),</div><div class="line"> <span class="function"><span class="title">model</span>-></span>other_node(),</div><div class="line"> <span class="function"><span class="title">model</span>-></span>mobile_node(),</div><div class="line"> <span class="function"><span class="title">model</span>-></span><span class="function"><span class="title">root_node</span>()-></span>meta_info_str());</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line">Value* BookmarkCodec::Encode(const BookmarkNode* bookmark_bar_node,</div><div class="line"> const BookmarkNode* other_folder_node,</div><div class="line"> const BookmarkNode* mobile_folder_node,</div><div class="line"> const std::string& model_meta_info) {</div><div class="line"> ids_reassigned_ = <span class="literal">false</span>;</div><div class="line"> InitializeChecksum();</div><div class="line"> DictionaryValue* roots = new DictionaryValue();</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>Set(kRootFolderNameKey, EncodeNode(bookmark_bar_node));</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>Set(kOtherBookmarkFolderNameKey, EncodeNode(other_folder_node));</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>Set(kMobileBookmarkFolderNameKey, EncodeNode(mobile_folder_node));</div><div class="line"> <span class="keyword">if</span> (!model_meta_info.empty())</div><div class="line"> <span class="function"><span class="title">roots</span>-></span>SetString(kMetaInfo, model_meta_info);</div><div class="line"> DictionaryValue* main = new DictionaryValue();</div><div class="line"> <span class="function"><span class="title">main</span>-></span>SetInteger(kVersionKey, kCurrentVersion);</div><div class="line"> FinalizeChecksum();</div><div class="line"> <span class="comment">// We are going to store the computed checksum. So set stored checksum to be</span></div><div class="line"> <span class="comment">// the same as computed checksum.</span></div><div class="line"> stored_checksum_ = computed_checksum_;</div><div class="line"> <span class="function"><span class="title">main</span>-></span>Set(kChecksumKey, new base::StringValue(computed_checksum_));</div><div class="line"> <span class="function"><span class="title">main</span>-></span>Set(kRootsKey, roots);</div><div class="line"> return main;</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line">Value* BookmarkCodec::EncodeNode(const BookmarkNode* node) {</div><div class="line"> DictionaryValue* value = new DictionaryValue();</div><div class="line"> <span class="function"><span class="title">std</span>::string id = base::Int64ToString(node-></span>id());</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kIdKey, id);</div><div class="line"> <span class="function"><span class="title">const</span> string16& <span class="built_in">title</span> = node-></span>GetTitle();</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kNameKey, <span class="built_in">title</span>);</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kDateAddedKey,</div><div class="line"> <span class="function"><span class="title">base</span>::Int64ToString(node-></span>date_added().ToInternalValue()));</div><div class="line"> <span class="function"><span class="title">if</span> (node-></span>is_url()) {</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kTypeKey, kTypeURL);</div><div class="line"> <span class="function"><span class="title">std</span>::string url = node-></span>url().possibly_invalid_spec();</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kURLKey, url);</div><div class="line"> UpdateChecksumWithUrlNode(id, <span class="built_in">title</span>, url);</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kTypeKey, kTypeFolder);</div><div class="line"> <span class="function"><span class="title">value</span>-></span>SetString(kDateModifiedKey,</div><div class="line"> <span class="function"><span class="title">base</span>::Int64ToString(node-></span>date_folder_modified().</div><div class="line"> ToInternalValue()));</div><div class="line"> UpdateChecksumWithFolderNode(id, <span class="built_in">title</span>);</div><div class="line"> ListValue* child_values = new ListValue();</div><div class="line"> <span class="function"><span class="title">value</span>-></span>Set(kChildrenKey, child_values);</div><div class="line"> <span class="function"><span class="title">for</span> (int i = 0; i < node-></span>child_count(); ++i)</div><div class="line"> <span class="function"><span class="title">child_values</span>-></span>A<span class="function"><span class="title">ppend</span>(EncodeNode(node-></span>GetChild(i)));</div><div class="line"> }</div><div class="line"> <span class="function"><span class="title">if</span> (!node-></span>meta_info_str().empty())</div><div class="line"> <span class="function"><span class="title">value</span>-></span>S<span class="function"><span class="title">etString</span>(kMetaInfo, node-></span>meta_info_str());</div><div class="line"> return value;</div><div class="line">}</div><div class="line"></div><div class="line">void BookmarkCodec::FinalizeChecksum() {</div><div class="line"> base::MD5Digest digest;</div><div class="line"> base::MD5Final(&digest, &md5_context_);</div><div class="line"> computed_checksum_ = base::MD5DigestToBase16(digest);</div><div class="line">}</div></pre></td></tr></table></figure>
<p>进行语义逻辑分析，可以看出来，Encode方法将bookmark_bar_node对象，other_node对象，mobile_node对象，meta_info_str对象作为入参，多维度进行摘要算法计算，字段结果就是checksum值。也就是说，Bookmarks文件中，任意书签的任意字段信息发生改变，checksum跟随变化，意思是所有书签数据的hashTag。摘要算法的细节还不清楚，在下文中说明。</p>
<h2 id="思路转换"><a href="#思路转换" class="headerlink" title="思路转换"></a>思路转换</h2><p>真是一波未平一波又起，chrome浏览器的复杂性远远超出预期，看来简单的通过分析android data目录文件，没办法满足需求了。换个角度思考，chrome浏览器的复杂性和稳定性是同时提供的，这意味着内部的复杂度被chrome上层程序掩盖。还知道chrome pc版是支持<a href="https://chrome.google.com/webstore/category/extensions?hl=zh-CN" target="_blank" rel="external">extension</a>的，这是一大亮点，意味着pc版开放了一些api给extension调用，辅助chrome运行。那有没有可能，由android chrome浏览器提供一个书签管理api，开放给书签助手APP来调用呢？</p>
<h2 id="content-provider"><a href="#content-provider" class="headerlink" title="content provider"></a>content provider</h2><p>通过一番Google搜索（通过google解决google的问题？），找到了可能的解决办法：<a href="http://stackoverflow.com/questions/14433480/add-bookmark-to-chrome-browser-on-android?rq=1" target="_blank" rel="external">Add bookmark to chrome browser on Android</a>。原理是android chrome浏览器开放了一对权限来管理书签和历史浏览：com.android.browser.permission.READ_HISTORY_BOOKMARKS，com.android.browser.permission.WRITE_HISTORY_BOOKMARKS，app在获取这些权限后，可以通过content provider的方式，将书签数据组织为chrome浏览器需要的格式，提交给chrome浏览器来存储。</p>
<p>“com.android.browser.permission.READ_HISTORY_BOOKMARKS”，该权限被用作查询chrome浏览器的书签和历史浏览记录，可以按条件过滤，<a href="http://stackoverflow.com/questions/31872028/which-is-the-alternate-permission-for-reading-bookmarks-in-android-m" target="_blank" rel="external">这个权限在6..0及以上的android版本已经被删除，意味着不再可用。</a></p>
<p>这2个的api由android chrome浏览器官方提供，可行性高了不少，附带的GAPPS问题和同步到云端的问题，也屏蔽了解构底层Bookmarks文件和SyncData.sqlite3文件。问题可以就此解决吗？</p>
<h3 id="solution-demo"><a href="#solution-demo" class="headerlink" title="solution demo"></a>solution demo</h3><p>整理出添加书签的demo代码段：<br><figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">private</span> <span class="keyword">void</span> test(){</div><div class="line"> ContentValues values = <span class="keyword">new</span> ContentValues();</div><div class="line"> values.put(BookmarkColumns.TITLE, <span class="string">"title"</span>);</div><div class="line"> values.put(BookmarkColumns.URL, <span class="string">"http://ss.com"</span>);</div><div class="line"> values.put(BookmarkColumns.BOOKMARK, <span class="number">1</span>);</div><div class="line"> values.put(BookmarkColumns.CREATED, <span class="number">0</span>);</div><div class="line"> values.put(BookmarkColumns.DATE, <span class="number">0</span>);</div><div class="line"> values.put(<span class="string">"parentId"</span>, <span class="number">3</span>); <span class="comment">// just for Chrome</span></div><div class="line"> getContentResolver().insert(Uri.parse(<span class="string">"content://com.android.chrome.ChromeBrowserProvider/bookmarks"</span>), values);</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">public</span> <span class="keyword">static</span> class BookmarkColumns implements BaseColumns {</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> URL = <span class="string">"url"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> VISITS = <span class="string">"visits"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> DATE = <span class="string">"date"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> BOOKMARK = <span class="string">"bookmark"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> TITLE = <span class="string">"title"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> CREATED = <span class="string">"created"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> FAVICON = <span class="string">"favicon"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> THUMBNAIL = <span class="string">"thumbnail"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> TOUCH_ICON = <span class="string">"touch_icon"</span>;</div><div class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">String</span> USER_ENTERED = <span class="string">"user_entered"</span>;</div><div class="line">}</div></pre></td></tr></table></figure></p>
<p>添加文件夹的api也是类似的写法吗？google没有找到印证。</p>
<h3 id="chromium源代码"><a href="#chromium源代码" class="headerlink" title="chromium源代码"></a>chromium源代码</h3><p>想起来chrome是基于chromium开源项目的，那content provider这部分的代码也是开源的吗？找到chromium源代码online page:<a href="https://www.chromium.org/developers/how-tos/get-the-code" target="_blank" rel="external">https://www.chromium.org/developers/how-tos/get-the-code</a>，选择<a href="https://chromium.googlesource.com/chromium/src/+/master/docs/android_build_instructions.md" target="_blank" rel="external">Android</a>版，先不下载，翻阅相关代码段：<a href="https://chromium.googlesource.com/chromium/src/+/master/chrome/android/java/src/org/chromium/chrome/browser/provider/ChromeBrowserProvider.java" target="_blank" rel="external">https://chromium.googlesource.com/chromium/src/+/master/chrome/android/java/src/org/chromium/chrome/browser/provider/ChromeBrowserProvider.java</a>，405行。</p>
<figure class="highlight groovy"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">public</span> Uri insert(Uri uri, ContentValues values) {</div><div class="line"> <span class="keyword">if</span> (!canHandleContentProviderApiCall() || !hasWriteAccess()) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">int</span> match = mUriMatcher.match(uri);</div><div class="line"> Uri res = <span class="literal">null</span>;</div><div class="line"> <span class="keyword">long</span> id;</div><div class="line"> <span class="keyword">switch</span> (match) {</div><div class="line"> <span class="keyword">case</span> <span class="string">URI_MATCH_BOOKMARKS:</span></div><div class="line"> id = addBookmark(values);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_BOOKMARK_ID) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_BOOKMARK_CONTENT:</span></div><div class="line"> values.put(BookmarkColumns.BOOKMARK, <span class="number">1</span>);</div><div class="line"> <span class="comment">//$FALL-THROUGH$</span></div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_BOOKMARK:</span></div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_HISTORY_CONTENT:</span></div><div class="line"> id = addBookmarkFromAPI(values);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_CONTENT_PROVIDER_ID) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"> <span class="keyword">case</span> <span class="string">URL_MATCH_API_SEARCHES:</span></div><div class="line"> id = addSearchTermFromAPI(values);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_CONTENT_PROVIDER_ID) <span class="keyword">return</span> <span class="literal">null</span>;</div><div class="line"> <span class="keyword">break</span>;</div><div class="line"><span class="symbol"> default:</span></div><div class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> IllegalArgumentException(TAG + <span class="string">": insert - unknown URL "</span> + uri);</div><div class="line"> }</div><div class="line"> res = ContentUris.withAppendedId(uri, id);</div><div class="line"> notifyChange(res);</div><div class="line"> <span class="keyword">return</span> res;</div><div class="line">}</div></pre></td></tr></table></figure>
<p>正好对应了content provider部分的对接部分，使用git将chromium android源代码clone下来，因为网速影响，只拉取master分支的最后一个commit版本，history change都先不下载了，约2.5GB。</p>
<figure class="highlight crmsh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">git <span class="keyword">clone</span> <span class="title">https</span>://chromium.googlesource.com/chromium/src -b <span class="keyword">master</span> <span class="title">--depth</span>=<span class="number">1</span></div></pre></td></tr></table></figure>
<p>android源代码，使用Android Studio浏览应该是最好的选择，因为机器配置（4核奔腾+8GB）不够强，对如此浩瀚的代码仓库，编译索引起来力不从心，一度死机。尝试缩减源代码范围，只编译“\chromium\src\chrome\android”部分，约30MB，很快就打开了，但是依赖包没有编译，无法正常浏览。</p>
<p>针对如此大规模的源代码，有更好的源代码浏览工具吗？可能混杂的编程语言有python、java、c系列。</p>
<h4 id="Source-Insight"><a href="#Source-Insight" class="headerlink" title="Source Insight"></a>Source Insight</h4><p>Source Insight.它是一个面向项目开发的程序编辑器和代码浏览器。Source Insight能分析你的源代码并在你工作的同时动态维护它自己的符号数据库，并自动为你显示有用的上下文信息。 它的强大之处在于不仅仅是可编辑的源代码，还包括对于代码中的变量和类进行关联和查找。比如java语言，你可以清晰的看到一个类中的成员变量以及方法，而且source insight 还提供了类的预览，比如源码中有一个类，那么你可以解转到那个类里查看源码。</p>
<p>Source Insight支持的编程语言列表：<br><img src="/img/chrome/sourceInsight-support-language.jpg" alt=""></p>
<h4 id="ChromeBrowserProvider-java"><a href="#ChromeBrowserProvider-java" class="headerlink" title="ChromeBrowserProvider.java"></a>ChromeBrowserProvider.java</h4><p>上一小节，已经提到ChromeBrowserProvider.java文件，这一节的就此着手，抽丝剥茧，一路挖下去吧！</p>
<p>当入参uri，是URI_MATCH_BOOKMARKS时，进入到addBookmark(values)代码段，URI_MATCH_BOOKMARKS定义：private static final int URI_MATCH_BOOKMARKS = 0;</p>
<h5 id="addBookmark方法"><a href="#addBookmark方法" class="headerlink" title="addBookmark方法"></a>addBookmark方法</h5><p>接下来看addBookmark方法：</p>
<figure class="highlight maxima"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line">private long addBookmark(ContentValues <span class="built_in">values</span>) {</div><div class="line"> String url = <span class="built_in">values</span>.getAsString(BookmarkColumns.URL);</div><div class="line"> String <span class="built_in">title</span> = <span class="built_in">values</span>.getAsString(BookmarkColumns.TITLE);</div><div class="line"> boolean isFolder = <span class="literal">false</span>;</div><div class="line"> <span class="keyword">if</span> (<span class="built_in">values</span>.containsKey(BOOKMARK_IS_FOLDER_PARAM)) {</div><div class="line"> isFolder = <span class="built_in">values</span>.getAsBoolean(BOOKMARK_IS_FOLDER_PARAM);</div><div class="line"> }</div><div class="line"> long parentId = INVALID_BOOKMARK_ID;</div><div class="line"> <span class="keyword">if</span> (<span class="built_in">values</span>.containsKey(BOOKMARK_PARENT_ID_PARAM)) {</div><div class="line"> parentId = <span class="built_in">values</span>.getAsLong(BOOKMARK_PARENT_ID_PARAM);</div><div class="line"> }</div><div class="line"> long id = nativeAddBookmark(mNativeChromeBrowserProvider, url, <span class="built_in">title</span>, isFolder, parentId);</div><div class="line"> <span class="keyword">if</span> (id == INVALID_BOOKMARK_ID) <span class="built_in">return</span> id;</div><div class="line"></div><div class="line"> <span class="keyword">if</span> (isFolder) {</div><div class="line"> updateLastModifiedBookmarkFolder(id);</div><div class="line"> } <span class="keyword">else</span> {</div><div class="line"> updateLastModifiedBookmarkFolder(parentId);</div><div class="line"> }</div><div class="line"> <span class="built_in">return</span> id;</div><div class="line">}</div></pre></td></tr></table></figure>
<p>第2、3两行，分别获取传入的URL和标题字段值，不做多说。第4至11行，验证是否为文件夹，以及获取上层文件夹的id值，主要是为了第15至19行，进行文件夹的最后更新时间做计算，猜测对应前文的date_modified字段，进行级联修改。BOOKMARK_IS_FOLDER_PARAM与BOOKMARK_PARENT_ID_PARAM的定义如下。<br><figure class="highlight php"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="comment">/** The parameter used to specify whether this is a bookmark folder. */</span></div><div class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String BOOKMARK_IS_FOLDER_PARAM = <span class="string">"isFolder"</span>;</div><div class="line"></div><div class="line"><span class="comment">/** The parameter used to specify a bookmark parent ID in ContentValues. */</span></div><div class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String BOOKMARK_PARENT_ID_PARAM = <span class="string">"parentId"</span>;</div></pre></td></tr></table></figure></p>
<p>可以很明显的感受到Google对Java代码的规范性，类名采用大驼峰；方法名、参数名、局部变量、成员变量名，采用小驼峰；常量名采用全大写蛇形，这里有一份完整的:<a href="http://www.hawstein.com/posts/google-java-style.html" target="_blank" rel="external">Google Java编程风格指南</a></p>
<h4 id="nativeAddBookmark方法"><a href="#nativeAddBookmark方法" class="headerlink" title="nativeAddBookmark方法"></a>nativeAddBookmark方法</h4><p>上文中的第12行代码，调用nativeAddBookmark方法后，返回long类型的id字段。这个方法对应的源码段是：<br><figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">private</span> <span class="keyword">native</span> <span class="keyword">long</span> nativeAddBookmark(<span class="keyword">long</span> nativeChromeBrowserProvider,</div><div class="line"> <span class="keyword">String</span> url, <span class="keyword">String</span> title, <span class="built_in">boolean</span> isFolder, <span class="keyword">long</span> parentId);</div></pre></td></tr></table></figure></p>
<p>方法定义为native，从这一步开始，书签添加的逻辑实现，从Java层交棒给了C层。第一个想到的是在同级目录下会有一个：org_chromium_chrome_browser_provider_nativeAddBookmark.h文件，但是没找到。</p>
<p>在“\chromium\src\chrome\android\”目录下，并没有关于jni的代码实现，我们使用EditPlus，对整个chromium src层级目录“\chromium\src\”，进行递归搜索“nativeAddBookmark”关键字，范围限定为：“<em>.cc </em>.h”。耗时 1 分 14 秒完成搜索，但一无所获，难道jni部分并未开源吗？或者在其他路径下开源，未囊括到chromium项目？前一种有可能，后一种可能性很低。</p>
<p>我们都知道，JNI函数的注册有两种方法，一种是静态方法，需要用javah为每个声明了native函数的java类编译出的class文件生成一个头文件，另一种是动态注册，通过数据结构保存关联关系实现注册。搜索“nativeAddBookmark”关键字进行jni函数查找的方式，是以静态方法为前提的。那目标jni函数可能是采用动态的方式注册的吗？</p>
<p>在搜索jni函数的过程中，发现android与ios（”\chromium\src\ios\chrome）共用了同一jni函数，不知道pc版是不是也会共用？一方面性能相比上层代码高，另一方面OS底层都对c代码有支持，产生了一定的跨平台性。</p>
<h2 id="content-provider事务性思考"><a href="#content-provider事务性思考" class="headerlink" title="content provider事务性思考"></a>content provider事务性思考</h2><p>——————-还没写完，持续更新中。2017年1月16日10:01:51——————-</p>
<h1 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h1><h1 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h1><h2 id="思路"><a href="#思路" class="headerlink" title="思路"></a>思路</h2><ul>
<li><p><a href="http://stackoverflow.com/questions/14433480/add-bookmark-to-chrome-browser-on-android?rq=1" target="_blank" rel="external">stackoverflow:about use content provider way to add bookmarks</a></p>
</li>
<li><p><a href="https://developer.android.com/reference/android/provider/Browser.BookmarkColumns.html" target="_blank" rel="external">Browser.BookmarkColumns 释义</a></p>
</li>
</ul>
<h2 id="Source-Insight-使用"><a href="#Source-Insight-使用" class="headerlink" title="Source Insight 使用"></a>Source Insight 使用</h2><ul>
<li><p><a href="http://www.cnblogs.com/octobershiner/archive/2012/03/18/2400805.html" target="_blank" rel="external">source insight 基础使用教程</a></p>
</li>
<li><p><a href="http://zydecx.github.io/2015/05/source-insight-for-java.html" target="_blank" rel="external">source insight 添加jdk源码</a>，android sdk 源码添加同理</p>
</li>
</ul>
<h2 id="Chromium源代码结构"><a href="#Chromium源代码结构" class="headerlink" title="Chromium源代码结构"></a>Chromium源代码结构</h2><ul>
<li><p><a href="http://www.jianshu.com/p/4afe92418bd9" target="_blank" rel="external">Chromium 源代码目录结构 - 1</a></p>
</li>
<li><p><a href="http://hyshucom.iteye.com/blog/1694642" target="_blank" rel="external">Chromium 源代码目录结构 - 2</a></p>
</li>
</ul>
<h2 id="助攻"><a href="#助攻" class="headerlink" title="助攻"></a>助攻</h2><ul>
<li><p><a href="http://stackoverflow.com/questions/22732001/connect-to-vpn-in-genymotion-android" target="_blank" rel="external">genymotion vpn setting</a></p>
</li>
<li><p><a href="https://www.genymotion.com/help/desktop/faq/#network-vpn" target="_blank" rel="external">Why doesn’t Genymotion run with my VPN client?</a></p>
</li>
</ul>
<h1 id="Tips"><a href="#Tips" class="headerlink" title="Tips"></a>Tips</h1><h2 id="Genymotion"><a href="#Genymotion" class="headerlink" title="Genymotion"></a>Genymotion</h2><h3 id="virtual-device-启动过慢"><a href="#virtual-device-启动过慢" class="headerlink" title="virtual device 启动过慢"></a>virtual device 启动过慢</h3><p>有时候virtual device 启动过慢，花了10分钟，还停留在android黑底白字阶段，甚至启动失败。这可能是genymotion自身进程冲突，现在强制退出启动，退出其他genymotion软件，在windows任务管理器，进程列表中，找到adb、vbox相关进程，强制结束。再次启动，发现快了很多！</p>
<h2 id="Linux"><a href="#Linux" class="headerlink" title="Linux"></a>Linux</h2><h3 id="busybox"><a href="#busybox" class="headerlink" title="busybox"></a>busybox</h3><p>BusyBox 是一个集成了一百多个最常用linux命令和工具的软件。BusyBox 包含了一些简单的工具，例如ls、 cat 和 echo等等，还包含了一些更大、更复杂的工具，例如 grep、find、mount 以及 telnet。有些人将 BusyBox 称为 Linux 工具里的瑞士军刀。简单的说BusyBox就好像是个大工具箱，它集成压缩了 Linux 的许多工具和命令。也包含了 Android 系统的自带的shell。</p>
<p>为什么要在Android中加入busybox?</p>
<p>用过adb shell的人应该知道，在默认情况下，adb shell下是不能用clear,grep, find,vi等指令的，甚至连Tab链自动补全功能都不能用，对于已经习惯了使用这些指令的码农们来说，这是件比较悲摧的事情。幸运地是，我们有了busybox!</p>
<p>Genymotion环境下，使用DOS-ADB安装Busybox</p>
<figure class="highlight vim"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div></pre></td><td class="code"><pre><div class="line">download link :http<span class="variable">s:</span>//busybox.net/downloads/binaries/<span class="number">1.21</span>.<span class="number">1</span>/busybox-armv7l</div><div class="line"></div><div class="line">D:\Android\sdk\platform-tools>adb <span class="keyword">shell</span></div><div class="line">root@vbox86p:/ # <span class="keyword">cd</span> /data/</div><div class="line">root@vbox86p:/data # <span class="built_in">mkdir</span> busybox</div><div class="line">root@vbox86p:/data # <span class="keyword">exit</span></div><div class="line"></div><div class="line">D:\Android\sdk\platform-tools>adb push busybox /data/busybox/</div><div class="line">[<span class="number">100</span>%] /data/busybox/busybox</div><div class="line"></div><div class="line">D:\Android\sdk\platform-tools>adb <span class="keyword">shell</span></div><div class="line"></div><div class="line">root@vbox86p:/ # chmod <span class="number">777</span> /data/busybox/busybox</div><div class="line"></div><div class="line">root@vbox86p:/ # /data/busybox/busybox</div><div class="line"></div><div class="line">BusyBox v1.<span class="number">26.2</span> (<span class="number">2017</span>-<span class="number">01</span>-<span class="number">11</span> <span class="number">08</span>:<span class="number">43</span>:<span class="number">16</span> UTC) multi-<span class="keyword">call</span> binary.</div><div class="line">BusyBox <span class="keyword">is</span> copyrighted by many authors between <span class="number">1998</span>-<span class="number">2015</span>.</div><div class="line">Licensed under GPLv2. See <span class="keyword">source</span> distribution <span class="keyword">for</span> detailed</div><div class="line">copyright notices.</div><div class="line"></div><div class="line">Usage: busybox [<span class="function"><span class="keyword">function</span> [<span class="title">arguments</span>]...]</span></div><div class="line"> <span class="built_in">or</span>: busybox --<span class="keyword">list</span>[-full]</div><div class="line"> <span class="built_in">or</span>: busybox --install [-s] [DIR]</div><div class="line"> <span class="built_in">or</span>: <span class="function"><span class="keyword">function</span> [<span class="title">arguments</span>]...</span></div><div class="line"></div><div class="line"> BusyBox <span class="keyword">is</span> <span class="keyword">a</span> multi-<span class="keyword">call</span> binary that combines many common Unix</div><div class="line"> utilities into <span class="keyword">a</span> single <span class="built_in">executable</span>. Most people will create <span class="keyword">a</span></div><div class="line"> link <span class="keyword">to</span> busybox <span class="keyword">for</span> each <span class="function"><span class="keyword">function</span> <span class="title">they</span> <span class="title">wish</span> <span class="title">to</span> <span class="title">use</span> <span class="title">and</span> <span class="title">BusyBox</span></span></div><div class="line"> will act like whatever it was invoked <span class="keyword">as</span>.</div><div class="line"></div><div class="line">Currently defined <span class="keyword">function</span><span class="variable">s:</span></div><div class="line"> [, [[, acpid, <span class="built_in">add</span>-<span class="keyword">shell</span>, addgroup, adduser, adjtimex, arp, arping, ash,</div><div class="line"> awk, base64, basename, beep, blkdiscard, blkid, blockdev, bootchartd,</div><div class="line"> brctl, bunzip2, bzcat, bzip2, <span class="keyword">cal</span>, <span class="keyword">cat</span>, catv, chat, chattr, chgrp,</div><div class="line"> chmod, chown, chpasswd, chpst, chroot, chrt, chvt, cksum, clear, cmp,</div><div class="line"> comm, conspy, <span class="keyword">cp</span>, cpio, crond, crontab, cryptpw, cttyhack, cut, date,</div><div class="line"> dc, dd, deallocvt, delgroup, deluser, depmod, devmem, df, dhcprelay,</div><div class="line"> diff, dirname, dmesg, dnsd, dnsdomainname, dos2unix, dpkg, dpkg-<span class="keyword">deb</span>,</div><div class="line"> du, dumpkmap, dumpleases, <span class="keyword">echo</span>, ed, egrep, eject, env, envdir,</div><div class="line"> envuidgid, ether-wake, <span class="built_in">expand</span>, expr, fakeidentd, false, fatattr, fbset,</div><div class="line"> fbsplash, fdflush, fdformat, fdisk, fgconsole, fgrep, <span class="keyword">find</span>, findfs,</div><div class="line"> flock, <span class="keyword">fold</span>, free, freeramdisk, fsck, fsck.minix, fstrim, fsync, ftpd,</div><div class="line"> ftpget, ftpput, fuser, getopt, getty, <span class="keyword">grep</span>, groups, gunzip, gzip, halt,</div><div class="line"> hd, hdparm, head, hexdump, hostid, <span class="built_in">hostname</span>, httpd, hush, hwclock,</div><div class="line"> i2cdetect, i2cdump, i2cget, i2cset, id, ifconfig, ifdown, ifenslave,</div><div class="line"> ifplugd, ifup, inetd, init, insmod, install, ionice, iostat, ip,</div><div class="line"> ipaddr, ipcalc, ipcrm, ipcs, iplink, ipneigh, iproute, iprule,</div><div class="line"> iptunnel, kbd_mode, kill, killall, killall5, klogd, less, linux32,</div><div class="line"> linux64, linuxrc, <span class="keyword">ln</span>, loadfont, loadkmap, logger, login, logname,</div><div class="line"> logread, losetup, lpd, lpq, lpr, <span class="keyword">ls</span>, lsattr, lsmod, lsof, lspci, lsusb,</div><div class="line"> lzcat, lzma, lzop, lzopcat, makedevs, makemime, man, md5sum, mdev,</div><div class="line"> mesg, microcom, <span class="built_in">mkdir</span>, mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.minix,</div><div class="line"> mkfs.vfat, mknod, mkpasswd, mkswap, mktemp, modinfo, modprobe, more,</div><div class="line"> mount, mountpoint, mpstat, mt, mv, nameif, nanddump, nandwrite,</div><div class="line"> nbd-client, nc, netstat, nice, nmeter, nohup, nslookup, ntpd, od,</div><div class="line"> openvt, passwd, patch, pgrep, pidof, ping, ping6, pipe_progress,</div><div class="line"> pivot_root, pkill, pmap, popmaildir, poweroff, powertop, printenv,</div><div class="line"> <span class="built_in">printf</span>, <span class="keyword">ps</span>, pscan, pstree, <span class="keyword">pwd</span>, pwdx, raidautorun, rdate, rdev,</div><div class="line"> readahead, readlink, readprofile, realpath, reboot, reformime,</div><div class="line"> <span class="built_in">remove</span>-<span class="keyword">shell</span>, renice, reset, <span class="keyword">resize</span>, rev, rm, rmdir, rmmod, route, rpm,</div><div class="line"> rpm2cpio, rtcwake, run-parts, runsv, runsvdir, rx, script,</div><div class="line"> scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont,</div><div class="line"> setkeycodes, setlogcons, setserial, setsid, setuidgid, <span class="keyword">sh</span>, sha1sum,</div><div class="line"> sha256sum, sha3sum, sha512sum, showkey, shuf, slattach, <span class="keyword">sleep</span>, smemcap,</div><div class="line"> softlimit, <span class="keyword">sort</span>, <span class="keyword">split</span>, start-<span class="keyword">stop</span>-daemon, stat, strings, stty, su,</div><div class="line"> sulogin, sum, <span class="keyword">sv</span>, svc, svlogd, swapoff, swapon, switch_root, <span class="keyword">sync</span>,</div><div class="line"> sysctl, syslogd, tac, tail, tar, tcpsvd, tee, telnet, telnetd, test,</div><div class="line"> tftp, tftpd, time, timeout, top, touch, <span class="keyword">tr</span>, traceroute, traceroute6,</div><div class="line"> true, truncate, tty, ttysize, tunctl, ubiattach, ubidetach, ubimkvol,</div><div class="line"> ubirename, ubirmvol, ubirsvol, ubiupdatevol, udhcpc, udhcpd, udpsvd,</div><div class="line"> uevent, umount, uname, unexpand, uniq, unix2dos, unlink, unlzma,</div><div class="line"> unlzop, unxz, unzip, uptime, usleep, uudecode, uuencode, vconfig, <span class="keyword">vi</span>,</div><div class="line"> vlock, volname, watch, watchdog, wc, wget, which, whoami, whois, xargs,</div><div class="line"> xz, xzcat, yes, zcat, zcip</div></pre></td></tr></table></figure>
<h4 id="FAQ"><a href="#FAQ" class="headerlink" title="FAQ:"></a>FAQ:</h4><figure class="highlight vim"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">Q:adb push命令出现错误：ad<span class="variable">b:</span> error: failed <span class="keyword">to</span> <span class="keyword">copy</span> <span class="string">'***'</span> <span class="keyword">to</span> <span class="string">'/data/***'</span>: Read-<span class="keyword">only</span> <span class="keyword">file</span> <span class="built_in">system</span></div><div class="line"></div><div class="line">A:请在模拟器内，使用RE，手动挂载为可读写，再次重试</div></pre></td></tr></table></figure>
<h3 id="inotify"><a href="#inotify" class="headerlink" title="inotify"></a>inotify</h3><p>inotify是用来监视文件系统事件的机制，在linux 2.6.13内核中引入。该机制可以用来监视文件和目录，当文件或目录发生变化时，内核会将文件或目录的变化发送给inotify文件描述符，在应用层只需调用read()就可以读取这些事件，非常的方便。更好的是，inotify文件描述符还可以使用select、poll、epoll这些接口来监听，当有事件发生是，inotify文件描述符会可读。</p>
<p>Genymotion模拟器也支持该特性，内核版本：root@vbox86p:/ # uname -r ——–> 3.10.0-genymotion-g08e528d</p>
<ul>
<li><a href="http://blog.csdn.net/justlinux2010/article/details/8680585" target="_blank" rel="external">Linux下监测目录或文件的变化</a></li>
<li><a href="系统监控工具----Inotify-Tools">系统监控工具—-Inotify-Tools</a></li>
</ul>
]]></content>
<summary type="html">
黑盒测试验证阶段，观察Chrome浏览器新加书签和新加文件夹时，重点对比对象是Bookmarks文件。这一阶段的观察结果，有很多疑问得不到结论。在观察Bookmarks文件的同时，我们使用Linux文件系统监视机制（内核级支持：inotify），来感知其他数据文件的联动变化，得到又一重点对象：SyncData.sqlite3，它的路径位于：“/data/data/com.android.chrome/app_chrome/Default/Sync Data/SyncData.sqlite3”，经分析，这是sqlite3数据库，它的变化与Bookmarks文件的变化紧密相连。
</summary>
<category term="Record" scheme="https://amao12580.github.io/categories/Record/"/>
<category term="Geek" scheme="https://amao12580.github.io/tags/Geek/"/>
<category term="Android" scheme="https://amao12580.github.io/tags/Android/"/>
<category term="Chrome" scheme="https://amao12580.github.io/tags/Chrome/"/>
<category term="Chromium" scheme="https://amao12580.github.io/tags/Chromium/"/>
<category term="SourceCode" scheme="https://amao12580.github.io/tags/SourceCode/"/>
<category term="SQLite" scheme="https://amao12580.github.io/tags/SQLite/"/>
<category term="JSON" scheme="https://amao12580.github.io/tags/JSON/"/>
<category term="Fastjson" scheme="https://amao12580.github.io/tags/Fastjson/"/>
<category term="Java" scheme="https://amao12580.github.io/tags/Java/"/>
<category term="C++" scheme="https://amao12580.github.io/tags/C/"/>
<category term="SourceInsight" scheme="https://amao12580.github.io/tags/SourceInsight/"/>
<category term="SublimText" scheme="https://amao12580.github.io/tags/SublimText/"/>
<category term="EditPlus" scheme="https://amao12580.github.io/tags/EditPlus/"/>
</entry>
<entry>
<title>Geek - Flyme书签导出到Via浏览器</title>
<link href="https://amao12580.github.io/post/2016/09/flyme-broswer-bookmarks-export/"/>
<id>https://amao12580.github.io/post/2016/09/flyme-broswer-bookmarks-export/</id>
<published>2016-09-29T01:43:52.234Z</published>
<updated>2017-08-11T08:55:22.278Z</updated>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>结尾部分的解析导出代码，已经整理完毕：<a href="https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java" target="_blank" rel="external">https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java</a></p>
<p>换了神族手机一年多，感觉各种受限制，没有官解Bootload，ROM没得刷，也就只能在线主题买买买，给老黄贡献订单，其实整来整去就那点界面变化。不Root也被各种广告、内置应用通知栏弹幕，索性Root折腾一把了！</p>
<p>Flyme内置浏览器（(V 4.5.0033)），不那么好用，实用的自定义UA功能不支持（突破各种云盘下载你懂的），我想android还是开放自由的吧，应该换个<a href="http://www.coolapk.com/apk/mark.via" target="_blank" rel="external">Via浏览器</a>(V 1.8.4，体积不超过300k，超轻量，可玩性也非常高)，再把Flyme浏览器里50+书签导出来，却发现没有导入导出书签的设置，WTF？？？</p>
<p><img src="/img/flyme-broswer-setting.jpg" alt=""><img src="/img/flyme-broswer-bookmarks.jpg" alt=""></p>
<p>好在咱也不是技术小白啊，这点小事，还是可以搞定的！</p>
<h1 id="实验"><a href="#实验" class="headerlink" title="实验"></a>实验</h1><p>既然魅族浏览器，在UI界面上没有提供书签导入导出，那我保存的书签记录在哪里呢？猜想是在/data/data/*目录，折腾前为防止数据丢失，先用<a href="http://www.coolapk.com/apk/com.keramidas.TitaniumBackup" target="_blank" rel="external">钛备份</a> backup一下魅族浏览器的程序和数据吧！</p>
<p>完成备份后，关闭魅族浏览器以及强制停止进程，尝试在内置存储目录（/storage/emulated/0/），使用<a href="http://www.coolapk.com/apk/com.speedsoftware.rootexplorer" target="_blank" rel="external">RE文件管理器</a>删除相关文件夹：/storage/emulated/0/.com.android.broswer/，重新打开，发现书签和设置并没有丢失！而且/storage/emulated/0/.com.android.broswer/文件夹又自动生成了！可见这个文件夹存放了一些临时文件，对用户的设置和书签数据不影响。</p>
<p>在/data/data/目录下（需Root才能读写），找了一番，定位到魅族浏览器的数据目录：/data/data/com.android.broswer/，啊这个文件夹还真挺丰富的！<br><img src="/img/flyme-broswer-datas.jpg" alt=""></p>
<p>究竟那个才是我们的书签存放目录呢？？？</p>
<h1 id="定位"><a href="#定位" class="headerlink" title="定位"></a>定位</h1><p>我想书签是一个用户的常用功能，会存在添加书签和删除的操作，应该会把数据放在一个适合快速存储和妥善保存的地方，比如说：database？还真有/data/data/com.android.broswer/databases/这个目录，啊哈，又近了一步呢！这个文件夹下都有些啥？竟是十多个数据库和临时WAL文件！<br><img src="/img/flyme-broswer-databases.jpg" alt=""><br>哪一个才是存放书签的呢？上排除法！根据一些命名过滤掉一些不可能的选项，余下的就是correct。正确答案是：/data/data/com.android.broswer/databases/broswer2.db这个文件，一起打开看看！（长按，右上角，选择打开方式，SQLite数据库浏览器）<br><img src="/img/flyme-broswer-databases-db2.jpg" alt=""></p>
<p>有很多大家熟悉的db：元数据、书签、历史访问、最常访问、图片、最近搜索、用户设置、账户同步。相信大家都知道选哪个了：bookmarks。终于看到需要的信息，所有的书签都在这里哦，想想办法弄出来吧！<br><img src="/img/flyme-broswer-databases-db2-bookmarks.jpg" alt=""></p>
<h1 id="导出"><a href="#导出" class="headerlink" title="导出"></a>导出</h1><p>为了编辑方便，我们将/data/data/com.android.broswer/databases/broswer2.db导出到PC操作，安装Navicat Premium（version:11.1.12），新建连接，选择SQLite。<br><img src="/img/flyme-broswer-databases-db2-bookmarks-navicat.jpg" alt=""></p>
<p>打开连接，选择：bookmarks，发现在、数据与RE管理器里看到是一样的！<br><img src="/img/flyme-broswer-databases-db2-bookmarks-navicat-datas.jpg" alt=""><br>我们直接使用navicat的数据导出向导，导出为json文件（还有很多格式可以选择，json方便下一步处理）bookmarks.json，可用的字段是title、url。<br><figure class="highlight perl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div><div class="line">226</div><div class="line">227</div><div class="line">228</div><div class="line">229</div><div class="line">230</div><div class="line">231</div></pre></td><td class="code"><pre><div class="line"><span class="comment">#表结构：</span></div><div class="line"></div><div class="line">CREATE TABLE bookmarks(_id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT,url TEXT,host TEXT, folder INTEGER NOT NULL DEFAULT <span class="number">0</span>,parent INTEGER,position INTEGER NOT NULL,insert_after INTEGER,deleted INTEGER NOT NULL DEFAULT <span class="number">0</span>,account_name TEXT,account_type TEXT,sourceid TEXT,version INTEGER NOT NULL DEFAULT <span class="number">1</span>,created INTEGER,modified INTEGER,dirty INTEGER NOT NULL DEFAULT <span class="number">0</span>,sync1 TEXT,sync2 TEXT,sync3 TEXT,sync4 TEXT,sync5 TEXT,mapping INTEGER NOT NULL DEFAULT <span class="number">0</span>);</div><div class="line"></div><div class="line"><span class="comment">#查询title、url两个字段</span></div><div class="line"></div><div class="line">SELECT title,url from bookmarks;</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">{</div><div class="line"><span class="string">"RECORDS"</span>:[</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"默认目录"</span>,</div><div class="line"><span class="string">"url"</span>:null</div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Google 镜像站搜集 - Techzero"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.itechzero.com\/google-mirror-sites-collect.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.baidu.com\/from=879a\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.1_18_\/baiduid=4C04AB048A505E6697DCB5C3C808A263\/w=0_10_oop+ooa+ood\/t=iphone\/l=3\/tc?ref=www_iphone&lid=2022227202196170648&order=5&vit=osres&tj=www_normal_5_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=9624&di=1c941110077479c6&bdenc=1&tch=124.237.42.912.2.930&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-7agTCcgsFOd7GdWWUm7hp0rK&eqid=1c1064ed1a08e8001000000556a6ba3c&wd="</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java NIO和IO的区别_java_脚本之家"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.jb51.net\/article\/50621.htm"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.importnew.com\/1993.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"java常用设计模式 - Leo Chin - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.cnblogs.com\/hnrainll\/archive\/2011\/12\/29\/2305582.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"正则表达式给力小伙伴 | 行思錄 | Travel Coder"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/liudanking.me\/smarttool\/regexp_smarttool\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"进阶正则表达式 - Barret Lee - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.cnblogs.com\/hustskyking\/p\/how-regular-expressions-work.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.myexception.cn\/operating-system\/1932122.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/my.oschina.net\/u\/554660\/blog\/284653"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/blog.csdn.net\/shagoo\/article\/details\/8191346"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"深圳市五险一金及税后工资计算器"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/salarycalculator.sinaapp.com\/city\/shenzhen"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"多图详解Spring框架的设计理念与设计模式(1) - 51CTO.COM"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/developer.51cto.com\/art\/201006\/205212.htm"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"java中的设计模式如何分类"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/zhidao.baidu.com\/question\/223541008.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=0&from=879a&uid=0&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java 设计模式分类"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=4732764"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"来自投资银行的20个Java面试题 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/20-java-interview-questions-from-investment-banks.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/bbym010.iteye.com\/blog\/2100868"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"HAProxy简介 - 张善友 - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.cnblogs.com\/shanyou\/archive\/2012\/10\/16\/2726768.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"什么是“极限编程”？"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/zhidao.baidu.com\/question\/5153690.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=0&from=879a&uid=0&pu=usm@1,sz@1320_1004,ta@iphone_2_5.1_18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"MongoDB Sharding机制分析 » 囧囧孙"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.jiongsun.com\/2013\/03\/45.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.baidu.com\/from=1001560s\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.1_3_537\/baiduid=4C04AB048A505E6697DCB5C3C808A263\/w=0_10_kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF\/t=iphone\/l=3\/tc?ref=www_iphone&lid=11304546645738609731&order=1&vit=osres&tj=www_normal_1_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D..._%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=10463&di=b0551844ad65db76&bdenc=1&tch=124.0.0.0.0.0&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=9ce1d147be1560001000000356d4d7f0&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"互联网系统可靠性基础：正确的异常处理"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.tuicool.com\/m\/articles\/UvayInY"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"服务器端 概述 - 极光文档"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/docs.jpush.io\/server\/server_overview\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"从ACID到CAP到BASE - xixicat - SegmentFault"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/segmentfault.com\/a\/1190000004468442?url_type=39&object_type=webpage&pos=1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Netty系列之Netty可靠性分析"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.infoq.com\/cn\/articles\/netty-reliability\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.tuicool.com\/m\/articles\/feYfInn"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"有趣的bloom过滤器 | OurMySQL"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/ourmysql.com\/archives\/510?f=wb"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=27109467"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/java-string-ansi-unicode-bmp-utf.html?url_type=39&object_type=webpage&pos=1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"看了极光推送技术原理的几点思考"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=8227469"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"腾讯微信技术架构"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/m.blog.csdn.net\/article\/details?id=8581852"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"鸟哥的 Linux 私房菜 -- 学习 bash shell"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/vbird.dic.ksu.edu.tw\/linux_basic\/0320bash.php"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/www.zhihu.com\/question\/20962240"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"动态规划：从新手到专家"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.hawstein.com\/posts\/dp-novice-to-advanced.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"商品搜索引擎—推荐系统设计 - ImportNew"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.importnew.com\/19834.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/coolshell.cn\/articles\/6790.html"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"32个算法"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/media.weibo.cn\/article?id=2309403979794248269609&jumpfrom=weibocom"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java对象之死-云栖社区"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/yq.aliyun.com\/articles\/53681?spm=5176.100239.bloglist.55.lOez9A"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"amao12580 · GitHub"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/github.com\/amao12580\/"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"微服务与架构师"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA3MDMwOTcwMg==&mid=2650004550&idx=1&sn=b2299d1f2456541fc6b600a480c27dd8&scene=1&srcid=0714I4IHCwFfqnIGigCv3pJb#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"图灵社区 : 阅读 : 什么时候你不应该使用微服务"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.ituring.com.cn\/article\/217430"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"微服务的两种模式：应用中心和任务中心 - DockOne.io"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/dockone.io\/m\/article\/1335"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/blogread.cn\/it\/wap\/article\/7770?f=wb"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"一个资深架构师是这样理解数据结构的"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"高性能服务器架构思路 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"Java编程常见问题汇总 – 码农网"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/www.codeceo.com\/article\/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/blogread.cn\/it\/wap\/article\/7816?f=wb"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"微服务实战：从架构到部署 - DockOne.io"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/dockone.io\/article\/1519"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"优酷蓝鲸近千节点的Redis集群运维经验总结"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"spring ioc DI 理解 - WhyWin - 博客园"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"https:\/\/zm6.sm-tc.cn\/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/dockone.io\/article\/1582"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"技改之路：从单块应用到微服务，我的血泪总结"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"华为内部如何实施微服务架构？基本就靠这5大原则"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/mp.weixin.qq.com\/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"</span></div><div class="line">},</div><div class="line">{</div><div class="line"><span class="string">"title"</span>:<span class="string">"悲观的看：互联网技术岗位红利期逐渐到头"</span>,</div><div class="line"><span class="string">"url"</span>:<span class="string">"http:\/\/media.weibo.cn\/article?id=2309404020851660960391&jumpfrom=weibocom"</span></div><div class="line">}</div><div class="line">]</div><div class="line">}</div></pre></td></tr></table></figure></p>
<h1 id="导入"><a href="#导入" class="headerlink" title="导入"></a>导入</h1><p>目前各家浏览器的导入导出格式并没有统一标准，我们来看看Via 浏览器的导出文件：/storage/emulated/0/Download/Via_书签_20160929.txt，与导出的文件一致，也是json格式。</p>
<figure class="highlight json"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">{<span class="attr">"title"</span>:<span class="string">"Cat's Blog - 一饮一啄，莫非前定. - Steven Cheng"</span>,<span class="attr">"url"</span>:<span class="string">"https:\/\/amao12580.github.io\/"</span>,<span class="attr">"folder"</span>:<span class="string">""</span>,<span class="attr">"order"</span>:<span class="number">0</span>}</div><div class="line">{<span class="attr">"title"</span>:<span class="string">"十周年 - Google 搜索"</span>,<span class="attr">"url"</span>:<span class="string">"https:\/\/www.google.com.hk\/search?q=%E5%8D%81%E5%91%A8%E5%B9%B4"</span>,<span class="attr">"folder"</span>:<span class="string">""</span>,<span class="attr">"order"</span>:<span class="number">0</span>}</div></pre></td></tr></table></figure>
<p>既然数据格式一致，剩下的事就简单了，只需要将魅族浏览器导出的broswer2.db文件整理成Via可接受的结构就行，写一段小程序吧，decode + code！</p>
<figure class="highlight perl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div><div class="line">138</div><div class="line">139</div><div class="line">140</div><div class="line">141</div><div class="line">142</div><div class="line">143</div><div class="line">144</div><div class="line">145</div><div class="line">146</div><div class="line">147</div><div class="line">148</div><div class="line">149</div><div class="line">150</div><div class="line">151</div><div class="line">152</div><div class="line">153</div><div class="line">154</div><div class="line">155</div><div class="line">156</div><div class="line">157</div><div class="line">158</div><div class="line">159</div><div class="line">160</div><div class="line">161</div><div class="line">162</div><div class="line">163</div><div class="line">164</div><div class="line">165</div><div class="line">166</div><div class="line">167</div><div class="line">168</div><div class="line">169</div><div class="line">170</div><div class="line">171</div><div class="line">172</div><div class="line">173</div><div class="line">174</div><div class="line">175</div><div class="line">176</div><div class="line">177</div><div class="line">178</div><div class="line">179</div><div class="line">180</div><div class="line">181</div><div class="line">182</div><div class="line">183</div><div class="line">184</div><div class="line">185</div><div class="line">186</div><div class="line">187</div><div class="line">188</div><div class="line">189</div><div class="line">190</div><div class="line">191</div><div class="line">192</div><div class="line">193</div><div class="line">194</div><div class="line">195</div><div class="line">196</div><div class="line">197</div><div class="line">198</div><div class="line">199</div><div class="line">200</div><div class="line">201</div><div class="line">202</div><div class="line">203</div><div class="line">204</div><div class="line">205</div><div class="line">206</div><div class="line">207</div><div class="line">208</div><div class="line">209</div><div class="line">210</div><div class="line">211</div><div class="line">212</div><div class="line">213</div><div class="line">214</div><div class="line">215</div><div class="line">216</div><div class="line">217</div><div class="line">218</div><div class="line">219</div><div class="line">220</div><div class="line">221</div><div class="line">222</div><div class="line">223</div><div class="line">224</div><div class="line">225</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> org.jooq.example;</div><div class="line"></div><div class="line">import java.util.List;</div><div class="line"></div><div class="line">/**</div><div class="line"> * Created with IntelliJ IDEA.</div><div class="line"> * User:ChengLiang</div><div class="line"> * Date:<span class="number">2016</span>/<span class="number">9</span>/<span class="number">29</span></div><div class="line"> * Time:<span class="number">15</span>:<span class="number">03</span></div><div class="line"> *<span class="regexp">/</span></div><div class="line"><span class="regexp">@lombok.Data</span></div><div class="line"><span class="regexp">public class MeizuBookmark {</span></div><div class="line"><span class="regexp"> List<item> RECORDS;</span></div><div class="line"><span class="regexp"> @lombok.Data</span></div><div class="line"><span class="regexp"> class item{</span></div><div class="line"><span class="regexp"> String title;</span></div><div class="line"><span class="regexp"> String url;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp">}</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">package org.jooq.example;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">import com.alibaba.fastjson.annotation.JSONField;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">/</span>**</div><div class="line"> * Created with IntelliJ IDEA.</div><div class="line"> * User:ChengLiang</div><div class="line"> * Date:<span class="number">2016</span>/<span class="number">9</span>/<span class="number">29</span></div><div class="line"> * Time:<span class="number">15</span>:<span class="number">03</span></div><div class="line"> *<span class="regexp">/</span></div><div class="line"><span class="regexp">@lombok.Data</span></div><div class="line"><span class="regexp">public class ViaBookmark {</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 1)</span></div><div class="line"><span class="regexp"> String title;</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 2)</span></div><div class="line"><span class="regexp"> String url;</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 3)</span></div><div class="line"><span class="regexp"> String folder = "";</span></div><div class="line"><span class="regexp"> @JSONField(ordinal = 4)</span></div><div class="line"><span class="regexp"> int order = 0;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"> public ViaBookmark(String title, String url,int order) {</span></div><div class="line"><span class="regexp"> this.title=title;</span></div><div class="line"><span class="regexp"> this.url=url;</span></div><div class="line"><span class="regexp"> this.order=order;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp">}</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">package org.jooq.example;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">import com.alibaba.fastjson.JSON;</span></div><div class="line"><span class="regexp">import com.alibaba.fastjson.JSONReader;</span></div><div class="line"><span class="regexp">import org.junit.Test;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">import java.io.File;</span></div><div class="line"><span class="regexp">import java.io.FileNotFoundException;</span></div><div class="line"><span class="regexp">import java.io.FileReader;</span></div><div class="line"><span class="regexp">import java.util.ArrayList;</span></div><div class="line"><span class="regexp">import java.util.List;</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">/</span>**</div><div class="line"> * Created with IntelliJ IDEA.</div><div class="line"> * User:ChengLiang</div><div class="line"> * Date:<span class="number">2016</span>/<span class="number">9</span>/<span class="number">29</span></div><div class="line"> * Time:<span class="number">15</span>:<span class="number">00</span></div><div class="line"> * <p></div><div class="line"> * fastJson 解析魅族浏览器书签json文件，输出via浏览可导入的结构，整理为json文件</div><div class="line"> *<span class="regexp">/</span></div><div class="line"><span class="regexp">public class BroswerTest {</span></div><div class="line"><span class="regexp"> @Test</span></div><div class="line"><span class="regexp"> public void readJSONFromFile() throws FileNotFoundException {</span></div><div class="line"><span class="regexp"> String fullPath = "C:\\Users\\Administrator\\Desktop\\bookmarks3.json";</span></div><div class="line"><span class="regexp"> JSONReader jsonReader = new JSONReader(new FileReader(new File(fullPath)));</span></div><div class="line"><span class="regexp"> MeizuBookmark meizuBookmark = jsonReader.readObject(MeizuBookmark.class);</span></div><div class="line"><span class="regexp"> jsonReader.close();</span></div><div class="line"><span class="regexp"> System.out.println("meizuBookmark:" + JSON.toJSONString(meizuBookmark));</span></div><div class="line"><span class="regexp"> if (meizuBookmark == null || meizuBookmark.getRECORDS() == null || meizuBookmark.getRECORDS().isEmpty()) {</span></div><div class="line"><span class="regexp"> System.out.println("解码失败，或源文件没有书签");</span></div><div class="line"><span class="regexp"> return;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> int index = 0;</span></div><div class="line"><span class="regexp"> int seq = 0;</span></div><div class="line"><span class="regexp"> System.out.println("完成解码，开始解析书签.");</span></div><div class="line"><span class="regexp"> List<ViaBookmark> results=new ArrayList<>();</span></div><div class="line"><span class="regexp"> for (MeizuBookmark.item item : meizuBookmark.getRECORDS()) {</span></div><div class="line"><span class="regexp"> index++;</span></div><div class="line"><span class="regexp"> System.out.println("第" + index + "条书签：" + JSON.toJSONString(item));</span></div><div class="line"><span class="regexp"> if(item.getTitle()==null||item.getTitle().isEmpty()||item.getUrl()==null||item.getUrl().isEmpty()){</span></div><div class="line"><span class="regexp"> System.out.println("数据不完整，跳过处理.");</span></div><div class="line"><span class="regexp"> continue;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> results.add(new ViaBookmark(item.getTitle(),item.getUrl(),seq));</span></div><div class="line"><span class="regexp"> seq++;</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> System.out.println("viaBookmark:");</span></div><div class="line"><span class="regexp"> for(ViaBookmark viaBookmark:results){</span></div><div class="line"><span class="regexp"> System.out.println(JSON.toJSONString(viaBookmark));</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp"> System.out.println("完成解析书签.");</span></div><div class="line"><span class="regexp"> }</span></div><div class="line"><span class="regexp">}</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">Console：</span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp"></span></div><div class="line"><span class="regexp">"D:\Program Files\Java\jdk1.8.0_74\bin\java" -ea -Didea.launcher.port=7537 -Didea.launcher.bin.path=D:\IDEA14\bin -Dfile.encoding=UTF-8 -classpath "D:\IDEA14\lib\idea_rt.jar;D:\IDEA14\plugins\junit\lib\junit-rt.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\charsets.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\deploy.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\access-bridge-64.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\cldrdata.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\dnsns.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\jaccess.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\jfxrt.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\localedata.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\nashorn.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunec.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunjce_provider.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunmscapi.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\sunpkcs11.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\ext\zipfs.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\javaws.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jce.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jfr.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jfxswt.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\jsse.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\management-agent.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\plugin.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\resources.jar;D:\Program Files\Java\jdk1.8.0_74\jre\lib\rt.jar;D:\GitHub\JOOQ-With-Spring\target\test-classes;D:\GitHub\JOOQ-With-Spring\target\classes;D:\maven\repository\org\apache\logging\log4j\log4j-api\2.6.2\log4j-api-2.6.2.jar;D:\maven\repository\org\apache\logging\log4j\log4j-core\2.6.2\log4j-core-2.6.2.jar;D:\maven\repository\org\slf4j\slf4j-api\1.7.21\slf4j-api-1.7.21.jar;D:\maven\repository\org\apache\logging\log4j\log4j-slf4j-impl\2.6.2\log4j-slf4j-impl-2.6.2.jar;D:\maven\repository\org\springframework\spring-context\4.2.5.RELEASE\spring-context-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-aop\4.2.5.RELEASE\spring-aop-4.2.5.RELEASE.jar;D:\maven\repository\aopalliance\aopalliance\1.0\aopalliance-1.0.jar;D:\maven\repository\org\springframework\spring-beans\4.2.5.RELEASE\spring-beans-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-core\4.2.5.RELEASE\spring-core-4.2.5.RELEASE.jar;D:\maven\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;D:\maven\repository\org\springframework\spring-expression\4.2.5.RELEASE\spring-expression-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-jdbc\4.2.5.RELEASE\spring-jdbc-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-tx\4.2.5.RELEASE\spring-tx-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-webmvc\4.2.5.RELEASE\spring-webmvc-4.2.5.RELEASE.jar;D:\maven\repository\org\springframework\spring-web\4.2.5.RELEASE\spring-web-4.2.5.RELEASE.jar;D:\maven\repository\org\aspectj\aspectjrt\1.5.4\aspectjrt-1.5.4.jar;D:\maven\repository\org\aspectj\aspectjweaver\1.8.9\aspectjweaver-1.8.9.jar;D:\maven\repository\javax\servlet\javax.servlet-api\4.0.0-b01\javax.servlet-api-4.0.0-b01.jar;D:\maven\repository\javax\servlet\jsp\jsp-api\2.2.1-b03\jsp-api-2.2.1-b03.jar;D:\maven\repository\junit\junit\4.12\junit-4.12.jar;D:\maven\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;D:\maven\repository\org\springframework\spring-test\4.2.5.RELEASE\spring-test-4.2.5.RELEASE.jar;D:\maven\repository\org\jooq\jooq-codegen\3.8.4\jooq-codegen-3.8.4.jar;D:\maven\repository\org\jooq\jooq\3.8.4\jooq-3.8.4.jar;D:\maven\repository\org\jooq\jooq-meta\3.8.4\jooq-meta-3.8.4.jar;D:\maven\repository\mysql\mysql-connector-java\5.1.38\mysql-connector-java-5.1.38.jar;D:\maven\repository\com\zaxxer\HikariCP\2.4.7\HikariCP-2.4.7.jar;D:\maven\repository\org\projectlombok\lombok\1.16.8\lombok-1.16.8.jar;D:\maven\repository\joda-time\joda-time\2.9.4\joda-time-2.9.4.jar;D:\maven\repository\com\alibaba\fastjson\1.2.15\fastjson-1.2.15.jar" com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 org.jooq.example.BroswerTest,readJSONFromFile</span></div><div class="line"><span class="regexp">meizuBookmark:{"rECORDS":[{"title":"默认目录"},{"title":"Google 镜像站搜集 - Techzero","url":"http:/</span><span class="regexp">/www.itechzero.com/google</span>-mirror-sites-collect.html<span class="string">"},{"</span>title<span class="string">":"</span>OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.baidu.com/from=<span class="number">879</span>a/bd_page_type=<span class="number">1</span>/ssid=<span class="number">0</span>/uid=<span class="number">0</span>/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.<span class="number">1_18_</span>/baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263/w=<span class="number">0_10_</span>oop+ooa+ood/t=iphone/l=<span class="number">3</span>/tc?<span class="keyword">ref</span>=www_iphone&lid=<span class="number">2022227202196170648</span>&order=<span class="number">5</span>&vit=osres&tj=www_normal_5_0_10_title&<span class="keyword">m</span>=<span class="number">8</span>&srd=<span class="number">1</span>&cltj=cloud_title&dict=<span class="number">30</span>&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=<span class="number">9624</span>&di=<span class="number">1</span>c941110077479c6&bdenc=<span class="number">1</span>&tch=<span class="number">124.237</span>.<span class="number">42.912</span>.<span class="number">2.930</span>&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-<span class="number">7</span>agTCcgsFOd7GdWWUm7hp0rK&eqid=<span class="number">1</span>c1064ed1a08e8001000000556a6ba3c&wd=<span class="string">"},{"</span>title<span class="string">":"</span>Java NIO和IO的区别_java<span class="number">_</span>脚本之家<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.jb51.net/article/<span class="number">50621</span>.htm<span class="string">"},{"</span>title<span class="string">":"</span>成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.importnew.com/<span class="number">1993</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>java常用设计模式 - Leo Chin - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.cnblogs.com/hnrainll/archive/<span class="number">2011</span>/<span class="number">12</span>/<span class="number">29</span>/<span class="number">2305582</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>正则表达式给力小伙伴 | 行思錄 | Travel Coder<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//liudanking</span>.me/smarttool/regexp_smarttool/<span class="string">"},{"</span>title<span class="string">":"</span>进阶正则表达式 - Barret Lee - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.cnblogs.com/hustskyking/p/how-regular-expressions-work.html<span class="string">"},{"</span>title<span class="string">":"</span>[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.myexception.cn/operating-<span class="keyword">system</span>/<span class="number">1932122</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//my</span>.oschina.net/u/<span class="number">554660</span>/blog/<span class="number">284653</span><span class="string">"},{"</span>title<span class="string">":"</span>[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//blog</span>.csdn.net/shagoo/article/details/<span class="number">8191346</span><span class="string">"},{"</span>title<span class="string">":"</span>深圳市五险一金及税后工资计算器<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//salarycalculator</span>.sinaapp.com/city/shenzhen<span class="string">"},{"</span>title<span class="string">":"</span>多图详解Spring框架的设计理念与设计模式(<span class="number">1</span>) - <span class="number">51</span>CTO.COM<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//developer</span>.<span class="number">51</span>cto.com/art/<span class="number">201006</span>/<span class="number">205212</span>.htm<span class="string">"},{"</span>title<span class="string">":"</span>java中的设计模式如何分类<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//zhidao</span>.baidu.com/question/<span class="number">223541008</span>.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=<span class="number">0</span>&from=<span class="number">879</span>a&uid=<span class="number">0</span>&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=<span class="number">1</span>&baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1<span class="string">"},{"</span>title<span class="string">":"</span>Java 设计模式分类<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">4732764</span><span class="string">"},{"</span>title<span class="string">":"</span>来自投资银行的<span class="number">20</span>个Java面试题 – 码农网<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.codeceo.com/article/<span class="number">20</span>-java-interview-questions-from-investment-banks.html<span class="string">"},{"</span>title<span class="string">":"</span>JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//bbym</span>01<span class="number">0</span>.iteye.com/blog/<span class="number">2100868</span><span class="string">"},{"</span>title<span class="string">":"</span>HAProxy简介 - 张善友 - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.cnblogs.com/shanyou/archive/<span class="number">2012</span>/<span class="number">10</span>/<span class="number">16</span>/<span class="number">2726768</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>什么是“极限编程”？<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//zhidao</span>.baidu.com/question/<span class="number">5153690</span>.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=<span class="number">0</span>&from=<span class="number">879</span>a&uid=<span class="number">0</span>&pu=usm@1,sz@1320_1004,ta@iphone_2_5.<span class="number">1_18_</span>&bd_page_type=<span class="number">1</span>&baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1<span class="string">"},{"</span>title<span class="string">":"</span>MongoDB Sharding机制分析 » 囧囧孙<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.jiongsun.com/<span class="number">2013</span>/<span class="number">03</span>/<span class="number">45</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.baidu.com/from=<span class="number">1001560</span><span class="keyword">s</span>/bd_page_type=<span class="number">1</span>/ssid=<span class="number">0</span>/uid=<span class="number">0</span>/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.<span class="number">1_3_537</span>/baiduid=<span class="number">4</span>C04AB048A505E6697DCB5C3C808A263/w=<span class="number">0_10_</span>kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF/t=iphone/l=<span class="number">3</span>/tc?<span class="keyword">ref</span>=www_iphone&lid=<span class="number">11304546645738609731</span>&order=<span class="number">1</span>&vit=osres&tj=www_normal_1_0_10_title&<span class="keyword">m</span>=<span class="number">8</span>&srd=<span class="number">1</span>&cltj=cloud_title&dict=<span class="number">30</span>&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D...<span class="number">_</span>%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=<span class="number">10463</span>&di=b0551844ad65db76&bdenc=<span class="number">1</span>&tch=<span class="number">124.0</span>.<span class="number">0</span>.<span class="number">0</span>.<span class="number">0</span>.<span class="number">0</span>&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=<span class="number">9</span>ce1d147be1560001000000356d4d7f<span class="number">0</span>&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D<span class="string">"},{"</span>title<span class="string">":"</span>互联网系统可靠性基础：正确的异常处理<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.tuicool.com/<span class="regexp">m/articles/</span>UvayInY<span class="string">"},{"</span>title<span class="string">":"</span>服务器端 概述 - 极光文档<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//docs</span>.jpush.io/server/server_overview/<span class="string">"},{"</span>title<span class="string">":"</span>从ACID到CAP到BASE - xixicat - SegmentFault<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//segmentfault</span>.com/a/<span class="number">1190000004468442</span>?url_type=<span class="number">39</span>&object_type=webpage&<span class="keyword">pos</span>=<span class="number">1</span><span class="string">"},{"</span>title<span class="string">":"</span>Netty系列之Netty可靠性分析<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.infoq.com/cn/articles/netty-reliability/<span class="string">"},{"</span>title<span class="string">":"</span>【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.tuicool.com/<span class="regexp">m/articles/fe</span>YfInn<span class="string">"},{"</span>title<span class="string">":"</span>有趣的bloom过滤器 | OurMySQL<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//ourmysql</span>.com/archives/<span class="number">510</span>?f=wb<span class="string">"},{"</span>title<span class="string">":"</span>Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">27109467</span><span class="string">"},{"</span>title<span class="string">":"</span>从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.codeceo.com/article/java-string-ansi-unicode-bmp-utf.html?url_type=<span class="number">39</span>&object_type=webpage&<span class="keyword">pos</span>=<span class="number">1</span><span class="string">"},{"</span>title<span class="string">":"</span>看了极光推送技术原理的几点思考<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">8227469</span><span class="string">"},{"</span>title<span class="string">":"</span>腾讯微信技术架构<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//m</span>.blog.csdn.net/article/details?id=<span class="number">8581852</span><span class="string">"},{"</span>title<span class="string">":"</span>鸟哥的 Linux 私房菜 -- 学习 bash shell<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//vbird</span>.dic.ksu.edu.tw/linux_basic/<span class="number">0320</span>bash.php<span class="string">"},{"</span>title<span class="string">":"</span>如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//www</span>.zhihu.com/question/<span class="number">20962240</span><span class="string">"},{"</span>title<span class="string">":"</span>动态规划：从新手到专家<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.hawstein.com/posts/dp-novice-to-advanced.html<span class="string">"},{"</span>title<span class="string">":"</span>商品搜索引擎—推荐系统设计 - ImportNew<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//www</span>.importnew.com/<span class="number">19834</span>.html<span class="string">"},{"</span>title<span class="string">":"</span>多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//coolshell</span>.cn/articles/<span class="number">6790</span>.html<span class="string">"},{"</span>title<span class="string">":"</span><span class="number">32</span>个算法<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//media</span>.weibo.cn/article?id=<span class="number">2309403979794248269609</span>&jumpfrom=weibocom<span class="string">"},{"</span>title<span class="string">":"</span>Java对象之死-云栖社区<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//yq</span>.aliyun.com/articles/<span class="number">53681</span>?spm=<span class="number">5176.100239</span>.bloglist.<span class="number">55</span>.lOez9A<span class="string">"},{"</span>title<span class="string">":"</span>amao1258<span class="number">0</span> · GitHub<span class="string">","</span>url<span class="string">":"</span>https:<span class="regexp">//github</span>.com/amao1258<span class="number">0</span>/<span class="string">"},{"</span>title<span class="string">":"</span>微服务与架构师<span class="string">","</span>url<span class="string">":"</span>http:<span class="regexp">//mp</span>.weixin.qq.com/<span class="keyword">s</span>?__biz=MzA3MDMwOTcwMg==&mid=<span class="number">2650004550</span>&idx=<span class="number">1</span>&sn=b2299d1f2456541fc6b600a480c27dd8&scene=<span class="number">1</span>&srcid=<span class="number">0714</span>I4IHCwFfqnIGigCv3pJb<span class="comment">#rd"},{"title":"图灵社区 : 阅读 : 什么时候你不应该使用微服务","url":"http://www.ituring.com.cn/article/217430"},{"title":"微服务的两种模式：应用中心和任务中心 - DockOne.io","url":"http://dockone.io/m/article/1335"},{"title":"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！","url":"http://blogread.cn/it/wap/article/7770?f=wb"},{"title":"一个资深架构师是这样理解数据结构的","url":"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"},{"title":"高性能服务器架构思路 – 码农网","url":"http://www.codeceo.com/article/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"},{"title":"Java编程常见问题汇总 – 码农网","url":"http://www.codeceo.com/article/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"},{"title":"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！","url":"http://blogread.cn/it/wap/article/7816?f=wb"},{"title":"微服务实战：从架构到部署 - DockOne.io","url":"http://dockone.io/article/1519"},{"title":"优酷蓝鲸近千节点的Redis集群运维经验总结","url":"http://mp.weixin.qq.com/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"},{"title":"spring ioc DI 理解 - WhyWin - 博客园","url":"https://zm6.sm-tc.cn/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"},{"title":"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io","url":"http://dockone.io/article/1582"},{"title":"技改之路：从单块应用到微服务，我的血泪总结","url":"http://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"},{"title":"华为内部如何实施微服务架构？基本就靠这5大原则","url":"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"},{"title":"悲观的看：互联网技术岗位红利期逐渐到头","url":"http://media.weibo.cn/article?id=2309404020851660960391&jumpfrom=weibocom"}]}</span></div><div class="line">完成解码，开始解析书签.</div><div class="line">第<span class="number">1</span>条书签：{<span class="string">"title"</span>:<span class="string">"默认目录"</span>}</div><div class="line">数据不完整，跳过处理.</div><div class="line">第<span class="number">2</span>条书签：{<span class="string">"title"</span>:<span class="string">"Google 镜像站搜集 - Techzero"</span>,<span class="string">"url"</span>:<span class="string">"http://www.itechzero.com/google-mirror-sites-collect.html"</span>}</div><div class="line">第<span class="number">3</span>条书签：{<span class="string">"title"</span>:<span class="string">"OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=879a/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.1_18_/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_oop+ooa+ood/t=iphone/l=3/tc?ref=www_iphone&lid=2022227202196170648&order=5&vit=osres&tj=www_normal_5_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=9624&di=1c941110077479c6&bdenc=1&tch=124.237.42.912.2.930&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-7agTCcgsFOd7GdWWUm7hp0rK&eqid=1c1064ed1a08e8001000000556a6ba3c&wd="</span>}</div><div class="line">第<span class="number">4</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java NIO和IO的区别_java_脚本之家"</span>,<span class="string">"url"</span>:<span class="string">"http://m.jb51.net/article/50621.htm"</span>}</div><div class="line">第<span class="number">5</span>条书签：{<span class="string">"title"</span>:<span class="string">"成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/1993.html"</span>}</div><div class="line">第<span class="number">6</span>条书签：{<span class="string">"title"</span>:<span class="string">"java常用设计模式 - Leo Chin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hnrainll/archive/2011/12/29/2305582.html"</span>}</div><div class="line">第<span class="number">7</span>条书签：{<span class="string">"title"</span>:<span class="string">"正则表达式给力小伙伴 | 行思錄 | Travel Coder"</span>,<span class="string">"url"</span>:<span class="string">"http://liudanking.me/smarttool/regexp_smarttool/"</span>}</div><div class="line">第<span class="number">8</span>条书签：{<span class="string">"title"</span>:<span class="string">"进阶正则表达式 - Barret Lee - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hustskyking/p/how-regular-expressions-work.html"</span>}</div><div class="line">第<span class="number">9</span>条书签：{<span class="string">"title"</span>:<span class="string">"[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统"</span>,<span class="string">"url"</span>:<span class="string">"http://m.myexception.cn/operating-system/1932122.html"</span>}</div><div class="line">第<span class="number">10</span>条书签：{<span class="string">"title"</span>:<span class="string">"从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://my.oschina.net/u/554660/blog/284653"</span>}</div><div class="line">第<span class="number">11</span>条书签：{<span class="string">"title"</span>:<span class="string">"[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET"</span>,<span class="string">"url"</span>:<span class="string">"http://blog.csdn.net/shagoo/article/details/8191346"</span>}</div><div class="line">第<span class="number">12</span>条书签：{<span class="string">"title"</span>:<span class="string">"深圳市五险一金及税后工资计算器"</span>,<span class="string">"url"</span>:<span class="string">"http://salarycalculator.sinaapp.com/city/shenzhen"</span>}</div><div class="line">第<span class="number">13</span>条书签：{<span class="string">"title"</span>:<span class="string">"多图详解Spring框架的设计理念与设计模式(1) - 51CTO.COM"</span>,<span class="string">"url"</span>:<span class="string">"http://developer.51cto.com/art/201006/205212.htm"</span>}</div><div class="line">第<span class="number">14</span>条书签：{<span class="string">"title"</span>:<span class="string">"java中的设计模式如何分类"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/223541008.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=0&from=879a&uid=0&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1"</span>}</div><div class="line">第<span class="number">15</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java 设计模式分类"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=4732764"</span>}</div><div class="line">第<span class="number">16</span>条书签：{<span class="string">"title"</span>:<span class="string">"来自投资银行的20个Java面试题 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/20-java-interview-questions-from-investment-banks.html"</span>}</div><div class="line">第<span class="number">17</span>条书签：{<span class="string">"title"</span>:<span class="string">"JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站"</span>,<span class="string">"url"</span>:<span class="string">"http://bbym010.iteye.com/blog/2100868"</span>}</div><div class="line">第<span class="number">18</span>条书签：{<span class="string">"title"</span>:<span class="string">"HAProxy简介 - 张善友 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/shanyou/archive/2012/10/16/2726768.html"</span>}</div><div class="line">第<span class="number">19</span>条书签：{<span class="string">"title"</span>:<span class="string">"什么是“极限编程”？"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/5153690.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=0&from=879a&uid=0&pu=usm@1,sz@1320_1004,ta@iphone_2_5.1_18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1"</span>}</div><div class="line">第<span class="number">20</span>条书签：{<span class="string">"title"</span>:<span class="string">"MongoDB Sharding机制分析 » 囧囧孙"</span>,<span class="string">"url"</span>:<span class="string">"http://www.jiongsun.com/2013/03/45.html"</span>}</div><div class="line">第<span class="number">21</span>条书签：{<span class="string">"title"</span>:<span class="string">"kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=1001560s/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.1_3_537/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF/t=iphone/l=3/tc?ref=www_iphone&lid=11304546645738609731&order=1&vit=osres&tj=www_normal_1_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D..._%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=10463&di=b0551844ad65db76&bdenc=1&tch=124.0.0.0.0.0&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=9ce1d147be1560001000000356d4d7f0&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D"</span>}</div><div class="line">第<span class="number">22</span>条书签：{<span class="string">"title"</span>:<span class="string">"互联网系统可靠性基础：正确的异常处理"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/UvayInY"</span>}</div><div class="line">第<span class="number">23</span>条书签：{<span class="string">"title"</span>:<span class="string">"服务器端 概述 - 极光文档"</span>,<span class="string">"url"</span>:<span class="string">"http://docs.jpush.io/server/server_overview/"</span>}</div><div class="line">第<span class="number">24</span>条书签：{<span class="string">"title"</span>:<span class="string">"从ACID到CAP到BASE - xixicat - SegmentFault"</span>,<span class="string">"url"</span>:<span class="string">"https://segmentfault.com/a/1190000004468442?url_type=39&object_type=webpage&pos=1"</span>}</div><div class="line">第<span class="number">25</span>条书签：{<span class="string">"title"</span>:<span class="string">"Netty系列之Netty可靠性分析"</span>,<span class="string">"url"</span>:<span class="string">"http://www.infoq.com/cn/articles/netty-reliability/"</span>}</div><div class="line">第<span class="number">26</span>条书签：{<span class="string">"title"</span>:<span class="string">"【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/feYfInn"</span>}</div><div class="line">第<span class="number">27</span>条书签：{<span class="string">"title"</span>:<span class="string">"有趣的bloom过滤器 | OurMySQL"</span>,<span class="string">"url"</span>:<span class="string">"http://ourmysql.com/archives/510?f=wb"</span>}</div><div class="line">第<span class="number">28</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=27109467"</span>}</div><div class="line">第<span class="number">29</span>条书签：{<span class="string">"title"</span>:<span class="string">"从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-string-ansi-unicode-bmp-utf.html?url_type=39&object_type=webpage&pos=1"</span>}</div><div class="line">第<span class="number">30</span>条书签：{<span class="string">"title"</span>:<span class="string">"看了极光推送技术原理的几点思考"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8227469"</span>}</div><div class="line">第<span class="number">31</span>条书签：{<span class="string">"title"</span>:<span class="string">"腾讯微信技术架构"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8581852"</span>}</div><div class="line">第<span class="number">32</span>条书签：{<span class="string">"title"</span>:<span class="string">"鸟哥的 Linux 私房菜 -- 学习 bash shell"</span>,<span class="string">"url"</span>:<span class="string">"http://vbird.dic.ksu.edu.tw/linux_basic/0320bash.php"</span>}</div><div class="line">第<span class="number">33</span>条书签：{<span class="string">"title"</span>:<span class="string">"如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎"</span>,<span class="string">"url"</span>:<span class="string">"https://www.zhihu.com/question/20962240"</span>}</div><div class="line">第<span class="number">34</span>条书签：{<span class="string">"title"</span>:<span class="string">"动态规划：从新手到专家"</span>,<span class="string">"url"</span>:<span class="string">"http://www.hawstein.com/posts/dp-novice-to-advanced.html"</span>}</div><div class="line">第<span class="number">35</span>条书签：{<span class="string">"title"</span>:<span class="string">"商品搜索引擎—推荐系统设计 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/19834.html"</span>}</div><div class="line">第<span class="number">36</span>条书签：{<span class="string">"title"</span>:<span class="string">"多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn"</span>,<span class="string">"url"</span>:<span class="string">"http://coolshell.cn/articles/6790.html"</span>}</div><div class="line">第<span class="number">37</span>条书签：{<span class="string">"title"</span>:<span class="string">"32个算法"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309403979794248269609&jumpfrom=weibocom"</span>}</div><div class="line">第<span class="number">38</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java对象之死-云栖社区"</span>,<span class="string">"url"</span>:<span class="string">"https://yq.aliyun.com/articles/53681?spm=5176.100239.bloglist.55.lOez9A"</span>}</div><div class="line">第<span class="number">39</span>条书签：{<span class="string">"title"</span>:<span class="string">"amao12580 · GitHub"</span>,<span class="string">"url"</span>:<span class="string">"https://github.com/amao12580/"</span>}</div><div class="line">第<span class="number">40</span>条书签：{<span class="string">"title"</span>:<span class="string">"微服务与架构师"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA3MDMwOTcwMg==&mid=2650004550&idx=1&sn=b2299d1f2456541fc6b600a480c27dd8&scene=1&srcid=0714I4IHCwFfqnIGigCv3pJb#rd"</span>}</div><div class="line">第<span class="number">41</span>条书签：{<span class="string">"title"</span>:<span class="string">"图灵社区 : 阅读 : 什么时候你不应该使用微服务"</span>,<span class="string">"url"</span>:<span class="string">"http://www.ituring.com.cn/article/217430"</span>}</div><div class="line">第<span class="number">42</span>条书签：{<span class="string">"title"</span>:<span class="string">"微服务的两种模式：应用中心和任务中心 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/m/article/1335"</span>}</div><div class="line">第<span class="number">43</span>条书签：{<span class="string">"title"</span>:<span class="string">"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7770?f=wb"</span>}</div><div class="line">第<span class="number">44</span>条书签：{<span class="string">"title"</span>:<span class="string">"一个资深架构师是这样理解数据结构的"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"</span>}</div><div class="line">第<span class="number">45</span>条书签：{<span class="string">"title"</span>:<span class="string">"高性能服务器架构思路 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"</span>}</div><div class="line">第<span class="number">46</span>条书签：{<span class="string">"title"</span>:<span class="string">"Java编程常见问题汇总 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"</span>}</div><div class="line">第<span class="number">47</span>条书签：{<span class="string">"title"</span>:<span class="string">"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7816?f=wb"</span>}</div><div class="line">第<span class="number">48</span>条书签：{<span class="string">"title"</span>:<span class="string">"微服务实战：从架构到部署 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1519"</span>}</div><div class="line">第<span class="number">49</span>条书签：{<span class="string">"title"</span>:<span class="string">"优酷蓝鲸近千节点的Redis集群运维经验总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"</span>}</div><div class="line">第<span class="number">50</span>条书签：{<span class="string">"title"</span>:<span class="string">"spring ioc DI 理解 - WhyWin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"https://zm6.sm-tc.cn/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"</span>}</div><div class="line">第<span class="number">51</span>条书签：{<span class="string">"title"</span>:<span class="string">"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1582"</span>}</div><div class="line">第<span class="number">52</span>条书签：{<span class="string">"title"</span>:<span class="string">"技改之路：从单块应用到微服务，我的血泪总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"</span>}</div><div class="line">第<span class="number">53</span>条书签：{<span class="string">"title"</span>:<span class="string">"华为内部如何实施微服务架构？基本就靠这5大原则"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"</span>}</div><div class="line">第<span class="number">54</span>条书签：{<span class="string">"title"</span>:<span class="string">"悲观的看：互联网技术岗位红利期逐渐到头"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309404020851660960391&jumpfrom=weibocom"</span>}</div><div class="line">viaBookmark:</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Google 镜像站搜集 - Techzero"</span>,<span class="string">"url"</span>:<span class="string">"http://www.itechzero.com/google-mirror-sites-collect.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">0</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"OOA、OOD和OOP的定义及之间的关系 - 雲克的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=879a/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1004%2Cta%40iphone_2_5.1_18_/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_oop+ooa+ood/t=iphone/l=3/tc?ref=www_iphone&lid=2022227202196170648&order=5&vit=osres&tj=www_normal_5_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&title=OOAOOD%E5%92%8COOP%E7%9A%84%E5%AE%9A%E4%B9%89%E5%8F%8A%E4%B9%8B%E9%97%B4%E7%9A%84%E5%85%B3%E7%B3%BB-%E9%9B%B2%E5%85%8B%E7%9A%84%E4%B8%AA%E4%BA%BA%E7%A9%BA%E9%97%B4-...&sec=9624&di=1c941110077479c6&bdenc=1&tch=124.237.42.912.2.930&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IGwvPKjtX_j35nI3tfeSaUbBvJ7HNBSzES-7agTCcgsFOd7GdWWUm7hp0rK&eqid=1c1064ed1a08e8001000000556a6ba3c&wd="</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">1</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java NIO和IO的区别_java_脚本之家"</span>,<span class="string">"url"</span>:<span class="string">"http://m.jb51.net/article/50621.htm"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">2</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"成为JavaGC专家Part I -- 深入浅出Java垃圾回收机制 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/1993.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">3</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"java常用设计模式 - Leo Chin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hnrainll/archive/2011/12/29/2305582.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">4</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"正则表达式给力小伙伴 | 行思錄 | Travel Coder"</span>,<span class="string">"url"</span>:<span class="string">"http://liudanking.me/smarttool/regexp_smarttool/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">5</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"进阶正则表达式 - Barret Lee - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/hustskyking/p/how-regular-expressions-work.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">6</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"[一] MQTT,mosquitto,Eclipse Paho-入门 - 操作系统"</span>,<span class="string">"url"</span>:<span class="string">"http://m.myexception.cn/operating-system/1932122.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">7</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"从B 树、B+ 树、B* 树谈到R 树 - 快看小强的个人空间 - 开源中国社区"</span>,<span class="string">"url"</span>:<span class="string">"http://my.oschina.net/u/554660/blog/284653"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">8</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"[存储] Cobar使用文档（可用作MySQL大型集群解决方案） - 石头君专栏 - 博客频道 - CSDN.NET"</span>,<span class="string">"url"</span>:<span class="string">"http://blog.csdn.net/shagoo/article/details/8191346"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">9</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"深圳市五险一金及税后工资计算器"</span>,<span class="string">"url"</span>:<span class="string">"http://salarycalculator.sinaapp.com/city/shenzhen"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">10</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"多图详解Spring框架的设计理念与设计模式(1) - 51CTO.COM"</span>,<span class="string">"url"</span>:<span class="string">"http://developer.51cto.com/art/201006/205212.htm"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">11</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"java中的设计模式如何分类"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/223541008.html?fr=ala&word=Java%20%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F%20%E5%88%86%E7%B1%BB&device=mobile&ssid=0&from=879a&uid=0&pu=usm@0,sz@224_220,ta@iphone___18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_3_0_10_l1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">12</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java 设计模式分类"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=4732764"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">13</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"来自投资银行的20个Java面试题 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/20-java-interview-questions-from-investment-banks.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">14</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"JAVA 中BIO,NIO,AIO的理解 - - ITeye技术网站"</span>,<span class="string">"url"</span>:<span class="string">"http://bbym010.iteye.com/blog/2100868"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">15</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"HAProxy简介 - 张善友 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://www.cnblogs.com/shanyou/archive/2012/10/16/2726768.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">16</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"什么是“极限编程”？"</span>,<span class="string">"url"</span>:<span class="string">"http://zhidao.baidu.com/question/5153690.html?fr=ala&word=%E6%9E%81%E9%99%90%E7%BC%96%E7%A8%8B&device=mobile&ssid=0&from=879a&uid=0&pu=usm@1,sz@1320_1004,ta@iphone_2_5.1_18_&bd_page_type=1&baiduid=4C04AB048A505E6697DCB5C3C808A263&tj=zhidao_5_0_10_l1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">17</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"MongoDB Sharding机制分析 » 囧囧孙"</span>,<span class="string">"url"</span>:<span class="string">"http://www.jiongsun.com/2013/03/45.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">18</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） - 李克华 - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"http://m.baidu.com/from=1001560s/bd_page_type=1/ssid=0/uid=0/pu=usm%400%2Csz%401320_1001%2Cta%40iphone_2_5.1_3_537/baiduid=4C04AB048A505E6697DCB5C3C808A263/w=0_10_kafka%E5%BA%94%E7%94%A8%E5%9C%BA%E6%99%AF/t=iphone/l=3/tc?ref=www_iphone&lid=11304546645738609731&order=1&vit=osres&tj=www_normal_1_0_10_title&m=8&srd=1&cltj=cloud_title&dict=30&nt=wnor&title=kafka%E5%85%A5%E9%97%A8%3A%E7%AE%80%E4%BB%8B%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E8%AE%BE%E8%AE%A1%E5%8E%9F%E7%90%86%E4%B8%BB%E8%A6%81%E9%85%8D..._%E5%8D%9A%E5%AE%A2%E5%9B%AD&sec=10463&di=b0551844ad65db76&bdenc=1&tch=124.0.0.0.0.0&nsrc=IlPT2AEptyoA_yixCFOxXnANedT62v3IEQGG_ytK1DK6mlrte4viZQRAXjb9NX3DIoCb9n00sqcCuX7i_mMe6so4g43&eqid=9ce1d147be1560001000000356d4d7f0&wd=&clk_info=%7B%22xpath%22%3A%22div-a-h3%22%2C%22t%22%3A1456789491134%7D"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">19</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"互联网系统可靠性基础：正确的异常处理"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/UvayInY"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">20</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"服务器端 概述 - 极光文档"</span>,<span class="string">"url"</span>:<span class="string">"http://docs.jpush.io/server/server_overview/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">21</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"从ACID到CAP到BASE - xixicat - SegmentFault"</span>,<span class="string">"url"</span>:<span class="string">"https://segmentfault.com/a/1190000004468442?url_type=39&object_type=webpage&pos=1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">22</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Netty系列之Netty可靠性分析"</span>,<span class="string">"url"</span>:<span class="string">"http://www.infoq.com/cn/articles/netty-reliability/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">23</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"【读后感】Netty系列之Netty高性能之道 - 相比 Mina 如何 ？"</span>,<span class="string">"url"</span>:<span class="string">"http://www.tuicool.com/m/articles/feYfInn"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">24</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"有趣的bloom过滤器 | OurMySQL"</span>,<span class="string">"url"</span>:<span class="string">"http://ourmysql.com/archives/510?f=wb"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">25</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java 并发专题 ： Timer的缺陷 用ScheduledExecutorService替代"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=27109467"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">26</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"从Java String实例来理解ANSI、Unicode、BMP、UTF等编码概念 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-string-ansi-unicode-bmp-utf.html?url_type=39&object_type=webpage&pos=1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">27</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"看了极光推送技术原理的几点思考"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8227469"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">28</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"腾讯微信技术架构"</span>,<span class="string">"url"</span>:<span class="string">"http://m.blog.csdn.net/article/details?id=8581852"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">29</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"鸟哥的 Linux 私房菜 -- 学习 bash shell"</span>,<span class="string">"url"</span>:<span class="string">"http://vbird.dic.ksu.edu.tw/linux_basic/0320bash.php"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">30</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"如何用简单易懂的例子解释隐马尔可夫模型？ - 知乎"</span>,<span class="string">"url"</span>:<span class="string">"https://www.zhihu.com/question/20962240"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">31</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"动态规划：从新手到专家"</span>,<span class="string">"url"</span>:<span class="string">"http://www.hawstein.com/posts/dp-novice-to-advanced.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">32</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"商品搜索引擎—推荐系统设计 - ImportNew"</span>,<span class="string">"url"</span>:<span class="string">"http://www.importnew.com/19834.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">33</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"多版本并发控制(MVCC)在分布式系统中的应用 | 酷 壳 - CoolShell.cn"</span>,<span class="string">"url"</span>:<span class="string">"http://coolshell.cn/articles/6790.html"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">34</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"32个算法"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309403979794248269609&jumpfrom=weibocom"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">35</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java对象之死-云栖社区"</span>,<span class="string">"url"</span>:<span class="string">"https://yq.aliyun.com/articles/53681?spm=5176.100239.bloglist.55.lOez9A"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">36</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"amao12580 · GitHub"</span>,<span class="string">"url"</span>:<span class="string">"https://github.com/amao12580/"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">37</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"微服务与架构师"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA3MDMwOTcwMg==&mid=2650004550&idx=1&sn=b2299d1f2456541fc6b600a480c27dd8&scene=1&srcid=0714I4IHCwFfqnIGigCv3pJb#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">38</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"图灵社区 : 阅读 : 什么时候你不应该使用微服务"</span>,<span class="string">"url"</span>:<span class="string">"http://www.ituring.com.cn/article/217430"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">39</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"微服务的两种模式：应用中心和任务中心 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/m/article/1335"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">40</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"更好的 SQL 模式的 10 条规则 -- MySQL -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7770?f=wb"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">41</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"一个资深架构师是这样理解数据结构的"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597718&idx=1&sn=6eda215ac36671acfa17d8b6d2af4f25#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">42</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"高性能服务器架构思路 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/high-performance-server-artch.html#0-tsina-1-52693-397232819ff9a47a7b7e80a40613cfe1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">43</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"Java编程常见问题汇总 – 码农网"</span>,<span class="string">"url"</span>:<span class="string">"http://www.codeceo.com/article/java-programming-tips.html#0-tsina-1-6630-397232819ff9a47a7b7e80a40613cfe1"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">44</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"技术领导要不要写代码？ -- 奋斗 -- IT技术博客大学习 -- 共学习 共进步！"</span>,<span class="string">"url"</span>:<span class="string">"http://blogread.cn/it/wap/article/7816?f=wb"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">45</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"微服务实战：从架构到部署 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1519"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">46</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"优酷蓝鲸近千节点的Redis集群运维经验总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5NzkxMzg1Nw==&mid=2653160253&idx=1&sn=52576dd6e5acfd7ba140efe67bc0fa6e#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">47</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"spring ioc DI 理解 - WhyWin - 博客园"</span>,<span class="string">"url"</span>:<span class="string">"https://zm6.sm-tc.cn/?src=http%3A%2F%2Fwww.cnblogs.com%2F0201zcr%2Farticles%2F4695490.aspx&uid=7fbca7ed4992d54fb25ab5ad60cfe185&hid=e13a3786fd1f5c68bfee3db4f1a2c643&pos=1&cid=9&time=1470211184068&from=click&restype=1&pagetype=0000004000000402&bu=news_natural&query=Spring+di+%E5%8F%8D%E5%B0%84&mode=&uc_param_str=dnntnwvepffrgibijbprsvdsei"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">48</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"DockOne微信分享（七十一）：基于Docker的负载均衡和服务发现 - DockOne.io"</span>,<span class="string">"url"</span>:<span class="string">"http://dockone.io/article/1582"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">49</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"技改之路：从单块应用到微服务，我的血泪总结"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2650993589&idx=1&sn=df5bdc30b58fd037fa9e83837a742c00"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">50</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"华为内部如何实施微服务架构？基本就靠这5大原则"</span>,<span class="string">"url"</span>:<span class="string">"http://mp.weixin.qq.com/s?__biz=MzA5Nzc4OTA1Mw==&mid=2659597985&idx=1&sn=5ca3f1a574bf61322abca404f64b6b49#rd"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">51</span>}</div><div class="line">{<span class="string">"title"</span>:<span class="string">"悲观的看：互联网技术岗位红利期逐渐到头"</span>,<span class="string">"url"</span>:<span class="string">"http://media.weibo.cn/article?id=2309404020851660960391&jumpfrom=weibocom"</span>,<span class="string">"folder"</span>:<span class="string">""</span>,<span class="string">"order"</span>:<span class="number">52</span>}</div><div class="line">完成解析书签.</div><div class="line"></div><div class="line">Process finished with <span class="keyword">exit</span> code <span class="number">0</span></div></pre></td></tr></table></figure>
<p><img src="/img/via-broswer-bookmarks.jpg" alt=""></p>
<p>解析导出代码，已经整理完毕：<a href="https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java" target="_blank" rel="external">https://github.com/amao12580/JOOQ-With-Spring/blob/master/src/test/java/org/jooq/example/BroswerTest.java</a></p>
<p>有心的小伙伴，完全可以按照这篇文章，写个lite app，帮助神族用户脱坑啦，哈哈哈！Android的可玩性也就体现出来啦！有时间我也做一个App自动化完成吧，maybe later！</p>
<h1 id="UA"><a href="#UA" class="headerlink" title="UA"></a>UA</h1><p>书签导入<a href="http://www.coolapk.com/apk/mark.via" target="_blank" rel="external">Via</a>后的事大家都懂得啦，第一要务就是改UA（User-Agent），分享几个我收集的几个UA。<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line">Mozilla/<span class="number">5.0</span> (BB10; Touch) AppleWebKit/<span class="number">537.10</span>+ (KHTML, like Gecko) Version/<span class="number">10.0</span><span class="number">.9</span><span class="number">.2372</span> Mobile Safari/<span class="number">537.10</span>+（BlackBerry Z30）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; U; Android <span class="number">4.3</span>; en-us; SM-N900T Build/JSS15J) AppleWebKit/<span class="number">534.30</span> (KHTML, like Gecko) Version/<span class="number">4.0</span> Mobile Safari/<span class="number">534.30</span>（Galaxy Note <span class="number">3</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Windows Phone <span class="number">10.0</span>; Android <span class="number">4.2</span><span class="number">.1</span>; Microsoft; Lumia <span class="number">950</span>) AppleWebKit/<span class="number">537.36</span> (KHTML, like Gecko) Chrome/<span class="number">46.0</span><span class="number">.2486</span><span class="number">.0</span> Mobile Safari/<span class="number">537.36</span> Edge/<span class="number">14.14263</span>（Microsoft Lumia <span class="number">950</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; U; en-us; KFAPWI Build/JDQ39) AppleWebKit/<span class="number">535.19</span> (KHTML, like Gecko) Silk/<span class="number">3.13</span> Safari/<span class="number">535.19</span> Silk-Accelerated=true（Kindle Fire HDX）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; Android <span class="number">4.3</span>; Nexus <span class="number">10</span> Build/JSS15Q) AppleWebKit/<span class="number">537.36</span> (KHTML, like Gecko) Chrome/<span class="number">48.0</span><span class="number">.2564</span><span class="number">.23</span> Safari/<span class="number">537.36</span>（Nexus <span class="number">10</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPad; CPU OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPad）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPad; CPU OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPad mini）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPhone; CPU iPhone OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPnone <span class="number">6</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (iPhone; CPU iPhone OS <span class="number">9</span>_1 like Mac OS X) AppleWebKit/<span class="number">601.1</span><span class="number">.46</span> (KHTML, like Gecko) Version/<span class="number">9.0</span> Mobile/<span class="number">13</span>B143 Safari/<span class="number">601.1</span>（iPnone <span class="number">6</span> plus）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; U; Android; unknown; zh-CN) AppleWebKit/<span class="number">534.30</span> (KHTML, like Gecko) Version/<span class="number">4.0</span> UCBrowser/<span class="number">10.10</span><span class="number">.8</span><span class="number">.822</span> U3/<span class="number">0.8</span><span class="number">.0</span> Mobile Safari/<span class="number">534.30</span>（UC）</div><div class="line"></div><div class="line">netdisk;<span class="number">7.0</span><span class="number">.0</span>（百度云手机版<span class="number">7.0</span><span class="number">.0</span>）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.2</span><span class="number">.0</span>（百度云手机版<span class="number">5.2</span><span class="number">.0</span>）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.2</span><span class="number">.7</span>;PC;PC-Windows;<span class="number">6.2</span><span class="number">.9200</span>;WindowsBaiduYunGuanJia（百度云电脑版<span class="number">5.2</span><span class="number">.7</span>）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.4</span><span class="number">.3</span>;PC;PC-Windows;<span class="number">6.2</span><span class="number">.9200</span>;WindowsBaiduYunGuanJia（百度云电脑版<span class="number">5.4</span><span class="number">.3</span>）</div><div class="line"></div><div class="line">Mozilla/<span class="number">5.0</span> (Linux; Android; unknown; zh-CN) AppleWebKit/<span class="number">537.36</span> (KHTML, like Gecko) Version/<span class="number">4.0</span> Chrome/<span class="number">53.0</span><span class="number">.2785</span><span class="number">.80</span> Mobile Safari/<span class="number">537.36</span> MicroMessenger/<span class="number">6.3</span><span class="number">.18</span><span class="number">.800</span> NetType/WIFI Language/zh_CN（微信标识）</div><div class="line"></div><div class="line">netdisk;<span class="number">5.4</span><span class="number">.10</span>;PC;PC-Windows;<span class="number">6.2</span><span class="number">.9200</span>;WindowsBaiduYunGuanJia（百度云电脑版<span class="number">5.4</span><span class="number">.10</span>）</div></pre></td></tr></table></figure></p>
<h1 id="小结"><a href="#小结" class="headerlink" title="小结"></a>小结</h1><p>有编程基础就是好，啥事难不倒！爱折腾，爱android！</p>
]]></content>
<summary type="html">
书签是一个用户的常用功能，会存在添加书签和删除的操作，romer应该会把数据放在一个适合快速存储和妥善保存的地方，比如说：database？还真有/data/data/com.android.broswer/databases/这个目录，啊哈，又近了一步呢！这个文件夹下都有些啥？竟是十多个数据库和临时WAL文件！哪一个才是存放书签的呢？上排除法！根据一些命名过滤掉一些不可能的选项，余下的就是correct。正确答案是：/data/data/com.android.broswer/databases/broswer2.db这个文件.有很多大家熟悉的db：元数据、书签、历史访问、最常访问、图片、最近搜索、用户设置、账户同步。相信大家都知道选哪个了：bookmarks。终于看到需要的信息，所有的书签都在这里哦，想想办法弄出来吧！
</summary>
<category term="Record" scheme="https://amao12580.github.io/categories/Record/"/>
<category term="Geek" scheme="https://amao12580.github.io/tags/Geek/"/>
<category term="SQLite" scheme="https://amao12580.github.io/tags/SQLite/"/>
<category term="JSON" scheme="https://amao12580.github.io/tags/JSON/"/>
<category term="Java" scheme="https://amao12580.github.io/tags/Java/"/>
<category term="Flyme" scheme="https://amao12580.github.io/tags/Flyme/"/>
<category term="Bookmarks" scheme="https://amao12580.github.io/tags/Bookmarks/"/>
</entry>
<entry>
<title>Solr实战 - 生产环境的设计原则</title>
<link href="https://amao12580.github.io/post/2016/09/solr-design-principles/"/>
<id>https://amao12580.github.io/post/2016/09/solr-design-principles/</id>
<published>2016-09-20T01:54:46.123Z</published>
<updated>2016-09-27T07:04:44.169Z</updated>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>中午追番（大力推荐【<a href="https://movie.douban.com/subject/25774052/" target="_blank" rel="external">寄生兽</a>】）没睡觉，现在有点困，洗了把脸，思路恢复清晰了。这篇文章经过数天的腹稿终于要发出来，早上来公司草拟了提纲就先搁置了，没想着一口气写完，但下午想到blog是<a href="http://amao12580.github.io/post/2016/06/How-to-better-write-the-blog/">自动up to date的</a>，不能给个提纲就完事啊！读者的感受在哪里？太不负责了！！！</p>
<p>自从接盘离职同事的代码，一直在做搜索引擎(Solr)方面的工作，都忙成狗了，各个业务线都有搜索的需求，零零碎碎20多个接口吧！维护的过程，看着十多年经验的同事写的代码，有褒有贬，自己也学到了一些小技巧，最最最主要的，对搜索引擎不再陌生，在工作中应用、调优没有问题了。</p>
<p>代码维护是艰难的事，维护shit般的代码是想死的事！如果从头到尾按照统一的规范来构建搜索业务，也许没这么多后续痛苦，看着没有编码规范和零注释的代码，经过数位同事的修改，变得一片浑浊，想理清却已找不到人，只能自己加班看，梳理思路，同时还得整理重构方案！体会到团队的规范性在后期节省了多少维护成本，然而这个收益在前期是看不到或容易被忽视的的，并且在工期和人力的压迫下，是最容易被leader妥协的点！不加规范的代码和设计就像脱缰的野马一样，早期不仔细调教，后面再想驯服就难上加难，也就是技术债务吧！接盘是个苦差事，希望大家谨慎，努力不给后人留坑吧！</p>
<p><img src="/img/rough-and-tumble.jpg" alt=""></p>
<h1 id="Core设计原则"><a href="#Core设计原则" class="headerlink" title="Core设计原则"></a>Core设计原则</h1><h2 id="理解Core"><a href="#理解Core" class="headerlink" title="理解Core"></a>理解Core</h2><p>在Client操作Solr时，通过HTTP RESTful接口完成request-response，使用Core指明操作范围，Core还是RESTful中最后一级资源路径。</p>
<p>Client常用的操作有5种：index、delete、query、optimize，含义同字面意义（自描述）。还有不常用的操作：create、rename、unload、reload、swap、ping、commit、rollback、flush。其中index、delete、query、optimize、rename、unload、reload、swap需要指定Core才可以操作，delete和optimize、flush有些特殊，可以不指定Core，默认对所有Core执行。而create、rename、unload、reload、swap仅在Solr-Admin页面(http ://your solr ip:8983/)上提供，在SolrJ客户端(面向Java语言)无法直接使用，不过可以使用HTTPClient组装HTTP报文模拟操作！</p>
<p>何不将Core想象成MySQL中的Table？在MySQL中，对Table进行CRUD操作时，不是也要加上table name吗？create table “test”… 、alter table “test” … 、select * from “test”…、delete from “test”…大多数的sql需要指定表名，但也有例外：show process list、show variables like ‘character ‘、show innodb engine status;等等。除了在查询时，solr的Core与MySQL的表有相同的功能（事实上Solr的查询远比MySQL的查询强大）。Solr就像是个操作系统，安装在操作系统中的软件就是Core，每个Core有自身的配置文件及数据。在磁盘的角度看，Core是一个文件夹，比如我在线上有名为user和order的2个Core，在Solr的工作目录(%SolrHome%/server/solr)下，有类似的文件目录：</p>
<p><img src="/img/Solr-core-dir-desc.jpg" alt=""><br>可以看到Solr依靠文件夹管理索引文件和日志文件，我们将在后文中说明这些文件的作用和相对路径配置！</p>
<h2 id="设计Core"><a href="#设计Core" class="headerlink" title="设计Core"></a>设计Core</h2><p>既然Core与Table类似，那我们是不是按照传统在MySQL的思路，按照业务功能，拆分出符合三范式的Table呢？例如在MySQL中有user和order两张表。按照这种思路，我们也在Solr建立两个核心user、order，数据一一对应即可？</p>
<p>值得注意的是，Solr对多核心join提供非常有限的支持。</p>
<p>1.仅支持最多2个Core进行join query（formIndex=user toIndex=order）<br>2.仅支持对formIndex中的Core field进行filter query以及response，这是打折的inner join<br>3.两个core都在一个solr instance上</p>
<p>这些限制意味着什么呢？在一次query中，2个核心之间的交互度很低。为了更方便的讲清楚这个问题，以及考虑下文，我们假设现在的user和order核心的结构定义文件（schema.xml）的内容如下:</p>
<figure class="highlight axapta"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">user与<span class="keyword">order</span>是<span class="number">1</span>:N的关系，即：一个用户拥有多个订单，一个订单只从属一个用户</div></pre></td></tr></table></figure>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">user/conf/schema.xml</div><div class="line"></div><div class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"utf-8"</span><span class="meta">?></span></span></div><div class="line"></div><div class="line"><span class="tag"><<span class="name">schema</span> <span class="attr">name</span>=<span class="string">"user"</span> <span class="attr">version</span>=<span class="string">"1.5"</span>></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"_version_"</span> <span class="attr">type</span>=<span class="string">"long"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">uniqueKey</span>></span>Id<span class="tag"></<span class="name">uniqueKey</span>></span></div><div class="line"> <span class="comment"><!-- 用户唯一编号 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"Id"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=普通用户；1=VIP用户 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"type"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 昵称 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"nickname"</span> <span class="attr">type</span>=<span class="string">"simplChinese"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=失效；1=有效 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"status"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 账户最后一次活动时间 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"updateTime"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"string"</span> <span class="attr">class</span>=<span class="string">"solr.StrField"</span> <span class="attr">sortMissingLast</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"int"</span> <span class="attr">class</span>=<span class="string">"solr.TrieIntField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"long"</span> <span class="attr">class</span>=<span class="string">"solr.TrieLongField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"simplChinese"</span> <span class="attr">class</span>=<span class="string">"solr.TextField"</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"index"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"query"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"></<span class="name">fieldType</span>></span></div><div class="line"><span class="tag"></<span class="name">schema</span>></span></div></pre></td></tr></table></figure>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div></pre></td><td class="code"><pre><div class="line">order/conf/schema.xml</div><div class="line"></div><div class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"utf-8"</span><span class="meta">?></span></span></div><div class="line"></div><div class="line"><span class="tag"><<span class="name">schema</span> <span class="attr">name</span>=<span class="string">"order"</span> <span class="attr">version</span>=<span class="string">"1.5"</span>></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"_version_"</span> <span class="attr">type</span>=<span class="string">"long"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">uniqueKey</span>></span>Id<span class="tag"></<span class="name">uniqueKey</span>></span></div><div class="line"> <span class="comment"><!-- 订单唯一编号 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"Id"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=微信订单；1=唯品会订单 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"type"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 订单概览 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"title"</span> <span class="attr">type</span>=<span class="string">"simplChinese"</span>/></span></div><div class="line"> <span class="comment"><!-- 订单备注 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"remark"</span> <span class="attr">type</span>=<span class="string">"simplChinese"</span>/></span></div><div class="line"> <span class="comment"><!-- 下单用户Id --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"uId"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 0=创建；1=付款；2=付款超时 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"status"</span> <span class="attr">type</span>=<span class="string">"int"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 金额、以分为单位 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"amount"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="comment"><!-- 订单最后一次更新时间 --></span></div><div class="line"> <span class="tag"><<span class="name">field</span> <span class="attr">name</span>=<span class="string">"updateTime"</span> <span class="attr">type</span>=<span class="string">"long"</span> <span class="attr">required</span>=<span class="string">"true"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"string"</span> <span class="attr">class</span>=<span class="string">"solr.StrField"</span> <span class="attr">sortMissingLast</span>=<span class="string">"true"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"int"</span> <span class="attr">class</span>=<span class="string">"solr.TrieIntField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"long"</span> <span class="attr">class</span>=<span class="string">"solr.TrieLongField"</span> <span class="attr">precisionStep</span>=<span class="string">"0"</span> <span class="attr">positionIncrementGap</span>=<span class="string">"0"</span>/></span></div><div class="line"></div><div class="line"> <span class="tag"><<span class="name">fieldType</span> <span class="attr">name</span>=<span class="string">"simplChinese"</span> <span class="attr">class</span>=<span class="string">"solr.TextField"</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"index"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"><<span class="name">analyzer</span> <span class="attr">type</span>=<span class="string">"query"</span>></span></div><div class="line"> <span class="tag"><<span class="name">tokenizer</span> <span class="attr">class</span>=<span class="string">"com.chenlb.mmseg4j.solr.MMSegTokenizerFactory"</span> <span class="attr">mode</span>=<span class="string">"max-word"</span> <span class="attr">dicPath</span>=<span class="string">"../../dic"</span>/></span></div><div class="line"> <span class="tag"><<span class="name">filter</span> <span class="attr">class</span>=<span class="string">"solr.StopFilterFactory"</span> <span class="attr">words</span>=<span class="string">"stopwords.txt"</span>/></span></div><div class="line"> <span class="tag"></<span class="name">analyzer</span>></span></div><div class="line"> <span class="tag"></<span class="name">fieldType</span>></span></div><div class="line"><span class="tag"></<span class="name">schema</span>></span></div></pre></td></tr></table></figure>
<h2 id="超级Core"><a href="#超级Core" class="headerlink" title="超级Core"></a>超级Core</h2><p>假设有需求功能：”查询有已付款订单的用户昵称和用户Id”，类似的SQL：SELECT u.Id,u.nickname FROM user u INNER JOIN order o ON u.Id=o.uId WHERE o.status=1;<a href="/post/2016/09/solr-design-principles/#研究Join-query">这样简单的查询需求是可以完成的</a>，但对于复杂需求，例如”查询在今天活跃过的用户和这些用户在今天的最新一笔有效微信订单，要求订单金额不能低于5元，需要打印用户和订单基本信息”，这种就无法满足了！如果必须要复杂join query，常规思路是加新Core，如userAndOrder，主体信息为user和order，因为1:N的关系，需要新加自定义联合主键Id，在index阶段，以用户和订单两个角度来维护数据。在单个用户角度，删除该用户的所有数据（deleteByQuery(“uId:1000”)），重新查询订单信息，index多条。在单个订单角度，删除一条数据（deleteByQuery(“orderId:5000”)），查询所属用户，index一条。为了减少冗余，原有的user Core和order Core都可以不保留。方案的不足之处在于数据冗余，这将造成超级Core现象。比如还有一个product Core（商品信息），order与product是N:M的关系，按这种思路，难道还是聚合吗？考虑一种极限情况：假设user数据有十万，order数据有一千万，product数据有一亿，平均1个用户有100个订单，每个订单有50个商品，这样3表聚合userAndOrderAndProduct Core数据量在50亿左右，平均每个user的信息被重复了50万次！冗余带来的去重问题同样严重，result set进行distinct、groupBy、facet的代价将会极其高昂！</p>
<p>由于user、order、product的关系可以视为单方向，即order一定从属于user，product从属于order。改进的做法是将user的所有order视为一个<a href="https://cwiki.apache.org/confluence/dosearchsite.action?where=solr&spaceSearch=true&queryString=child+document" target="_blank" rel="external">child document set</a>处理，以此类推，有3层递进关系，此时的问题是product会存在冗余问题，相较于full-mix方案冗余度降低！</p>
<h2 id="定位Solr"><a href="#定位Solr" class="headerlink" title="定位Solr"></a>定位Solr</h2><p>产生超级Core问题是错误地定位Solr，Solr在信息检索领域是高效的，如多维度检索、关键词建议、关键词高亮、关键词补全、关键词纠错、更多相关、同义词、近义词、多义词、停用词、保护词、语义转换、空间搜索等等，但在复杂关系型查询以及强事务处理方面不如传统数据库，如MySQL！不应该将复杂join query交给solr来处理，这也侧面验证专业的工具解决特定领域的问题。除了在关键词方面的处理能力，Solr还对分维度权重、部分匹配、相关度等高度复杂需求可以轻松应对，但是搜索领域目前还存在的难题，Solr也无法解决，需要凭借外部辅助系统，例如：自动发现新词、识别流行词、歧义消除、搜索预测等等。目前都是通过NLP辅以海量语料库预处理，结合HDFS等平台海量计算发现。</p>
<p>我们在设计Core时，应理清需求，思考方案与Solr的契合度，保持KISS原则，尽可能的避免超级 Core。</p>
<h1 id="字段设计原则"><a href="#字段设计原则" class="headerlink" title="字段设计原则"></a>字段设计原则</h1><p>字段的含义与MySQL具备类似性，如：name、type、requireed、default。</p>
<p>Solr为了自身的索引高效与内存控制，还加上了很多属性：</p>
<ul>
<li><p>required：是否为必须字段，默认为false，是否必需，对应MySQL NOT NULL</p>
</li>
<li><p>default：index时字段未填，使用这个默认值，常见的应用如：sequence number</p>
</li>
<li><p>multiValued：是否为多值，字段可以是List，但泛型收主type控制，多个值只能是同一种类型，默认值是false</p>
</li>
<li><p>indexed：是否索引，参与到query计算的字段必需设置为true，默认值是true</p>
</li>
<li><p>stored：是否保留原始数据，如果字段需要在query时使用原始值，则需要设置为true，默认值是true</p>
</li>
<li><p>docValues：<a href="https://cwiki.apache.org/confluence/display/solr/DocValues" target="_blank" rel="external">针对大数据量复杂计算加速</a>，使用document-to-value数据结构，例如：需要聚合的字段，包括sort，agg，group，facet等、需要提供函数查询的字段、需要高亮的字段、自定义评分的字段。</p>
</li>
</ul>
<p>这些是常用属性，更多的信息参阅官方wiki：<a href="https://cwiki.apache.org/confluence/display/solr/Defining+Fields" target="_blank" rel="external">https://cwiki.apache.org/confluence/display/solr/Defining+Fields</a></p>
<h2 id="字段的分类"><a href="#字段的分类" class="headerlink" title="字段的分类"></a>字段的分类</h2><p>1.按照结构分类：field、copyField、dynamicField<br><figure class="highlight vim"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">copyField</div><div class="line"></div><div class="line">需要将一个或多个字段的数据汇总到多个字段时用到。典型的作为默认搜索时设计。新闻类<span class="built_in">index</span>应用中，client不指定qf参数时，默认从title、author、catalog、content字段中综合检索。<span class="keyword">source</span>和destination都支持通配符，maxChars，<span class="keyword">int</span>类型参数，用于配置复制字符数的上限。注意：如果dest由多个<span class="keyword">source</span>构成，就需要将其指定为multiValued。</div><div class="line"><copyField <span class="keyword">source</span>=<span class="string">"*t"</span> dest=<span class="string">"content"</span> maxChars=<span class="string">"30000"</span> /></div><div class="line"></div><div class="line"></div><div class="line">dynamicField</div><div class="line"></div><div class="line">动态字段（Dynamic fields）允许 solr 索引没有在 schema 中明确定义的字段。这个在忘记定义一些字段时很有用。动态字段可以让系统更灵活，通用性更强。它主要用作扩展备用，因为添加Field需要重建索引，而这可能是一个漫长的过程（而且没法使用增量更新索引），建议设计之初，多备一些dynamicField。</div><div class="line"></div><div class="line">Solr在Field <span class="keyword">list</span>中没法<span class="keyword">match</span>到query指定的field时，会尝试从dynamicField <span class="keyword">list</span>中<span class="keyword">match</span>，如果没有配置dynamicField则跳过。</div><div class="line"><dynamicField name=<span class="string">"*_i"</span> <span class="built_in">type</span>=<span class="string">"sint"</span> indexed=<span class="string">"true"</span> stored=<span class="string">"true"</span>/></div></pre></td></tr></table></figure></p>
<p>2.按照计算进行分类：keyword、filter、sort、function、join query<br><figure class="highlight stata"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line">1. keyword</div><div class="line">查询字段</div><div class="line"></div><div class="line"><span class="comment">//标题、副标题、区域、商圈、号码、其他号码</span></div><div class="line"><span class="keyword">query</span>.<span class="keyword">set</span>(<span class="string">"qf"</span>, <span class="string">"title"</span>, <span class="string">"slogan"</span>, <span class="string">"expectDistrictName"</span>, <span class="string">"expectTradeAreaName"</span>, <span class="string">"mobile"</span>, <span class="string">"otherContact"</span>);</div><div class="line"></div><div class="line">2. filter</div><div class="line">过滤字段</div><div class="line"></div><div class="line"><span class="keyword">query</span>.addFilterQuery(<span class="string">"maxRent:[* TO 100]"</span>);</div><div class="line"></div><div class="line">3. <span class="keyword">sort</span></div><div class="line">排序字段</div><div class="line"></div><div class="line"><span class="keyword">List</span><SolrQuery.SortClause> sorts = new ArrayList<>();</div><div class="line">sorts.add(SolrQuery.SortClause.<span class="keyword">desc</span>(<span class="string">"hasCharged"</span>));</div><div class="line">sorts.add(SolrQuery.SortClause.<span class="keyword">desc</span>(<span class="string">"updateTime"</span>));</div><div class="line"><span class="keyword">query</span>.setSorts(sorts);</div><div class="line"></div><div class="line">4. function & boost function</div><div class="line">函数、评分、权重</div><div class="line"></div><div class="line"><span class="keyword">query</span>.addSort(<span class="string">"if(isCertificated,1,0)"</span>, SolrQuery.<span class="keyword">ORDER</span>.<span class="keyword">desc</span>);</div><div class="line"><span class="keyword">query</span>.addSort(<span class="string">"hasSalesinProduct"</span>, SolrQuery.<span class="keyword">ORDER</span>.<span class="keyword">desc</span>);</div><div class="line"><span class="keyword">query</span>.addSort(<span class="string">"map(geodist(),0,5,5,map(geodist(),5.00000000001,20,4,map(geodist(),20.000001,50,3,map(geodist(),50.00000000001,100,2,map(geodist(),100.00000000001,100000,1,0)))))"</span>, SolrQuery.<span class="keyword">ORDER</span>.<span class="keyword">desc</span>);</div><div class="line"></div><div class="line"></div><div class="line">5. join <span class="keyword">query</span></div><div class="line">Core 联接查询，field用于联接条件</div><div class="line"></div><div class="line">{!join from=uId to=Id fromIndex=<span class="keyword">order</span> toIndex=user}status:1</div></pre></td></tr></table></figure></p>
<h2 id="字段的命名规范"><a href="#字段的命名规范" class="headerlink" title="字段的命名规范"></a>字段的命名规范</h2><p>一般来说命名规范的目的是：前期降低开发成本，后期减少维护成本。在前期可以省去（在某些场景下甚至不用）编写各式说明文档，keep focus on业务开发、技术攻坚；在后期在维护工作转手时，降低交接成本、理解成本。</p>
<p>与程序开发规范一致，在solr中，我们只需要对命名保持标准统一和易于理解，能做到简短有力就更好了。</p>
<ul>
<li><p>可读性<br>避开毫无意义的命名：i,j</p>
</li>
<li><p>无二义性<br>避免歧义，如：缩写往往会产生多义的问题，一个缩写词可能在不同人理解时，引发语义上不一致(<a href="https://zh.wikipedia.org/wiki/%E9%9F%93%E5%9C%8B%E9%AB%98%E9%80%9F%E9%90%B5%E9%81%93#.E8.BB.BC.E4.BA.8B" target="_blank" rel="external">KTX 09年建设事故</a>)。</p>
</li>
<li><p>统一<br>统一大小写，混合时推荐使用：小驼峰式。统一特殊字符使用规范，如慎用”*”（与dynamicField冲突），保持高效。</p>
</li>
<li><p>简洁性<br>在能说明用途的前提下，保持尽可能短。</p>
</li>
<li><p>使用常用词<br>降低理解成本（思维惯性）。这也是约定大于配置的一种表现吗？</p>
</li>
</ul>
<h2 id="字段的存储结构"><a href="#字段的存储结构" class="headerlink" title="字段的存储结构"></a>字段的存储结构</h2><p>Index document时，solr首先对field(取决于analyzer配置)进行分词，创建index库和document库。所谓的分词是指：将字符文本按照一定的规则分成若干个单词。</p>
<h3 id="Index库"><a href="#Index库" class="headerlink" title="Index库"></a>Index库</h3><p>lucene的倒排索引存储结构为：词项的字符串+词项的文档频率+记录词项的频率信息+记录词项的位置信息+跳跃偏移量。</p>
<h3 id="Document库"><a href="#Document库" class="headerlink" title="Document库"></a>Document库</h3><p>lucene词典中词的顺序是按照英文字母的顺序排列的，这样就可以采用压缩存储：假设有term，termagancy，termagant，termina四个词。每个字母需要1byte的空间，常规存储一共需要35byte。而压缩存储之后为：”term4agancy8t4inal”，一共需要22byte。通过DRY原则，节省大量的空间，这一点与<a href="https://github.com/amao12580/algorithm/blob/master/src/main/java/tree/huffman/HuffmanTree.java" target="_blank" rel="external">Huffman编码</a>思想一致。</p>
<h2 id="Additional-field"><a href="#Additional-field" class="headerlink" title="Additional field"></a>Additional field</h2><p>在实际使用中，为了满足QPS要求，将不参与计算的字段进行数据冗余，以便更快速的响应接口。实际上就是cache一部分信息在solr，避免发起RPC从外部源获取，减少进程上下文切换和网络往返时间。</p>
<p>典型场景是：图片审核，按照用户是否有头像过滤，response头像URL。常见的设计，在user Core中，有字段：id（int型），用户编号、hasHeadPhoto（boolean型），标识用户是否有头像。另一字段headPhotoURL(String型)，头像URI。此时headPhotoURL就是我们说的Additional field。在cache等基础设施不完善时，常这样处理。随着业务的复杂，Additional field不加以限制，会使用的越来越广泛，以至于后续的schema.xml中，参与计算的field反而占少数了。冗余URL这种可预估长度并且低频率变化的field还好理解，我还见过将用户个人说明（simplChinese：Text）进行冗余的，这完全不能接受了，拖累了所有涉及document的RT以及index rebuild。建议在前期基础设施不完善时，可以适量的加入Additional field（技术债务），但请务必在后续完善后，留出重构时间fix，并安排技术沟通会，讨论这种妥协方案的改进办法。</p>
<h2 id="Mix-core"><a href="#Mix-core" class="headerlink" title="Mix core"></a>Mix core</h2><p>单个Core为了兼容多个业务场景使用，将不同类别的业务数据混合塞入。不同的业务场景之间同时存在重叠和交叉的情况（不存在业务重叠是不需要放到一起的）。为了达到数据兼容，程序在index和query两个阶段都需要进行compatible。这样的Core我们称之为Mix core。</p>
<ul>
<li><p>field级别，有字段只对特定业务才会用到（垂直冗余）<br>对不同类型的Client提供不同级别的Query</p>
</li>
<li><p>row级别，有些数据记录对特定业务才会用到（水平冗余）<br>数据聚合类应用中，对所有结果进行计算：top N、sort、groupBy</p>
</li>
</ul>
<p>由于Solr高效检索的前提是将index文件load到内存进行计算，为了说明row数据量以及field分布与index大小的关系（间接分析memory以及disk的占用关系），统计了一份仿真环境的数据分布，供参考：</p>
<figure class="highlight lsl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">一些说明：</div><div class="line"><span class="number">1.</span>未计算dynamicField分布，实际项目中仅用作版本过渡，没有长期使用。</div><div class="line"><span class="number">2.</span>index大小和tlog大小是指相应dir所有文件大小之和。</div><div class="line"><span class="number">3.</span>不存在未commit的数据。</div><div class="line"><span class="number">4.</span><span class="string">"field分布"</span>，这一列中，<span class="string">"6：TrieIntField；"</span>，表示存在<span class="number">6</span>个solr.TrieIntField类型的字段。</div><div class="line"><span class="number">5.</span>仅一个solr节点（singleton node模式），不启用cloud，也意味着没有replica。</div><div class="line"><span class="number">6.</span>所有类型的field的length均不超过<span class="number">256</span>位，多数在<span class="number">11</span>位以内。</div></pre></td></tr></table></figure>
<p><img src="/img/Solr-core-field-analysis.jpg" alt="仿真环境的数据分布"></p>
<p>从中不难发现：最为常用的是整形，1个普通row约占用1 KB的空间，由于需要在数据量增长时，不线性降低solr查询性能。我们需要保持：混合核心重叠部分的field和row处于大比例，仅仅将极少数用作业务兼容！</p>
<h1 id="SolrIndexSearcher"><a href="#SolrIndexSearcher" class="headerlink" title="SolrIndexSearcher"></a>SolrIndexSearcher</h1><p>Solr查询的核心类就是SolrIndexSearcher，在同一时刻只由当前的SolrIndexSearcher供上层的handler使用（当切换SolrIndexSearcher时可能会有两个同时提供服务），而各种Cache是依附于SolrIndexSearcher的，SolrIndexSearcher在则Cache生，SolrIndexSearcher亡则被clear（solrIndexSearcher.close()）。<br>在solr4.0之后，有2种commit，hard commit、soft commit。Hard commit操作会触发SolrIndexSearcher切换，而close操作隐含flush AND commit，也会触发。</p>
<p>AutoCommit，Commit操作也可以按条件自动完成（hard commit），maxDocs：按未commit的最大文档数量，event watch机制；maxTime：按上一次未commit距离现在的时间，定时器。同样提供API，手工commit，这样性能比较差，平均每次commit 600ms；<br>AutoCommit 参考：<a href="http://wiki.apache.org/solr/SolrConfigXml" target="_blank" rel="external">http://wiki.apache.org/solr/SolrConfigXml</a>，solrconfig.xml配置：<br><figure class="highlight dts"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="params"><updateHandler class="solr.DirectUpdateHandler2"></span></div><div class="line"> <span class="params"><updateLog></span></div><div class="line"> <span class="params"><str name="dir"></span>${solr.ulog.dir:}<span class="params"></str></span></div><div class="line"> <span class="params"><int name="numVersionBuckets"></span>${solr.ulog.numVersionBuckets:<span class="number">65536</span>}<span class="params"></int></span></div><div class="line"> <span class="params"></updateLog></span></div><div class="line"> <span class="params"><autoCommit></span></div><div class="line"> <span class="params"><maxTime></span>${solr.autoCommit.maxTime:<span class="number">15000</span>}<span class="params"></maxTime></span></div><div class="line"> <span class="params"><openSearcher></span>false<span class="params"></openSearcher></span></div><div class="line"> <span class="params"></autoCommit></span></div><div class="line"> <span class="params"><autoSoftCommit></span></div><div class="line"> <span class="params"><maxTime></span>${solr.autoSoftCommit.maxTime:<span class="number">1000</span>}<span class="params"></maxTime></span></div><div class="line"> <span class="params"></autoSoftCommit></span></div><div class="line"></div><div class="line"> <span class="params"><!--</span></div><div class="line"><span class="params"> <listener event="postCommit" class="solr.RunExecutableListener"></span></div><div class="line"> <span class="params"><str name="exe"></span>solr<span class="meta-keyword">/bin/</span>snapshooter<span class="params"></str></span></div><div class="line"> <span class="params"><str name="dir"></span>.<span class="params"></str></span></div><div class="line"> <span class="params"><bool name="wait"></span>true<span class="params"></bool></span></div><div class="line"> <span class="params"><arr name="args"></span> <span class="params"><str></span>arg1<span class="params"></str></span> <span class="params"><str></span>arg2<span class="params"></str></span> <span class="params"></arr></span></div><div class="line"> <span class="params"><arr name="env"></span> <span class="params"><str></span>MYVAR=val1<span class="params"></str></span> <span class="params"></arr></span></div><div class="line"> <span class="params"></listener></span></div><div class="line"> --></div><div class="line"><span class="params"></updateHandler></span></div></pre></td></tr></table></figure></p>
<p>Hard commit时，除了向Directory对象（实际是Disk proxy）提交索引变化(new tlog)，SolrIndexSearcher需要重新建立。commit提交后，index文件flush到硬盘（flush：从内存刷回磁盘保存：fsync），并触发listener，建立new SolrIndexSearcher(新的insexReader，从硬盘中load index)，这样后续的Query使用new SolrIndexSearcher。建议不要频繁修改document，特别是要避免大批量reload，影响RT稳定和服务中断。</p>
<p>Hard commit时，如果有配置auto-warm（autowarmCount=？），则会对new SolrIndexSearcher swap一定数量的old SolrIndexSearcher index文件；而soft commit是在NRT（Near Real Time）实时搜索中提出的(如log、analysis)，不会flush到disk，也可以使得document被搜索到，代价比hard commit要小的多；对实时性要求比较高的场景下，可以做soft commit操作，不过还是要定时hard commit，确保索引持久化到disk。遭遇突发性故障时（机器断电），未flush到disk的数据可能会丢失，在updatelog开启时（updatelog是Solr的概念，在Lucene并没有出现）,solr启动时会进行recover check，尽可能的恢复数据。</p>
<h1 id="Cache机制"><a href="#Cache机制" class="headerlink" title="Cache机制"></a>Cache机制</h1><p>如果把Schema定义为Solr的Model的话，那么Solrconfig.xml就是Solr的Configuration，它定义Solr如果处理索引、高亮、搜索等很多请求，同时还配置cache策略。</p>
<p>solrconfig.xml<br><figure class="highlight dust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div></pre></td><td class="code"><pre><div class="line"><span class="xml">1.指定索引文件的存储路径</span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">dataDir</span>></span>$</span><span class="template-variable">{solr.data.dir:./solr/data}</span><span class="xml"><span class="tag"></<span class="name">dataDir</span>></span></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">2.缓存配置</span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">Solr在Lucene之上开发了很多Cache功能，目前提供的Cache类型有：</span></div><div class="line"><span class="xml"><span class="comment"><!-- 1.filterCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">filterCache</span> <span class="attr">class</span>=<span class="string">"solr.FastLRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> initialSize="512"</span></div><div class="line"><span class="xml"> autowarmCount="0"/></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 2.queryResultCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">queryResultCache</span> <span class="attr">class</span>=<span class="string">"solr.LRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> initialSize="512"</span></div><div class="line"><span class="xml"> autowarmCount="0"/></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 3.documentCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">documentCache</span> <span class="attr">class</span>=<span class="string">"solr.LRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> initialSize="512"</span></div><div class="line"><span class="xml"> autowarmCount="0"/></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 4.自定义缓存策略针对数据块交换 --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">cache</span> <span class="attr">name</span>=<span class="string">"perSegFilter"</span></span></span></div><div class="line"><span class="xml"> class="solr.search.LRUCache"</span></div><div class="line"><span class="xml"> size="10"</span></div><div class="line"><span class="xml"> initialSize="0"</span></div><div class="line"><span class="xml"> autowarmCount="10"</span></div><div class="line"><span class="xml"> regenerator="solr.NoOpRegenerator" /></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml"><span class="comment"><!-- 5.fieldValueCache --></span></span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">fieldValueCache</span> <span class="attr">class</span>=<span class="string">"solr.FastLRUCache"</span></span></span></div><div class="line"><span class="xml"> size="512"</span></div><div class="line"><span class="xml"> autowarmCount="128"</span></div><div class="line"><span class="xml"> showItems="32" /></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">// 是否能使用到filtercache关键配置</span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">useFilterForSortedQuery</span>></span>true<span class="tag"></<span class="name">useFilterForSortedQuery</span>></span></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">// queryresult的结果集控制</span></div><div class="line"><span class="xml"> <span class="tag"><<span class="name">queryResultWindowSize</span>></span>20<span class="tag"></<span class="name">queryResultWindowSize</span>></span></span></div><div class="line"><span class="xml"></span></div><div class="line"><span class="xml">// 是否启用懒加载field</span></div><div class="line"><span class="xml"><span class="tag"><<span class="name">enableLazyFieldLoading</span>></span>true<span class="tag"></<span class="name">enableLazyFieldLoading</span>></span></span></div></pre></td></tr></table></figure></p>
<h2 id="LRU-VS-FastLRU"><a href="#LRU-VS-FastLRU" class="headerlink" title="LRU VS FastLRU"></a>LRU VS FastLRU</h2><p>Solr提供了两种SolrCache接口实现类：solr.search.LRUCache和solr.search.FastLRUCache。FastLRUCache是1.4版本中引入的，其速度在普遍意义上要比LRUCache更快些。</p>
<p>LRUCache可配置参数：<br>1）size：cache中可保存的最大的项数，默认是1024。<br>2）initialSize：cache初始化时的大小，默认是1024。</p>
<p>3）autowarmCount：当系统启动时（firstSearcher）或切换SolrIndexSearcher时，可以对新生成的SolrIndexSearcher做autowarm（预热）处理。autowarmCount表示从旧的SolrIndexSearcher中取多少项来在新的SolrIndexSearcher中被重新生成，如何重新生成由CacheRegenerator实现。</p>
<p>查看Solr源码可以发现，在实现上，LRUCache直接使用LinkedHashMap来缓存数据，由initialSize来限定cache的大小，淘汰策略也是使用LinkedHashMap的内置的LRU方式，读写操作都是对map的全局锁，所以并发性效果方面稍差。</p>
<p>filterCache和fieldValueCache使用FastLRUCache实现，FastLRUCache内部使用了ConcurrentLRUCache来缓存数据，它是个加了LRU淘汰策略的ConcurrentHashMap，所以其并发性要好很多，这也是多数Java版Cache的极典型实现。</p>
<h2 id="filterCache"><a href="#filterCache" class="headerlink" title="filterCache"></a>filterCache</h2><p>filterCache中存储了无序的lucene document Id集合，即FilterCache存储了一些无序的文档id，这些Id并不是我们在schema.xml里配置的unique key，而是solr内部的一个文档标识。filterCache存储了filter queries(“fq”参数)得到的document Id集合结果。Solr中的query参数有两种，即q和fq。如果fq存在，Solr是先查询fq（因为fq可以多个，所以多个fq查询是个取结果交集的过程（Map-reduce模型）)，之后将fq结果和q结果取并。在这一过程中，filterCache就是key为单个fq（类型为Query），value为document Id集合（类型为DocSet）的cache。从后面的分析你将会看到对于fq为range query来说，filterCache将表现出其更有价值的一面。</p>
<h2 id="queryResultCache"><a href="#queryResultCache" class="headerlink" title="queryResultCache"></a>queryResultCache</h2><p>顾名思义，queryResultCache是对查询结果的缓存（SolrIndexSearcher中的cache缓存的都是document Id set），这个结果就是针对查询条件的完全有序的结果。因为查询参数是有start和rows的，所以某个QueryResultKey可能命中了cache，但start和rows却不在cache的document Id set范围内。当然，document Id set是越大命中的概率越大，但这也会很浪费内存，这就需要个参数：queryResultWindowSize来指定document Id set的大小。</p>
<h2 id="documentCache"><a href="#documentCache" class="headerlink" title="documentCache"></a>documentCache</h2><p>documentCache用来保存”doc_Id,document”键值对(正排索引)。如果使用documentCache，就尽可能开大些，至少要大过max_results * max_concurrent_queries，否则因为cache的淘汰，一次请求期间还需要重新获取document一次。也要注意document中存储的字段的多少，避免大量的内存消耗。</p>
<h1 id="数据维护原则"><a href="#数据维护原则" class="headerlink" title="数据维护原则"></a>数据维护原则</h1><p>在类ELK准实时日志分析方案中，各节点的logs被聚合到Solr中存储，在内存积累一定批次后，触发hard commit持久化到disk，在disk积累到一定量后进行转移（或直接删除），不涉及到update业务。而在类LBS（Location-Based Service）应用中，经纬度和用户信息被定时上传到database存储，然后sync到Solr，document可能被频繁update。提交document到Solr索引后，那修改documents的策略是什么呢？最简单的，Solr提供full-replace的方式进行更新，如果配置为覆盖旧版本在，则先按照uniqueKey找到old document并标记为删除，让后add new document，这主要是lucene是按照“delete-add”的模式来维护数据。另外，Solr支持document按field局部更新。</p>
<h2 id="Atomic-Updates"><a href="#Atomic-Updates" class="headerlink" title="Atomic Updates"></a>Atomic Updates</h2><p>局部更新允许修改一个或多个field，而不必重新索引整个document。在multiValued类型的field中，它甚至允许按正则表达式局部更新value，以便加速Solr对index的索引处理，减轻QPS波动。</p>
<p><img src="/img/solr-atomic-updates.jpg" alt=""></p>
<h2 id="Optimistic-Concurrency-Control"><a href="#Optimistic-Concurrency-Control" class="headerlink" title="Optimistic Concurrency Control"></a>Optimistic Concurrency Control</h2><p><a href="https://www.google.com.hk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0ahUKEwi3qqHH3qzPAhUJs48KHYRaBPQQFggpMAE&url=https%3A%2F%2Fzh.wikipedia.org%2Fzh-cn%2F%25E4%25B9%2590%25E8%25A7%2582%25E5%25B9%25B6%25E5%258F%2591%25E6%258E%25A7%25E5%2588%25B6&usg=AFQjCNHJ4Y6Fpckt8rQxpRc5NSC_CHQtIA" target="_blank" rel="external">乐观并发控制(OCC)，或称乐观锁</a>。它是一种多数NoSQL数据库具备的特性，它允许client基于version number按条件并发update same document，为了防止ABA问题，Solr使用字段”_version_“来控制document版本，为了可比较，被设置为数值型，默认情况下，这个字段被定义在schema.xml。client在提交document时，不需要提供_version_的值，它在document每次发生变化时，被Solr自动修改，修改的值全局唯一。</p>
<figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><field <span class="attribute">name</span>=<span class="string">"_version_"</span> <span class="attribute">type</span>=<span class="string">"long"</span>/></div></pre></td></tr></table></figure>
<p>在默认配置下，”_version_“配置为”indexed=true”，对于某些操作系统来说，海量的document，FieldCache增加需要消耗太多内存。由于该字段不参与计算，可以关闭index，将”_version_“定义为DocValues。<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><field <span class="attribute">name</span>=<span class="string">"_version_"</span> <span class="attribute">type</span>=<span class="string">"long"</span> <span class="attribute">indexed</span>=<span class="string">"false"</span> <span class="attribute">stored</span>=<span class="string">"true"</span> <span class="attribute">required</span>=<span class="string">"true"</span> <span class="attribute">docValues</span>=<span class="string">"true"</span>/></div></pre></td></tr></table></figure></p>
<h2 id="Field-Storage"><a href="#Field-Storage" class="headerlink" title="Field Storage"></a>Field Storage</h2><p>如果field想要使用atomic update功能，在schema.xml中，除了copyField的destinations需要配置为“stored=’false’”之外，目标field必需配置为“stored=’true’”。在copyField中，Solr会index所有的source fields最新值。如果copyField的destinations配置为“stored=’true’”，如果同时修改source fields和destination field的值，那destination field就会看起来像被source fields修改的值覆盖一样。</p>
<h2 id="小结"><a href="#小结" class="headerlink" title="小结"></a>小结</h2><p>document被修改后相应的documentCache、fieldCache被clear，被删除后filterCache、queryResultCache、fieldCache需要update。应该谨慎对待update操作，尽可能少的发起hard commit操作，减少solrIndexSearcher rebuild。对少量field更新采取atomic update的方式进行，对于大量field更新采取full-replace策略。</p>
<h1 id="索引重建原则"><a href="#索引重建原则" class="headerlink" title="索引重建原则"></a>索引重建原则</h1><p>减少服务中断时间</p>
<p>C端流量控制</p>
<h2 id="单个Solr集群实例"><a href="#单个Solr集群实例" class="headerlink" title="单个Solr集群实例"></a>单个Solr集群实例</h2><p>Core要有版本号<br>opportunity_V1</p>
<p>1.先创建新Core<br>opportunity_V2</p>
<p>2.额外进程构建V2</p>
<p>3.检查V2构建成功</p>
<p>4.流量切换到V2 或删除(unload)V1后 v2-rename-v1</p>
<p>V2 swap V1</p>
<p>5.确认V1Core不再需要后再销毁<br>同时运行多版本时，谨慎</p>
<h2 id="多个Solr集群"><a href="#多个Solr集群" class="headerlink" title="多个Solr集群"></a>多个Solr集群</h2><p>先构建好新集群</p>
<p>流量切换到新集群</p>
<p>旧集群停机</p>
<h1 id="使用总结"><a href="#使用总结" class="headerlink" title="使用总结"></a>使用总结</h1><h2 id="命令行启动"><a href="#命令行启动" class="headerlink" title="命令行启动"></a>命令行启动</h2><figure class="highlight taggerscript"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">F:<span class="symbol">\s</span>olr<span class="symbol">\b</span>in>solr start -m 1g -f</div></pre></td></tr></table></figure>
<h2 id="清空index"><a href="#清空index" class="headerlink" title="清空index"></a>清空index</h2><figure class="highlight xml"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">curl http://<span class="tag"><<span class="name">host</span>></span>:<span class="tag"><<span class="name">port</span>></span>/<span class="tag"><<span class="name">solr_base</span>></span>/update?commit=true -d '<span class="tag"><<span class="name">delete</span>></span><span class="tag"><<span class="name">query</span>></span>*:*<span class="tag"></<span class="name">query</span>></span><span class="tag"></<span class="name">delete</span>></span>'</div></pre></td></tr></table></figure>
<h2 id="Core-别名"><a href="#Core-别名" class="headerlink" title="Core 别名"></a>Core 别名</h2><p>在文件/core.properties中，定义name字段，既可作为别名。同时还可以设置使用的solrconfig.xml路径，以达到多Core共用同一份solrconfig.xml的目的，schema.xml也是类似的设置。</p>
<p>dataDir可以配置Core数据目录，以达到程序与数据分离的效果，方便备份和容灾。<br><figure class="highlight ini"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="comment">#我的Core文件夹名是：order，设置别名为myOrder。在client中就可以使用myOrder来调用我的Core</span></div><div class="line"><span class="attr">name</span>=myOrder</div><div class="line"><span class="attr">config</span>=solrconfig.xml</div><div class="line"><span class="attr">schema</span>=schema.xml</div><div class="line"><span class="attr">dataDir</span>=data</div></pre></td></tr></table></figure></p>
<h2 id="查询技巧"><a href="#查询技巧" class="headerlink" title="查询技巧"></a>查询技巧</h2><p>1.只查询字段orderStatus值在1到3的数据<br>方案1.orderStatus可以是int、string类型<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="attribute">q</span>=*:*&fq={!frange <span class="attribute">l</span>=1 <span class="attribute">u</span>=3}orderStatus</div></pre></td></tr></table></figure></p>
<p>方案2.orderStatus确定是整数<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">q=*<span class="symbol">:*&fq=orderStatus</span><span class="symbol">:</span>[<span class="number">1</span> TO <span class="number">3</span>]</div></pre></td></tr></table></figure></p>
<h1 id="研究Join-query"><a href="#研究Join-query" class="headerlink" title="研究Join query"></a>研究Join query</h1><p>附上研究Join query的实战代码<br><figure class="highlight processing"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div></pre></td><td class="code"><pre><div class="line"><span class="comment">//测试用的Solr环境为：</span></div><div class="line"><span class="comment">// solr 6.1</span></div><div class="line"><span class="comment">// jetty容器</span></div><div class="line"><span class="comment">//启动参数：solr start -m 1g -f</span></div><div class="line"><span class="comment">//windows 7 X64</span></div><div class="line"><span class="comment">//solrj maven: <groupId>org.apache.solr</groupId> <artifactId>solr-solrj</artifactId> <version>5.3.0</version></span></div><div class="line"><span class="comment">//<luceneMatchVersion>6.1.0</luceneMatchVersion></span></div><div class="line"></div><div class="line"><span class="keyword">String</span> solrHostName = <span class="string">"http://127.0.0.1:8983/solr/"</span>;</div><div class="line">SolrClient userClient = <span class="keyword">new</span> HttpSolrClient(solrHostName + <span class="string">"user"</span>);</div><div class="line">SolrClient orderClient = <span class="keyword">new</span> HttpSolrClient(solrHostName + <span class="string">"order"</span>);</div><div class="line"></div><div class="line">List<SolrInputDocument> userDocuments = <span class="keyword">new</span> ArrayList<>();</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"添加用户“张三”"</span>);</div><div class="line">SolrInputDocument userDocument = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">userDocument.addField(<span class="string">"Id"</span>, <span class="number">1000</span>);<span class="comment">//用户唯一编号</span></div><div class="line">userDocument.addField(<span class="string">"type"</span>, <span class="number">0</span>);<span class="comment">//0=普通用户；1=VIP用户</span></div><div class="line">userDocument.addField(<span class="string">"nickname"</span>, <span class="string">"张三"</span>);<span class="comment">//昵称</span></div><div class="line">userDocument.addField(<span class="string">"status"</span>, <span class="number">0</span>);<span class="comment">//0=失效；1=有效</span></div><div class="line">userDocument.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis());<span class="comment">//账户最后一次活动时间</span></div><div class="line">userDocuments.<span class="built_in">add</span>(userDocument);</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"添加用户“李四”"</span>);</div><div class="line">SolrInputDocument userDocument2 = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">userDocument2.addField(<span class="string">"Id"</span>, <span class="number">2000</span>);<span class="comment">//用户唯一编号</span></div><div class="line">userDocument2.addField(<span class="string">"type"</span>, <span class="number">1</span>);<span class="comment">//0=普通用户；1=VIP用户</span></div><div class="line">userDocument2.addField(<span class="string">"nickname"</span>, <span class="string">"李四"</span>);<span class="comment">//昵称</span></div><div class="line">userDocument2.addField(<span class="string">"status"</span>, <span class="number">0</span>);<span class="comment">//0=失效；1=有效</span></div><div class="line">userDocument2.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis() + <span class="number">1</span>);<span class="comment">//账户最后一次活动时间</span></div><div class="line">userDocuments.<span class="built_in">add</span>(userDocument2);</div><div class="line">userClient.<span class="built_in">add</span>(userDocuments);</div><div class="line"></div><div class="line"><span class="comment">//为"张三"加了2条订单信息</span></div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"用户“李四”添加2条订单信息"</span>);</div><div class="line">List<SolrInputDocument> orderDocuments = <span class="keyword">new</span> ArrayList<>();</div><div class="line">SolrInputDocument orderDocument = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">orderDocument.addField(<span class="string">"Id"</span>, <span class="number">3000</span>);<span class="comment">//订单唯一编号</span></div><div class="line">orderDocument.addField(<span class="string">"type"</span>, <span class="number">0</span>);<span class="comment">//0=微信订单；1=唯品会订单</span></div><div class="line">orderDocument.addField(<span class="string">"title"</span>, <span class="string">"张三的订单：商品001、商品002"</span>);<span class="comment">//订单概览</span></div><div class="line">orderDocument.addField(<span class="string">"remark"</span>, <span class="string">"我是备注..."</span>);<span class="comment">//订单备注</span></div><div class="line">orderDocument.addField(<span class="string">"uId"</span>, <span class="number">1000</span>);<span class="comment">//下单用户Id</span></div><div class="line">orderDocument.addField(<span class="string">"status"</span>, <span class="number">0</span>);<span class="comment">//0=创建；1=付款；2=付款超时</span></div><div class="line">orderDocument.addField(<span class="string">"amount"</span>, <span class="number">188</span>);<span class="comment">//金额、以分为单位</span></div><div class="line">orderDocument.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis() + <span class="number">2</span>);<span class="comment">//订单最后一次更新时间</span></div><div class="line">orderDocuments.<span class="built_in">add</span>(orderDocument);</div><div class="line">SolrInputDocument orderDocument2 = <span class="keyword">new</span> SolrInputDocument();</div><div class="line">orderDocument2.addField(<span class="string">"Id"</span>, <span class="number">4000</span>);<span class="comment">//订单唯一编号</span></div><div class="line">orderDocument2.addField(<span class="string">"type"</span>, <span class="number">1</span>);<span class="comment">//0=微信订单；1=唯品会订单</span></div><div class="line">orderDocument2.addField(<span class="string">"title"</span>, <span class="string">"张三的订单：商品041、商品082"</span>);<span class="comment">//订单概览</span></div><div class="line">orderDocument2.addField(<span class="string">"remark"</span>, <span class="string">"我是备注..*******."</span>);<span class="comment">//订单备注</span></div><div class="line">orderDocument2.addField(<span class="string">"uId"</span>, <span class="number">1000</span>);<span class="comment">//下单用户Id</span></div><div class="line">orderDocument2.addField(<span class="string">"status"</span>, <span class="number">1</span>);<span class="comment">//0=创建；1=付款；2=付款超时</span></div><div class="line">orderDocument2.addField(<span class="string">"amount"</span>, <span class="number">9800</span>);<span class="comment">//金额、以分为单位</span></div><div class="line">orderDocument2.addField(<span class="string">"updateTime"</span>, System.currentTimeMillis() + <span class="number">3</span>);<span class="comment">//订单最后一次更新时间</span></div><div class="line">orderDocuments.<span class="built_in">add</span>(orderDocument2);</div><div class="line">orderClient.<span class="built_in">add</span>(orderDocuments);</div><div class="line"></div><div class="line"><span class="comment">//准实时索引大约需要1s time window</span></div><div class="line"><span class="keyword">try</span> {</div><div class="line"> TimeUnit.SECONDS.sleep(<span class="number">2</span>);</div><div class="line">} <span class="keyword">catch</span> (InterruptedException e) {</div><div class="line"> e.printStackTrace();</div><div class="line">}</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"完成“张三”订单add"</span>);</div><div class="line"><span class="comment">//查询有已付款订单的用户昵称和用户Id</span></div><div class="line"><span class="comment">//类似的SQL：SELECT u.Id,u.nickname FROM user u INNER JOIN order o ON u.Id=o.uId WHERE o.status=1;</span></div><div class="line"></div><div class="line">SolrQuery query = <span class="keyword">new</span> SolrQuery();</div><div class="line">query.addField(<span class="string">"Id"</span>);</div><div class="line">query.addField(<span class="string">"nickname"</span>);</div><div class="line">query.setQuery(<span class="string">"*:*"</span>);</div><div class="line"><span class="comment">//status:1作为filter是必需的，不填会报错</span></div><div class="line"><span class="comment">//*Index与当前client相同时可省略</span></div><div class="line"><span class="comment">//query.addFilterQuery("{!join from=uId to=Id fromIndex=order toIndex=user}status:1");</span></div><div class="line">query.addFilterQuery(<span class="string">"{!join from=uId to=Id fromIndex=order}status:1"</span>);</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"query params:"</span> + query.toString());</div><div class="line">SolrDocumentList result = userClient.query(query).getResults();</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"result size:"</span> + result.<span class="built_in">size</span>());</div><div class="line">System.out.<span class="built_in">println</span>(<span class="string">"查询有已付款订单的用户昵称和用户Id result:"</span> + Utils.objectToJsonWithoutException(result));</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">----------------------------------------------------------------------------</div><div class="line"></div><div class="line">console：</div><div class="line">添加用户“张三”</div><div class="line">添加用户“李四”</div><div class="line">用户“李四”添加<span class="number">2</span>条订单信息</div><div class="line">完成“张三”订单<span class="built_in">add</span></div><div class="line">query params:fl=Id%<span class="number">2</span>Cnickname&q=*%<span class="number">3</span>A*&fq=%<span class="number">7</span>B%<span class="number">21</span><span class="built_in">join</span>+from%<span class="number">3</span>DuId+to%<span class="number">3</span>DId+fromIndex%<span class="number">3</span>Dorder%<span class="number">7</span>Dstatus%<span class="number">3</span>A1</div><div class="line">result <span class="built_in">size</span>:<span class="number">1</span></div><div class="line">查询有已付款订单的用户昵称和用户Id result:[{<span class="string">"Id"</span>:<span class="number">1000</span>,<span class="string">"nickname"</span>:<span class="string">"张三"</span>}]</div></pre></td></tr></table></figure></p>
<h1 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h1><ul>
<li><a href="https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-JoinQueryParser" target="_blank" rel="external">Solr join query wiki</a></li>
<li><a href="http://josh-persistence.iteye.com/blog/2247289" target="_blank" rel="external">Solr cache分析</a></li>
<li><a href="http://blog.csdn.net/yangbutao/article/details/9179347" target="_blank" rel="external">Solr API 分析</a></li>
<li><a href="http://aoyouzi.iteye.com/blog/2291913" target="_blank" rel="external">Solr dataStructure 分析</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents#UpdatingPartsofDocuments-AtomicUpdates" target="_blank" rel="external">Solr update index 策略</a></li>
</ul>
]]></content>
<summary type="html">
产生超级Core问题是错误地定位Solr，Solr在信息检索领域是高效的，如多维度检索、关键词建议、关键词高亮、关键词补全、关键词纠错、更多相关、同义词、近义词、多义词、停用词、保护词、语义转换、空间搜索等等，但在复杂关系型查询以及强事务处理方面不如传统数据库，如MySQL！不应该将复杂join query交给solr来处理，这也侧面验证专业的工具解决特定领域的问题。除了在关键词方面的处理能力，Solr还对分维度权重、部分匹配、相关度等高度复杂需求可以轻松应对，但是搜索领域目前还存在的难题，Solr也无法解决，需要凭借外部辅助系统，例如：自动发现新词、识别流行词、歧义消除、搜索预测等等。目前都是通过NLP辅以海量语料库预处理，结合HDFS等平台海量计算发现。我们在设计Core时，应理清需求，思考方案与Solr的契合度，保持KISS原则，尽可能的避免超级 Core。
</summary>
<category term="Summary" scheme="https://amao12580.github.io/categories/Summary/"/>
<category term="Solr" scheme="https://amao12580.github.io/tags/Solr/"/>
<category term="Design" scheme="https://amao12580.github.io/tags/Design/"/>
</entry>
<entry>
<title>开源贡献 - 升级Solr中文分词器 mmseg4j-solr</title>
<link href="https://amao12580.github.io/post/2016/07/mmseg4j-solr-compatible-solr-6-1/"/>
<id>https://amao12580.github.io/post/2016/07/mmseg4j-solr-compatible-solr-6-1/</id>
<published>2016-07-25T04:43:45.943Z</published>
<updated>2016-07-25T06:09:59.526Z</updated>
<content type="html"><![CDATA[<h2 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h2><p>最近因为团队职责变动，接手了Solr的维护工作。仔细比对各个环境的使用版本才发现，都是使用的5.3，查了一下这个就快发布一年了。为什么不用最新版的6.1呢？有点忍不住技痒了，整一下升级到6.1的方案吧！</p>
<h2 id="简单介绍Solr"><a href="#简单介绍Solr" class="headerlink" title="简单介绍Solr"></a>简单介绍Solr</h2><p>Apache Solr (读音: SOLer) 是一个开源的搜索服务器。Solr 使用 Java 语言开发，主要基于 HTTP 和 Apache Lucene 实现。Apache Solr 中存储的资源是以 Document 为对象进行存储的。每个文档由一系列的 Field 构成，每个 Field 表示资源的一个属性。Solr 中的每个 Document 需要有能唯一标识其自身的属性，默认情况下这个属性的名字是 id，在 Schema 配置文件中使用：id进行描述。 Solr是一个高性能，采用Java开发，基于Lucene的全文搜索服务器。文档通过Http利用XML加到一个搜索集合中。查询该集合也是通过 http收到一个XML/JSON响应来实现。它的主要特性包括：高效、灵活的缓存功能，垂直搜索功能，高亮显示搜索结果，通过索引复制来提高可用性，提 供一套强大Data Schema来定义字段，类型和设置文本分析，提供基于Web的管理界面等。</p>
<h2 id="什么是mmseg4j-solr？"><a href="#什么是mmseg4j-solr？" class="headerlink" title="什么是mmseg4j-solr？"></a>什么是mmseg4j-solr？</h2><p>mmseg4j 用 Chih-Hao Tsai 的 <a href="http://technology.chtsai.org/mmseg/" target="_blank" rel="external">MMSeg 算法</a>实现的中文分词器，并实现 lucene 的 analyzer 和 solr 的TokenizerFactory 以方便在Lucene和Solr中使用。</p>
<p>2、MMSeg 算法有两种分词方法：Simple和Complex，都是基于正向最大匹配。Complex 加了四个规则过虑。官方说：词语的正确识别率达到了 98.41%。mmseg4j 已经实现了这两种分词算法。 <em> 1.5版的分词速度simple算法是 1100kb/s左右、complex算法是 700kb/s左右，（测试机：AMD athlon 64 2800+ 1G内存 xp）。 </em> 1.6版在complex基础上实现了最多分词(max-word)。“很好听” -> “很好|好听”; “中华人民共和国” -> “中华|华人|共和|国”; “中国人民银行” -> “中国|人民|银行”。 <em> 1.7-beta 版, 目前 complex 1200kb/s左右, simple 1900kb/s左右, 但内存开销了50M左右. 上几个版都是在10M左右. </em> 1.8 后,增加 CutLetterDigitFilter过虑器，切分“字母和数”混在一起的过虑器。</p>
<h2 id="升级方案"><a href="#升级方案" class="headerlink" title="升级方案"></a>升级方案</h2><h3 id="尝试强制升级"><a href="#尝试强制升级" class="headerlink" title="尝试强制升级"></a>尝试强制升级</h3><p>首先尝试直接强制升级。下载6.1版的zip，解压后，将5.3版本的核心配置文件、lib包拷贝并覆盖过来，启动时大量报错了：</p>
<p>lib包主要是:</p>
<ul>
<li>mmseg4j-core-1.10.0.jar</li>
<li>mmseg4j-solr-2.3.0.jar</li>
<li>pinyin4j-2.5.jar</li>
<li>pinyinFilter-0.2.jar</li>
</ul>
<p><img src="/img/solr-libs.png" alt=""></p>
<figure class="highlight applescript"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">启动solr时报错</div><div class="line"></div><div class="line">cd /solr/bin</div><div class="line">solr start -m <span class="number">1</span>g -f</div><div class="line"></div><div class="line"><span class="number">2016</span><span class="number">-07</span><span class="number">-25</span> <span class="number">05</span>:<span class="number">03</span>:<span class="number">45.776</span> WARN (coreLoadExecutor<span class="number">-6</span>-thread<span class="number">-1</span>) [ ] o.a.s.s.FieldTypePluginLoader TokenFilterFactory <span class="keyword">is</span> using deprecated <span class="number">5.3</span><span class="number">.0</span> emulation. You should <span class="keyword">at</span> <span class="keyword">some</span> point declare <span class="keyword">and</span> reindex <span class="keyword">to</span> <span class="keyword">at</span> least <span class="number">6.0</span>, because <span class="number">5.</span>x emulation <span class="keyword">is</span> deprecated <span class="keyword">and</span> will be removed <span class="keyword">in</span> <span class="number">7.0</span></div><div class="line"><span class="number">2016</span><span class="number">-07</span><span class="number">-25</span> <span class="number">05</span>:<span class="number">03</span>:<span class="number">45.783</span> ERROR (coreLoadExecutor<span class="number">-6</span>-thread<span class="number">-2</span>) [ ] o.a.s.c.CoreContainer Error creating core [opportunity]: org.apache.solr.core.SolrResourceLoader.getInstanceDir()Ljava/lang/String;</div><div class="line">java.lang.NoSuchMethodError: org.apache.solr.core.SolrResourceLoader.getInstanceDir()Ljava/lang/String;</div><div class="line"> <span class="keyword">at</span> com.chenlb.mmseg4j.solr.Utils.getDict(Utils.java:<span class="number">18</span>)</div><div class="line"> <span class="keyword">at</span> com.chenlb.mmseg4j.solr.MMSegTokenizerFactory.inform(MMSegTokenizerFactory.java:<span class="number">65</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:<span class="number">699</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:<span class="number">184</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:<span class="number">56</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:<span class="number">75</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:<span class="number">108</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:<span class="number">79</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.CoreContainer.create(CoreContainer.java:<span class="number">810</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.core.CoreContainer.lambda$load$<span class="number">0</span>(CoreContainer.java:<span class="number">466</span>)</div><div class="line"> <span class="keyword">at</span> java.util.concurrent.FutureTask.<span class="built_in">run</span>(FutureTask.java:<span class="number">266</span>)</div><div class="line"> <span class="keyword">at</span> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$<span class="number">22</span>(ExecutorUtil.java:<span class="number">229</span>)</div><div class="line"> <span class="keyword">at</span> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:<span class="number">1142</span>)</div><div class="line"> <span class="keyword">at</span> java.util.concurrent.ThreadPoolExecutor$Worker.<span class="built_in">run</span>(ThreadPoolExecutor.java:<span class="number">617</span>)</div><div class="line"> <span class="keyword">at</span> java.lang.Thread.<span class="built_in">run</span>(Thread.java:<span class="number">745</span>)</div></pre></td></tr></table></figure>
<p>看样子是mmseg4j-solr.jar包中，文件com.chenlb.mmseg4j.solr.Utils.java，第18行有方法调用错误，猜测可能是因为API变动，导致版本不兼容。</p>
<p>Google一些关于mmseg4j-solr对solr版本的兼容情况，发现mmseg4j-solr还停留在对5.3版本的支持，没人维护了，那我们自食其力，看能否解决？</p>
<p>好在<a href="https://github.com/chenlb/mmseg4j-solr/blob/master/README.md(https://github.com/chenlb/mmseg4j-solr/blob/master/README.md" target="_blank" rel="external">mmseg4j-solr已经在GitHub开放了源码</a>，开始干吧！</p>
<h3 id="改源码进行兼容"><a href="#改源码进行兼容" class="headerlink" title="改源码进行兼容"></a>改源码进行兼容</h3><p>首先fork源码，clone到本机，对文件com.chenlb.mmseg4j.solr.Utils.java，第18行修改，这一行果然报Compile error。这就完了吗？试试package一下吧！maven install过程中，执行MMSegTokenizerFactoryTest case也报错了。</p>
<figure class="highlight vhdl"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">Tests run: <span class="number">1</span>, Failures: <span class="number">0</span>, Errors: <span class="number">1</span>, Skipped: <span class="number">0</span>, <span class="built_in">Time</span> elapsed: <span class="number">10.921</span> sec <<< <span class="literal">FAILURE</span>! - <span class="keyword">in</span> com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest</div><div class="line">com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest <span class="built_in">Time</span> elapsed: <span class="number">10.905</span> sec <<< <span class="literal">ERROR</span>!</div><div class="line">java.lang.RuntimeException: org.apache.solr.common.SolrException: Solr no longer supports forceful unlocking via the <span class="symbol">'unlockOnStartup</span>' option. This <span class="keyword">is</span> no longer necessary <span class="keyword">for</span> the <span class="keyword">default</span> lockType except <span class="keyword">in</span> situations where it would be dangerous <span class="keyword">and</span> should <span class="keyword">not</span> be done. <span class="keyword">For</span> other lockTypes <span class="keyword">and</span>/<span class="keyword">or</span> directoryFactory options it may also be dangerous <span class="keyword">and</span> users must resolve problematic locks manually.</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.beforeClass(MMSegTokenizerFactoryTest.java:<span class="number">33</span>)</div><div class="line">Caused by: org.apache.solr.common.SolrException: Solr no longer supports forceful unlocking via the <span class="symbol">'unlockOnStartup</span>' option. This <span class="keyword">is</span> no longer necessary <span class="keyword">for</span> the <span class="keyword">default</span> lockType except <span class="keyword">in</span> situations where it would be dangerous <span class="keyword">and</span> should <span class="keyword">not</span> be done. <span class="keyword">For</span> other lockTypes <span class="keyword">and</span>/<span class="keyword">or</span> directoryFactory options it may also be dangerous <span class="keyword">and</span> users must resolve problematic locks manually.</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.beforeClass(MMSegTokenizerFactoryTest.java:<span class="number">33</span>)</div><div class="line"></div><div class="line"></div><div class="line">Results :</div><div class="line"></div><div class="line">Tests <span class="keyword">in</span> <span class="literal">error</span>:</div><div class="line"> MMSegTokenizerFactoryTest.beforeClass:<span class="number">33</span>->SolrTestCaseJ4.initCore:<span class="number">443</span>->SolrTestCaseJ4.initCore:<span class="number">436</span>->SolrTestCaseJ4.initCore:<span class="number">594</span>->SolrTestCaseJ4.createCore:<span class="number">601</span> ? Runtime</div><div class="line"></div><div class="line">Tests run: <span class="number">6</span>, Failures: <span class="number">0</span>, Errors: <span class="number">1</span>, Skipped: <span class="number">0</span></div><div class="line"></div><div class="line">[INFO] <span class="comment">------------------------------------------------------------------------</span></div><div class="line">[INFO] BUILD <span class="literal">FAILURE</span></div><div class="line">[INFO] <span class="comment">------------------------------------------------------------------------</span></div><div class="line">[INFO] Total <span class="built_in">time</span>: <span class="number">17.375</span> s</div><div class="line">[INFO] Finished at: <span class="number">2016</span>-<span class="number">07</span>-<span class="number">25</span>T13:<span class="number">18</span>:<span class="number">43</span>+<span class="number">08</span>:<span class="number">00</span></div><div class="line">[INFO] Final Memory: <span class="number">32</span>M/<span class="number">218</span>M</div><div class="line">[INFO] <span class="comment">------------------------------------------------------------------------</span></div><div class="line">[<span class="literal">ERROR</span>] Failed <span class="keyword">to</span> execute goal org.apache.maven.plugins:maven-surefire-plugin:<span class="number">2.19</span>.<span class="number">1</span>:test (<span class="keyword">default</span>-test) <span class="keyword">on</span> project mmseg4j-solr: There are test failures.</div><div class="line">[<span class="literal">ERROR</span>]</div><div class="line">[<span class="literal">ERROR</span>] Please refer <span class="keyword">to</span> D:\GitHub\mmseg4j-solr\target\surefire-reports <span class="keyword">for</span> the individual test results.</div><div class="line">[<span class="literal">ERROR</span>] -> [Help <span class="number">1</span>]</div><div class="line">[<span class="literal">ERROR</span>]</div><div class="line">[<span class="literal">ERROR</span>] <span class="keyword">To</span> see the full stack trace <span class="keyword">of</span> the errors, re-run Maven <span class="keyword">with</span> the -e switch.</div><div class="line">[<span class="literal">ERROR</span>] Re-run Maven using the -X switch <span class="keyword">to</span> enable full debug logging.</div><div class="line">[<span class="literal">ERROR</span>]</div><div class="line">[<span class="literal">ERROR</span>] <span class="keyword">For</span> more information about the errors <span class="keyword">and</span> possible solutions, please read the following articles:</div><div class="line">[<span class="literal">ERROR</span>] [Help <span class="number">1</span>] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException</div></pre></td></tr></table></figure>
<p>看样子是unlockOnStartup选项不再受支持，需要移除，我们移除一下试试吧，找到该选项位于Test Resource包下的文件solrconfig.xml中，删掉了。再尝试install！还是报错！</p>
<figure class="highlight stata"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">Tests <span class="keyword">run</span>: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.586 sec <<< FAILURE! - <span class="keyword">in</span> com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest</div><div class="line">test_mmseg4j(com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest) Time elapsed: 0.012 sec <<< <span class="keyword">ERROR</span>!</div><div class="line">org.apache.solr.common.SolrException: SolrCore 'mmseg4j_core' is not available due to init failure: <span class="keyword">Error</span> loading <span class="keyword">class</span> 'org.apache.solr.handler.admin.AdminHandlers'</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.getDictionaryByFieldType(MMSegTokenizerFactoryTest.java:37)</div><div class="line"> at com.chenlb.mmseg4j.solr.MMSegTokenizerFactoryTest.test_mmseg4j(MMSegTokenizerFactoryTest.java:79)</div><div class="line">Caused <span class="keyword">by</span>: org.apache.solr.common.SolrException: <span class="keyword">Error</span> loading <span class="keyword">class</span> 'org.apache.solr.handler.admin.AdminHandlers'</div><div class="line">Caused <span class="keyword">by</span>: org.apache.solr.common.SolrException: <span class="keyword">Error</span> loading <span class="keyword">class</span> 'org.apache.solr.handler.admin.AdminHandlers'</div><div class="line">Caused <span class="keyword">by</span>: java.lang.ClassNotFoundException: org.apache.solr.handler.admin.AdminHandlers</div><div class="line"></div><div class="line"></div><div class="line">Results :</div><div class="line"></div><div class="line">Tests <span class="keyword">in</span> <span class="keyword">error</span>:</div><div class="line"> MMSegTokenizerFactoryTest.test_mmseg4j:79->getDictionaryByFieldType:37 ? Solr ...</div><div class="line"></div><div class="line">Tests <span class="keyword">run</span>: 6, Failures: 0, Errors: 1, Skipped: 0</div><div class="line"></div><div class="line">[INFO] ------------------------------------------------------------------------</div><div class="line">[INFO] BUILD FAILURE</div><div class="line">[INFO] ------------------------------------------------------------------------</div><div class="line">[INFO] <span class="keyword">Total</span> time: 6.131 <span class="built_in">s</span></div><div class="line">[INFO] Finished at: 2016-07-25T13:20:50+08:00</div><div class="line">[INFO] Final <span class="keyword">Memory</span>: 23M/226M</div><div class="line">[INFO] ------------------------------------------------------------------------</div><div class="line">[<span class="keyword">ERROR</span>] Failed to execute goal org.apache.maven.plugins:maven-surefire-<span class="keyword">plugin</span>:2.19.1:<span class="keyword">test</span> (default-<span class="keyword">test</span>) <span class="keyword">on</span> project mmseg4j-solr: There are <span class="keyword">test</span> failures.</div><div class="line">[<span class="keyword">ERROR</span>]</div><div class="line">[<span class="keyword">ERROR</span>] Please refer to <span class="keyword">D</span>:\GitHub\mmseg4j-solr\target\surefire-reports <span class="keyword">for</span> the individual <span class="keyword">test</span> results.</div><div class="line">[<span class="keyword">ERROR</span>] -> [<span class="keyword">Help</span> 1]</div><div class="line">[<span class="keyword">ERROR</span>]</div><div class="line">[<span class="keyword">ERROR</span>] To see the full <span class="keyword">stack</span> trace of the errors, re-<span class="keyword">run</span> Maven with the -<span class="keyword">e</span> switch.</div><div class="line">[<span class="keyword">ERROR</span>] Re-<span class="keyword">run</span> Maven using the -X switch to enable full debug logging.</div><div class="line">[<span class="keyword">ERROR</span>]</div><div class="line">[<span class="keyword">ERROR</span>] <span class="keyword">For</span> <span class="keyword">more</span> information <span class="keyword">about</span> the errors and possible solutions, please <span class="keyword">read</span> the following articles:</div><div class="line">[<span class="keyword">ERROR</span>] [<span class="keyword">Help</span> 1] http:<span class="comment">//cwiki.apache.org/confluence/display/MAVEN/MojoFailureException</span></div></pre></td></tr></table></figure>
<p>看样子是org.apache.solr.handler.admin.AdminHandlers无法加载，该配置位于solrconfig.xml。google发现该api已经换了，与5.3版不再兼容。找了一份6.1的basic solrconfig.xml进行替换，再试着install就成功啦！</p>
<h2 id="PR"><a href="#PR" class="headerlink" title="PR"></a>PR</h2><p>看来改源码，也没有很难嘛！持续几个问题解决后，我们也得到了兼容性的jar包，实验在测试环境可以用！考虑到很多人也需要版本升级，本着取之于开源，反哺开源，则不竭的精神。对源repo提交了pull request。PR地址：<a href="https://github.com/chenlb/mmseg4j-solr/pull/26" target="_blank" rel="external">https://github.com/chenlb/mmseg4j-solr/pull/26</a>。</p>
<p>该PR还没有merger到master，如果等不及请下载我提供的：<a href="/file/mmseg4j-solr-2.3.2.rar">mmseg4j-solr-2.3.2.jar</a></p>
]]></content>
<summary type="html">
Google关于mmseg4j-solr对solr版本的兼容情况，发现mmseg4j-solr还停留在对5.3版本的支持，没人维护了，那我们自食其力，看能否解决？最终还是成功完成，看来改源码，也没有很难嘛！持续几个问题解决后，我们也得到了兼容性的jar包，实验在测试环境可以用！考虑到很多人也需要版本升级，本着取之于开源，反哺开源，则不竭的精神。对源repo提交了pull request。
</summary>
<category term="Record" scheme="https://amao12580.github.io/categories/Record/"/>
<category term="Solr" scheme="https://amao12580.github.io/tags/Solr/"/>
<category term="Mmseg4j-solr" scheme="https://amao12580.github.io/tags/Mmseg4j-solr/"/>
<category term="Open-Source-Contribution" scheme="https://amao12580.github.io/tags/Open-Source-Contribution/"/>
</entry>
<entry>
<title>MySQL集锦 - IN 真会导致全表扫描吗？</title>
<link href="https://amao12580.github.io/post/2016/07/MySQL-in-operator-must-lead-to-full-scan/"/>
<id>https://amao12580.github.io/post/2016/07/MySQL-in-operator-must-lead-to-full-scan/</id>
<published>2016-07-04T06:47:12.807Z</published>
<updated>2016-07-05T02:07:14.562Z</updated>
<content type="html"><![CDATA[<h1 id="写在前面"><a href="#写在前面" class="headerlink" title="写在前面"></a>写在前面</h1><p>项目版本迭代慢了下来，有了段空闲时间，就对近些阶段的代码进行Code review，主要还是保持设计正确性和易维护性吧。</p>
<h1 id="有意思的代码"><a href="#有意思的代码" class="headerlink" title="有意思的代码"></a>有意思的代码</h1><p>由于积累了数个里程碑，review的工作量还挺大，我们分工协作，我这块主要看推荐模块，发现了几段有意思的代码，贴出来看看吧。</p>
<figure class="highlight lasso"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">Result<Record1<<span class="built_in">Integer</span>>> ids = create.<span class="keyword">select</span>(SITING_OPPORTUNITY_DISTRICT.OPPORTUNITY_ID).</div><div class="line"> from(SITING_OPPORTUNITY_DISTRICT).</div><div class="line"> <span class="keyword">where</span>(SITING_OPPORTUNITY_DISTRICT.DISTRICT_ID.like(districtId + <span class="string">"%"</span>)).</div><div class="line"> groupBy(SITING_OPPORTUNITY_DISTRICT.OPPORTUNITY_ID).</div><div class="line"> fetch();</div><div class="line"><span class="built_in">Set</span><<span class="built_in">Integer</span>> opportunityIds = <span class="literal">new</span> HashSet<>(ids.size());</div><div class="line">for (Record1<<span class="built_in">Integer</span>> r : ids) {</div><div class="line"> opportunityIds.add(r.getValue(SITING_OPPORTUNITY_DISTRICT.OPPORTUNITY_ID));</div><div class="line">}</div><div class="line"><span class="keyword">if</span> (opportunityIds.size() == <span class="number">1</span>) {</div><div class="line"> condition = condition.<span class="literal">and</span>(SITING_OPPORTUNITY.OPPORTUNITY_ID.<span class="literal">eq</span>(getSetFirstElement(opportunityIds)));</div><div class="line">} <span class="keyword">else</span> {</div><div class="line"> condition = condition.<span class="literal">and</span>(SITING_OPPORTUNITY.OPPORTUNITY_ID.<span class="keyword">in</span>(opportunityIds));</div><div class="line">}</div></pre></td></tr></table></figure>
<p>这段代码的大意是：在MySQL数据库，表SITING_OPPORTUNITY_DISTRICT中，查询符合DISTRICT_ID字段条件的不重复OPPORTUNITY_ID字段，将这些字段放到一个集合。为描述方便，这个集合下文用ids代替。</p>
<p>1.如果ids的大小等于一，则拼上了一个SQL where条件，使用equal过滤ids中的第一个元素值，我们称这个SQL为A。</p>
<figure class="highlight n1ql"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">SQL A template</div><div class="line"></div><div class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> SITING_OPPORTUNITY <span class="keyword">WHERE</span> OPPORTUNITY_ID=?;</div></pre></td></tr></table></figure>
<p>2.如果ids的大小大于一，则拼上了一个SQL where条件，使用in过滤ids中的所有元素值，我们称这个SQL为B。</p>
<figure class="highlight n1ql"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">SQL B template</div><div class="line"></div><div class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> SITING_OPPORTUNITY <span class="keyword">WHERE</span> OPPORTUNITY_ID <span class="keyword">in</span>(?,?,?,?,?...);</div></pre></td></tr></table></figure>
<h1 id="存疑"><a href="#存疑" class="headerlink" title="存疑"></a>存疑</h1><p>项目中，类似这样的写法大概有10多处，都是按照集合大小来动态build SQL语句。ids的大小是跟用户的个性化数据有关的，统计了一下，超过70%的用户数据，ids大小是小于等于一的，所以这样调整代码的目的是为了降低全表扫描的概率？问了一下，得到确定的答复！但是SITING_OPPORTUNITY表在数据库结构设计评审时，已经加上了UNIQUE索引，为什么还需要这样写代码呢？难道MySQL强大的SQL优化器不能自动来完成？非得要在应用层来做？</p>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line">目标表SITING_OPPORTUNITY结构示意：</div><div class="line"></div><div class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> <span class="string">`siting_opportunity`</span> (</div><div class="line"> <span class="string">`id`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> AUTO_INCREMENT <span class="keyword">COMMENT</span> <span class="string">'ID'</span>,</div><div class="line"> <span class="string">`opportunity_id`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'商机ID'</span>,</div><div class="line"> <span class="string">`industry_id`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'行业id'</span>,</div><div class="line"> <span class="string">`min_area`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最小面积'</span>,</div><div class="line"> <span class="string">`max_area`</span> <span class="built_in">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最大面积'</span>,</div><div class="line"> <span class="string">`min_rent`</span> <span class="built_in">bigint</span>(<span class="number">15</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最小租金'</span>,</div><div class="line"> <span class="string">`max_rent`</span> <span class="built_in">bigint</span>(<span class="number">15</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'最大租金'</span>,</div><div class="line"> <span class="string">`is_deal`</span> tinyint(<span class="number">1</span>) <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> <span class="string">'0'</span> <span class="keyword">COMMENT</span> <span class="string">'是否成交。0=未成交；1=已成交'</span>,</div><div class="line"> <span class="string">`slogan`</span> <span class="built_in">varchar</span>(<span class="number">20</span>) <span class="keyword">COLLATE</span> utf8mb4_unicode_ci <span class="keyword">DEFAULT</span> <span class="literal">NULL</span> <span class="keyword">COMMENT</span> <span class="string">'广告语'</span>,</div><div class="line"> PRIMARY <span class="keyword">KEY</span> (<span class="string">`id`</span>),</div><div class="line"> <span class="keyword">UNIQUE</span> <span class="keyword">KEY</span> <span class="string">`opportunity_id`</span> (<span class="string">`opportunity_id`</span>),</div><div class="line"> <span class="keyword">KEY</span> <span class="string">`fk_sitingOpportunity_industry`</span> (<span class="string">`industry_id`</span>),</div><div class="line"> <span class="keyword">CONSTRAINT</span> <span class="string">`fk_sitingOpportunity_industry`</span> FOREIGN <span class="keyword">KEY</span> (<span class="string">`industry_id`</span>) <span class="keyword">REFERENCES</span> <span class="string">`industry`</span> (<span class="string">`code`</span>) <span class="keyword">ON</span> <span class="keyword">UPDATE</span> <span class="keyword">NO</span> <span class="keyword">ACTION</span></div><div class="line">) <span class="keyword">ENGINE</span>=<span class="keyword">InnoDB</span> <span class="keyword">DEFAULT</span> <span class="keyword">CHARSET</span>=utf8mb4 <span class="keyword">COLLATE</span>=utf8mb4_unicode_ci <span class="keyword">COMMENT</span>=<span class="string">'选址商机表'</span>;</div></pre></td></tr></table></figure>
<h1 id="解谜"><a href="#解谜" class="headerlink" title="解谜"></a>解谜</h1><p>我们绝大多数人，对in操作是否会带来全表扫描开销，还停留在MySQL很古老的版本认识上。网上一篇<a href="http://itindex.net/detail/55421-mysql-sql-%E8%AF%AD%E5%8F%A5" target="_blank" rel="external">30条SQL优化军规</a>流传甚广，大多数人奉此为神道。</p>
<p>其中有一条这样描述：“5.in 和 not in 也要慎用，否则会导致全表扫描”，但没说为什么这样认为，以及面向的MySQL版本和配置也没有说明。</p>
<p>这句话在Google上查询，至少在2004年7月就已经发布到网上了，起始发布的站点出自<a href="http://hovertree.com/h/bjaf/u935eb54.htm" target="_blank" rel="external">《SQL语句优化原则_Sql Server_何问起》</a>，整整12年过去了，还有很多圈内知名IT站点在发布同样的文章。</p>
<ul>
<li>segmentfault：<a href="https://segmentfault.com/a/1190000005008401" target="_blank" rel="external">《mysql语句优化建议-2016年4月26日发布》</a></li>
</ul>
<ul>
<li>红黑联盟：<a href="http://www.2cto.com/database/201606/514022.html" target="_blank" rel="external">《数据库查询优化方法总结-2016年6月1日发布》</a></li>
</ul>
<p>首先，起始发布的文章是针对SQL server数据库的47条优化建议，已经太过久远，这些优化技巧是否还完全适用现今的数据库呢？更别提是否还适用于MySQL数据库了。我猜想是哪位“DBA大牛”应付交差，东抄一段西抄一段放上网，哪想到会有人当了真，更惨的是还完全信了。还真是：尽信书，不如无书。干我们这行，获取信息很容易，但甄别信息很难，希望大家引以为戒。</p>
<h1 id="求是"><a href="#求是" class="headerlink" title="求是"></a>求是</h1><p>怀抱求是精神，我们一起来实践，看看MySQL对于IN操作符的处理。</p>
<p>下文中用的MySQL版本：5.7.13 windows x64 解压安装版.</p>
<figure class="highlight makefile"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div></pre></td><td class="code"><pre><div class="line"><span class="section">MySQL配置文件:</span></div><div class="line"></div><div class="line">[client]</div><div class="line">default-character-set=utf8</div><div class="line">[mysql]</div><div class="line">default-character-set=utf8</div><div class="line">[mysqld]</div><div class="line"><span class="comment">########basic settings########</span></div><div class="line"><span class="comment">#skip-grant-tables,默认注释掉。重置root密码时用得上：http://www.apelearn.com/bbs/thread-9205-1-1.html</span></div><div class="line"><span class="comment">#skip-grant-tables</span></div><div class="line"></div><div class="line">skip-ssl</div><div class="line">secure-file-priv = NULL</div><div class="line"><span class="comment">#服务器唯一ID，默认是1，一般取IP最后一段</span></div><div class="line">log-bin=mysql-bin</div><div class="line">server-id = 100</div><div class="line">port = 3306</div><div class="line">user = mysql</div><div class="line">bind_address = 127.0.0.1</div><div class="line">autocommit = 0</div><div class="line"><span class="comment">#character_set_server=utf8mb4</span></div><div class="line">character_set_server=utf8</div><div class="line">skip_name_resolve = 1</div><div class="line">max_connections = 8000</div><div class="line">max_connect_errors = 10000</div><div class="line">basedir =<span class="string">"F:\mysql/"</span></div><div class="line">datadir =<span class="string">"F:\mysql/data/"</span></div><div class="line">tmpdir =<span class="string">"F:\mysql/temp/"</span></div><div class="line">socket =<span class="string">"F:\mysql/data/mysql.sock"</span></div><div class="line"><span class="section">pid-file="F:\mysql/data/current.pid"</span></div><div class="line">transaction_isolation = READ-COMMITTED</div><div class="line">explicit_defaults_for_timestamp = 1</div><div class="line">join_buffer_size = 134217728</div><div class="line">tmp_table_size = 67108864</div><div class="line">max_allowed_packet = 128MB</div><div class="line">sql_mode = <span class="string">"STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER"</span></div><div class="line">interactive_timeout = 1800</div><div class="line">wait_timeout = 1800</div><div class="line">read_buffer_size = 16777216</div><div class="line">read_rnd_buffer_size = 33554432</div><div class="line">sort_buffer_size = 33554432</div><div class="line"><span class="comment">########log settings########</span></div><div class="line"><span class="section">log-error="F:\mysql/log/error/error.log"</span></div><div class="line">slow_query_log = ON</div><div class="line">slow_query_log_file = <span class="string">"F:\mysql/log/slow/slow.log"</span></div><div class="line">log_queries_not_using_indexes = 1</div><div class="line">log_slow_admin_statements = 1</div><div class="line">log_slow_slave_statements = 1</div><div class="line">log_throttle_queries_not_using_indexes = 10</div><div class="line"><span class="comment">#保留7天的日志</span></div><div class="line">expire_logs_days = 7</div><div class="line"><span class="comment">#记录执行时间超过5秒的慢查询</span></div><div class="line">long_query_time = 5</div><div class="line">min_examined_row_limit = 100</div><div class="line"><span class="comment">########replication settings########</span></div><div class="line">master_info_repository = TABLE</div><div class="line">relay_log_info_repository = TABLE</div><div class="line">log_bin = bin.log</div><div class="line">sync_binlog = 1</div><div class="line">gtid_mode = on</div><div class="line">enforce_gtid_consistency = 1</div><div class="line">log_slave_updates</div><div class="line">binlog_format = row</div><div class="line">relay_log = relay.log</div><div class="line">relay_log_recovery = 1</div><div class="line">binlog_gtid_simple_recovery = 1</div><div class="line">slave_skip_errors = ddl_exist_errors</div><div class="line"><span class="comment">########innodb settings########</span></div><div class="line">innodb_page_size = 16384</div><div class="line"><span class="comment">#innodb_buffer_pool_size = 6G</span></div><div class="line">innodb_buffer_pool_instances = 8</div><div class="line">innodb_buffer_pool_load_at_startup = 1</div><div class="line">innodb_buffer_pool_dump_at_shutdown = 1</div><div class="line">innodb_lru_scan_depth = 2000</div><div class="line">innodb_lock_wait_timeout = 5</div><div class="line">innodb_io_capacity = 4000</div><div class="line">innodb_io_capacity_max = 8000</div><div class="line"><span class="comment">#innodb_flush_method = O_DIRECT</span></div><div class="line">innodb_flush_method=normal</div><div class="line"><span class="comment">#innodb_file_format = Barracuda</span></div><div class="line"><span class="comment">#innodb_file_format_max = Barracuda</span></div><div class="line">innodb_log_group_home_dir = <span class="string">"F:\mysql/log/redolog\"</span></div><div class="line"><span class="string">innodb_undo_directory = "</span>F:\mysql/log/undolog/<span class="string">"</span></div><div class="line"><span class="string">innodb_undo_logs = 128</span></div><div class="line"><span class="string">innodb_undo_tablespaces = 3</span></div><div class="line"><span class="string">innodb_flush_neighbors = 0</span></div><div class="line"><span class="string">#innodb_log_file_size = 4G</span></div><div class="line"><span class="string">innodb_log_file_size = 256MB</span></div><div class="line"><span class="string">innodb_log_buffer_size = 16777216</span></div><div class="line"><span class="string">innodb_purge_threads = 4</span></div><div class="line"><span class="string">innodb_large_prefix = 1</span></div><div class="line"><span class="string">innodb_thread_concurrency = 64</span></div><div class="line"><span class="string">innodb_print_all_deadlocks = 1</span></div><div class="line"><span class="string">innodb_strict_mode = 1</span></div><div class="line"><span class="string">innodb_sort_buffer_size = 67108864</span></div><div class="line"><span class="string">innodb_flush_log_at_trx_commit = 2</span></div><div class="line"><span class="string">########semi sync replication settings########</span></div><div class="line"><span class="string">plugin_dir="</span>F:\mysql/lib/plugin<span class="string">"</span></div><div class="line"><span class="string">plugin_load = "</span>rpl_semi_sync_master=semisync_master.dll;rpl_semi_sync_slave=semisync_slave.dll<span class="string">"</span></div><div class="line"><span class="string">loose_rpl_semi_sync_master_enabled = 1</span></div><div class="line"><span class="string">loose_rpl_semi_sync_slave_enabled = 1</span></div><div class="line"><span class="string">loose_rpl_semi_sync_master_timeout = 5000</span></div><div class="line"><span class="string"></span></div><div class="line"><span class="string">[mysqld-5.7]</span></div><div class="line"><span class="string">innodb_buffer_pool_dump_pct = 40</span></div><div class="line"><span class="string">innodb_page_cleaners = 4</span></div><div class="line"><span class="string">innodb_undo_log_truncate = 1</span></div><div class="line"><span class="string">innodb_max_undo_log_size = 2G</span></div><div class="line"><span class="string">innodb_purge_rseg_truncate_frequency = 128</span></div><div class="line"><span class="string">binlog_gtid_simple_recovery=1</span></div><div class="line"><span class="string">log_timestamps=system</span></div><div class="line"><span class="string">#transaction_write_set_extraction=MURMUR32</span></div><div class="line"><span class="string">show_compatibility_56=on</span></div></pre></td></tr></table></figure>
<p>先用show index from siting_opportunity;语句查看一下索引情况。</p>
<p><img src="/img/showIndexFromSitingOpportunity.png" alt=""><br>由于索引opportunity_id（key_name）是unique类型，可以看到其散列程度（Cardinality）是非常高的，因此该索引是很有效的。相应的该表的记录数是：476772行。</p>
<h2 id="无索引"><a href="#无索引" class="headerlink" title="无索引"></a>无索引</h2><p>字段min_area没有索引，<code>min_area</code> int(11) NOT NULL DEFAULT ‘0’ COMMENT ‘最小面积’。<br><img src="/img/withoutIndex.png" alt=""></p>
<p>很显然，优化器告诉我们，需要进行476772行扫描才能完成计算，也就是全表扫描了。</p>
<h2 id="Unique索引"><a href="#Unique索引" class="headerlink" title="Unique索引"></a>Unique索引</h2><p>先看在表SITING_OPPORTUNITY，OPPORTUNITY_ID字段的处理情况。<br><img src="/img/userUniqueIndex.png" alt=""><br>明显的可以看到，MySQL优化器是选择了索引，没有进行全表扫描，而且rows也为7。</p>
<h2 id="主键索引"><a href="#主键索引" class="headerlink" title="主键索引"></a>主键索引</h2><p>我们知道主键一定是唯一性索引,那情况应该跟上一节差不多？动动手吧！<br><img src="/img/userPrimaryIndex.png" alt=""><br>情况确实跟Unique索引差不多，pass！</p>
<h2 id="外键索引"><a href="#外键索引" class="headerlink" title="外键索引"></a>外键索引</h2><p>在设置外键时，数据库会同时为这个字段设置一个普通索引，所以外键的设置也应该遵循索引的设置策略。<br><img src="/img/useFKIndex.png" alt=""><br>使用外键字段时，使用in操作符，也使用了索引。</p>
<h2 id="普通索引"><a href="#普通索引" class="headerlink" title="普通索引"></a>普通索引</h2><p>我们换一个有普通索引，但该字段不是外键的表来验证。<br><img src="/img/userNormalIndex.png" alt=""></p>
<h1 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h1><p>1.在in操作中，集合体积过大时，索引还会生效吗？</p>
<p>2.在嵌套查询中，内层查询使用in，是否会有效率问题？</p>
<h1 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h1><p>本文中的存储引擎都是innodb，因时间有限，没有对Myisam等其他存储引擎进行类似实验，希望动手能力强的你可以完成！</p>
<p>MySQL在2010年发布5.5版本中，优化器对in操作符可以自动完成优化，针对建立了索引的列可以使用索引，没有索引的列还是会走全表扫描。因此应用层不需要再进行画蛇添足啦，当然优化的精神是可嘉的，但在猜测中进行的优化可能与实际背离甚远！</p>
<p>互联网上的信息太过广泛，但这不应该成为我们掉以轻心的借口，抱着存疑求是的精神进行甄别，任何信息只有在我们实践验证后方可全信，用在生产开发上的知识，不容有失！</p>
]]></content>
<summary type="html">
起始发布的文章是针对SQL server数据库的47条优化建议，已经太过久远，这些优化技巧是否还完全适用现今的数据库呢？更别提是否还适用于MySQL数据库了。我猜想是哪位“DBA大牛”应付交差，东抄一段西抄一段放上网，哪想到会有人当了真，更惨的是还完全信了。还真是：尽信书，不如无书。干我们这行，获取信息很容易，但甄别信息很难，希望大家引以为戒。
</summary>
<category term="CodeReview" scheme="https://amao12580.github.io/categories/CodeReview/"/>
<category term="SQL" scheme="https://amao12580.github.io/tags/SQL/"/>
<category term="MySQL" scheme="https://amao12580.github.io/tags/MySQL/"/>
</entry>
</feed>