-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
241 lines (111 loc) · 189 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>Mongodb/搭建mongodb分片复制集</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Mongodb/%E6%90%AD%E5%BB%BAmongodb%E5%88%86%E7%89%87%E5%A4%8D%E5%88%B6%E9%9B%86/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Mongodb/%E6%90%AD%E5%BB%BAmongodb%E5%88%86%E7%89%87%E5%A4%8D%E5%88%B6%E9%9B%86/</url>
<content type="html"><![CDATA[<pre><code>搭建mongodb分片复制集</code></pre><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">解压缩</span><br></pre></td></tr></table></figure><ul><li><p>添加复制集配置文件 (yidian_repl1) : mongo.conf</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">fork=false</span><br><span class="line"><span class="meta">#</span><span class="bash">数据路径</span></span><br><span class="line">dbpath=/data/db</span><br><span class="line">port=21001</span><br><span class="line">bind_ip=[111.230.248.15,106.52.107.31]</span><br><span class="line"><span class="meta">#</span><span class="bash"> 日志路径</span></span><br><span class="line"><span class="meta">#</span><span class="bash">logpath=/usr.local/mongodb_1/logs/mongodb/<span class="built_in">log</span></span></span><br><span class="line">logappend=true</span><br><span class="line">replSet=cresign_RS_1</span><br><span class="line"><span class="meta">#</span><span class="bash">smallfiles=<span class="literal">true</span></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 分片集群必须要有的属性</span></span><br><span class="line">shardsvr=true</span><br><span class="line">directoryperdb=true </span><br><span class="line"><span class="meta">#</span><span class="bash">keyFile=/keyfile</span></span><br></pre></td></tr></table></figure></li><li><p>添加复制集配置文件 (yidian_repl2) : mongo.conf</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">fork=false</span><br><span class="line"><span class="meta">#</span><span class="bash">数据路径</span></span><br><span class="line">dbpath=/data/db</span><br><span class="line">port=27018</span><br><span class="line">bind_ip=0.0.0.0</span><br><span class="line"><span class="meta">#</span><span class="bash"> 日志路径</span></span><br><span class="line">logpath=/usr.local/mongodb_2/logs/mongodb/log</span><br><span class="line">logappend=true</span><br><span class="line">replSet=yidian_rep2</span><br><span class="line">smallfiles=true</span><br><span class="line"><span class="meta">#</span><span class="bash"> 分片集群必须要有的属性</span></span><br><span class="line">shardsvr=true</span><br><span class="line">directoryperdb=true # 每个数据库一个文件夹存放</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>启动docker mongodb _1</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name mongodb_1_1 --privileged -p 21002:21002 -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_1/data:/data/db -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_1/conf:/data/configdb -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_1/logs:/data/log/ mongo:4.2.3 -f /data/configdb/mongodb.conf</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name mongodb_1_2 --privileged -p 21002:21002 -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_2/data:/data/db -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_2/conf:/data/configdb -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_2/logs:/data/log/ mongo:4.2.3 -f /data/configdb/mongodb.conf</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name mongodb_1_arbiter --privileged -p 21000:21000 -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_arbiter/data:/data/db -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_arbiter/conf:/data/configdb -v /home/dockerConf/mongodb/mongodb_rs/mongodb_1_arbiter/logs:/data/log/ mongo:4.2.3 -f /data/configdb/mongodb.conf</span><br></pre></td></tr></table></figure></li></ul> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name mongodb_2 --privileged -p 27018:27018 -v /usr/local/mongodb_2/data:/data/db -v /usr/local/mongodb_2/conf:/data/configdb -v /usr/local/mongodb_2/logs:/data/log/ mongo:4.2.3 -f /data/configdb/mongodb.conf</span><br></pre></td></tr></table></figure><ul><li><p>进入客户端后，执行初始化命令 (yidian_repl1)</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">var rsconf = {</span><br><span class="line">"_id": "cresign_RS_1",// 这里_id要与配置文件中指定的服务所属的复制集相同</span><br><span class="line">"members": [{// 复制集成员</span><br><span class="line">"_id": 1,</span><br><span class="line">"host": "10.0.2.142:21001"</span><br><span class="line">},</span><br><span class="line">{</span><br><span class="line">"_id": 2,</span><br><span class="line">"host": "10.0.2.142:21002"</span><br><span class="line">},</span><br><span class="line">{</span><br><span class="line">"_id": 3,</span><br><span class="line">"host": "10.0.2.142:21000",</span><br><span class="line">"arbiterOnly":true</span><br><span class="line">}</span><br><span class="line">]</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># 执行初始化命令</span><br><span class="line">rs.initiate(rsconf);</span><br></pre></td></tr></table></figure></li><li><p>查看状态</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">rs.status();</span><br></pre></td></tr></table></figure></li><li><p>进入客户端后，执行初始化命令 (yidian_repl2)</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">var rsconf = {</span><br><span class="line">"_id": "cresign_RS_2",// 这里_id要与配置文件中指定的服务所属的复制集相同</span><br><span class="line">"members": [{// 复制集成员</span><br><span class="line">"_id": 1,</span><br><span class="line">"host": "10.0.0.14:22001"</span><br><span class="line">},</span><br><span class="line">{</span><br><span class="line">"_id": 2,</span><br><span class="line">"host": "10.0.0.14:22002"</span><br><span class="line">},</span><br><span class="line">{</span><br><span class="line">"_id": 3,</span><br><span class="line">"host": "10.0.0.14:22000",</span><br><span class="line">"arbiterOnly":true</span><br><span class="line">}</span><br><span class="line">]</span><br><span class="line">}</span><br></pre></td></tr></table></figure></li><li><p>执行命令</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">rs.initiate(rsconf);</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>进入主实例,创建管理员用户</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">use admin</span><br><span class="line"></span><br><span class="line">db.createUser({</span><br><span class="line"> user:'7F6E37DC-D554-4008-A895-3F14384ADBB3',</span><br><span class="line"> pwd:'321AE5DD-2496-440A-8E33-B360BEEC55D4',</span><br><span class="line"> roles:[{role:"userAdminAnyDatabase", db:"admin"}, {role:"readWriteAnyDatabase", db:"admin"}]</span><br><span class="line">})</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">db.createUser(</span><br><span class="line"> {</span><br><span class="line"> user: "jackson",</span><br><span class="line"> pwd: "321123",</span><br><span class="line"> roles: [ { role: "readWrite", db: "D" } ]</span><br><span class="line"> }</span><br><span class="line">)</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">db.createUser(</span><br><span class="line"> {</span><br><span class="line"> user: "cluster",</span><br><span class="line"> pwd: "321123",</span><br><span class="line"> roles: [ { role: "clusterAdmin", db: "D" } ]</span><br><span class="line"> }</span><br><span class="line">)</span><br></pre></td></tr></table></figure></li></ul><ul><li><h3 id="为副本集增加权限认证"><a href="#为副本集增加权限认证" class="headerlink" title="为副本集增加权限认证"></a>为副本集增加权限认证</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 生成keyfile文件</span></span><br><span class="line">openssl rand -base64 90 > /data/mongodb1/conf/keyfile</span><br><span class="line"></span><br><span class="line">cp /data/mongodb1/conf/keyfile /data/mongodb2/conf/keyfile</span><br><span class="line">cp /data/mongodb1/conf/keyfile /data/mongodb3/conf/keyfile</span><br></pre></td></tr></table></figure><ul><li><p>另外需要注意，keyfile文件权限必须是X00，也就是说，不能给group和other成员分配任何权限，否则实例无法启动。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">chmod 600 /data/mongodb1/conf/keyfile</span><br><span class="line">chmod 600 /data/mongodb2/conf/keyfile</span><br><span class="line">chmod 600 /data/mongodb3/conf/keyfile</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">chown 999 /data/mongodb1/conf/keyfile</span><br><span class="line">chown 999 /data/mongodb2/conf/keyfile</span><br><span class="line">chown 999 /data/mongodb3/conf/keyfile</span><br></pre></td></tr></table></figure></li></ul></li></ul><ul><li><p>生成好keyfile之后，将keyfile写入mongodb.conf配置文件中，在mongodb.conf配置文件中增加如下配置：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">keyFile=/data/mongodb1/conf/keyfile</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>创建config节点配置文件：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line">systemLog:</span><br><span class="line"> destination: file</span><br><span class="line"> # 日志存储位置</span><br><span class="line"> path : /data/log/mongodb.log</span><br><span class="line"> logAppend : true</span><br><span class="line">storage:</span><br><span class="line"> journal:</span><br><span class="line"> enabled: true</span><br><span class="line"> # 数据存储位置</span><br><span class="line"> dbPath: /data/db</span><br><span class="line"> # 是否一个库一个文件夹</span><br><span class="line"> directoryPerDB: true</span><br><span class="line"> wiredTiger:</span><br><span class="line"> engineConfig:</span><br><span class="line"> # 最大使用cache</span><br><span class="line"> cacheSizeGB: 1</span><br><span class="line"> # 是否将索引也按照数据库名单独存储</span><br><span class="line"> directoryForIndexes: true</span><br><span class="line"> collectionConfig:</span><br><span class="line"> # 表压缩配置</span><br><span class="line"> blockCompressor: zlib</span><br><span class="line"> indexConfig: </span><br><span class="line"> prefixCompression: true</span><br><span class="line">net:</span><br><span class="line"> bindIp: 192.168.111.131</span><br><span class="line"> port: 28018</span><br><span class="line">replication:</span><br><span class="line"> oplogSizeMB: 2048</span><br><span class="line"> # 配置节点的复制集名字</span><br><span class="line"> replSetName: cresign_cfg_rs</span><br><span class="line">sharding:</span><br><span class="line"> clusterRole: configsvr</span><br><span class="line">processManagement:</span><br><span class="line"> fork: false</span><br><span class="line">#security:</span><br><span class="line"> # keyFile: /keyfile</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>创建容器</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name mongodb_1_cfg --privileged -p 20001:20001 -v /usr/local/mongodb_1_cfg/data:/data/db -v /usr/local/mongodb_1_cfg/conf:/data/configdb -v /usr/local/mongodb_1_cfg/logs:/data/log/ mongo:4.2.3 -f /data/configdb/mongodb.conf</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>启动配置复制集</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mongod -f /data/configdb/mongo-cfg.conf</span><br></pre></td></tr></table></figure></li><li><p>登录配置节点</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># 登录 指定IP 和端口</span><br><span class="line">mongo -host 192.168.111.131 --port 28018</span><br></pre></td></tr></table></figure></li><li><p>初始化命令</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">rs.initiate(</span><br><span class="line">{</span><br><span class="line">"_id" : "cresign_cfg_rs",</span><br><span class="line">"configsvr" : true,</span><br><span class="line">members: [</span><br><span class="line">{ _id : 1, host : "10.0.2.142:20001"},</span><br><span class="line">{ _id : 2, host : "10.0.2.142:20002"},</span><br><span class="line">{ _id : 3, host : "10.0.0.14:20003"}</span><br><span class="line">]</span><br><span class="line">}</span><br><span class="line">)</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>配置节点mongos</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">#systemLog:</span><br><span class="line"> #destination: file</span><br><span class="line"> #path: /log/mongos.log</span><br><span class="line"> #logAppend: true</span><br><span class="line">net:</span><br><span class="line"> bindIp: 0.0.0.0</span><br><span class="line"> port: 28001</span><br><span class="line">sharding:</span><br><span class="line"> configDB: cresign_cfg_rs/111.230.248.15:20001,111.230.248.15:20002,106.52.107.31:20003</span><br><span class="line">processManagement:# docker里 fork就用false</span><br><span class="line"> fork: false</span><br><span class="line">#security:</span><br><span class="line"> #keyFile: /keyfile</span><br></pre></td></tr></table></figure></li><li><p>启动</p></li><li><p>由于镜像的默认入口是 <code>mongod</code>，所以要通过 <code>--entrypoint "mongos"</code> 将其改为 <code>mongos</code>：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name mongodb_mongos --privileged --entrypoint "mongos" -p 28002:28001 -v /home/dockerConf/mongodb/mongodb_mongos/data:/data/db -v /home/dockerConf/mongodb/mongodb_mongos/conf:/data/configdb -v /home/dockerConf/mongodb/mongodb_mongos/logs:/data/log/ mongo:4.2.3 -f /data/configdb/mongodb.conf</span><br></pre></td></tr></table></figure></li><li><p>登录mongos节点</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mongo --port 28017</span><br></pre></td></tr></table></figure></li><li><p>切换到 admin 库</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">use admin</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>创建集群角色</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">db.createUser({</span><br><span class="line"> user: "jackson",</span><br><span class="line"> pwd: "321123",</span><br><span class="line"> roles: [{role: "clusterAdmin",db: "admin"}]</span><br><span class="line">})</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>添加集群中的分片节点</p><ul><li><p>添加 shard1 复制集</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">db.runCommand({</span><br><span class="line">addshard : "cresign_RS_1/10.0.2.142:21001,10.0.2.142:21002,10.0.2.142:21000",</span><br><span class="line">name : "shard1"</span><br><span class="line">})</span><br></pre></td></tr></table></figure></li><li><p>添加 shard2 复制集</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">db.runCommand({</span><br><span class="line">addshard : "cresign_RS_2/10.0.0.14:22001,10.0.0.14:22002,10.0.0.14:22000",</span><br><span class="line">name : "shard2"</span><br><span class="line">})</span><br></pre></td></tr></table></figure></li><li><p>查看分片</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># mongos</span><br><span class="line">db.runCommand({ listshards : 1 })</span><br></pre></td></tr></table></figure></li><li><p>查看分片状态</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># mongos</span><br><span class="line">sh.status();</span><br></pre></td></tr></table></figure></li><li><p>刷新路由</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">db.adminCommand("flushRouterConfig")</span><br></pre></td></tr></table></figure></li></ul></li></ul><ul><li><p>测试分片集群</p><ul><li><p>开启数据库分片配置</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">db.runCommand({ enablesharding : "D" })</span><br></pre></td></tr></table></figure></li></ul></li></ul><ul><li><p>创建索引（如果不是空集合，不是第一次操作）</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">use testdb</span><br><span class="line">db.users.ensureIndex({</span><br><span class="line">id : 1</span><br><span class="line">})</span><br></pre></td></tr></table></figure></li><li><p>创建分片的键(id)</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">db.runCommand({</span><br><span class="line">shardcollection : "D.lSProd", </span><br><span class="line">key : {</span><br><span class="line">_id : 1</span><br><span class="line">}</span><br><span class="line">})</span><br></pre></td></tr></table></figure></li><li><p>添加测试数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">var arr=[];</span><br><span class="line">for(var i=0;i<1000;i++){</span><br><span class="line">var uid = i;</span><br><span class="line">var name = "name" + i;</span><br><span class="line">arr.push({"id" : uid, "name" : name,"age":0});</span><br><span class="line">}</span><br><span class="line">db.Users.insertMany(arr);</span><br></pre></td></tr></table></figure></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">服务器mongodb 用户密码：jackson 147258369</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>kubernetes/安装</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/kubernetes/%E5%AE%89%E8%A3%85/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/kubernetes/%E5%AE%89%E8%A3%85/</url>
<content type="html"><![CDATA[<p>安装</p><h1 id="K8s-安装"><a href="#K8s-安装" class="headerlink" title="K8s 安装"></a>K8s 安装</h1><h2 id="准备虚拟机"><a href="#准备虚拟机" class="headerlink" title="准备虚拟机"></a>准备虚拟机</h2><h3 id="调整操作系统"><a href="#调整操作系统" class="headerlink" title="调整操作系统"></a>调整操作系统</h3><h4 id="调整yum源"><a href="#调整yum源" class="headerlink" title="调整yum源"></a>调整yum源</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install epel-release</span><br></pre></td></tr></table></figure><h4 id="安装epel-release"><a href="#安装epel-release" class="headerlink" title="安装epel-release"></a>安装epel-release</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install epel-release</span><br></pre></td></tr></table></figure><h4 id="关闭SElinux-和-firewalld"><a href="#关闭SElinux-和-firewalld" class="headerlink" title="关闭SElinux 和 firewalld"></a>关闭SElinux 和 firewalld</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/selinux/config </span><br><span class="line">#修改SELINUX=disabled</span><br><span class="line">SELINUX=disabled</span><br><span class="line">#保存退出 重启后永久生效</span><br><span class="line"></span><br><span class="line">setenforce 0 #当前生效，重启电脑变回原来的</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">systemctl stop firewalld # 关闭防火墙</span><br></pre></td></tr></table></figure><h4 id="安装必要工具"><a href="#安装必要工具" class="headerlink" title="安装必要工具"></a>安装必要工具</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils -y</span><br></pre></td></tr></table></figure><h4 id="安装-bind9"><a href="#安装-bind9" class="headerlink" title="安装 bind9"></a>安装 bind9</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#安装bind9</span><br><span class="line">yum install bind -y</span><br></pre></td></tr></table></figure><p>修改文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line">//</span><br><span class="line">// named.conf</span><br><span class="line">//</span><br><span class="line">// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS</span><br><span class="line">// server as a caching only nameserver (as a localhost DNS resolver only).</span><br><span class="line">//</span><br><span class="line">// See /usr/share/doc/bind*/sample/ for example named configuration files.</span><br><span class="line">//</span><br><span class="line">// See the BIND Administrator's Reference Manual (ARM) for details about the</span><br><span class="line">// configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html</span><br><span class="line"></span><br><span class="line">options {</span><br><span class="line">listen-on port 53 { 10.4.7.11; };</span><br><span class="line">listen-on-v6 port 53 { ::1; };</span><br><span class="line">directory "/var/named";</span><br><span class="line">dump-file "/var/named/data/cache_dump.db";</span><br><span class="line">statistics-file "/var/named/data/named_stats.txt";</span><br><span class="line">memstatistics-file "/var/named/data/named_mem_stats.txt";</span><br><span class="line">recursing-file "/var/named/data/named.recursing";</span><br><span class="line">secroots-file "/var/named/data/named.secroots";</span><br><span class="line">allow-query { any; };</span><br><span class="line"> forwarders { 10.4.7.254; };</span><br><span class="line"></span><br><span class="line">/* </span><br><span class="line"> - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.</span><br><span class="line"> - If you are building a RECURSIVE (caching) DNS server, you need to enable </span><br><span class="line"> recursion. </span><br><span class="line"> - If your recursive DNS server has a public IP address, you MUST enable access </span><br><span class="line"> control to limit queries to your legitimate users. Failing to do so will</span><br><span class="line"> cause your server to become part of large scale DNS amplification </span><br><span class="line"> attacks. Implementing BCP38 within your network would greatly</span><br><span class="line"> reduce such attack surface </span><br><span class="line">*/</span><br><span class="line">recursion yes;</span><br><span class="line"></span><br><span class="line">dnssec-enable no;</span><br><span class="line">dnssec-validation no;</span><br><span class="line"></span><br><span class="line">/* Path to ISC DLV key */</span><br><span class="line">bindkeys-file "/etc/named.root.key";</span><br><span class="line"></span><br><span class="line">managed-keys-directory "/var/named/dynamic";</span><br><span class="line"></span><br><span class="line">pid-file "/run/named/named.pid";</span><br><span class="line">session-keyfile "/run/named/session.key";</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line">logging {</span><br><span class="line"> channel default_debug {</span><br><span class="line"> file "data/named.run";</span><br><span class="line"> severity dynamic;</span><br><span class="line"> };</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line">zone "." IN {</span><br><span class="line">type hint;</span><br><span class="line">file "named.ca";</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line">include "/etc/named.rfc1912.zones";</span><br><span class="line">include "/etc/named.root.key";</span><br></pre></td></tr></table></figure><p>检查是否正确配置</p><p><code>named-checkconf</code></p><p>修改文件 /etc/named.rfc1912.zones，在尾部添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">zone "host.com" IN {</span><br><span class="line"> type master;</span><br><span class="line"> file "host.com.zone";</span><br><span class="line"> allow-update { 10.4.7.11; };</span><br><span class="line">}; </span><br><span class="line"></span><br><span class="line">zone "od.com" IN {</span><br><span class="line"> type master;</span><br><span class="line"> file "od.com.zone";</span><br><span class="line"> allow-update { 10.4.7.11; };</span><br><span class="line">};</span><br></pre></td></tr></table></figure><p>修改文件 /var/named/host.com.zone</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">$ORIGIN host.com.</span><br><span class="line">$TTL 600 ; 10 minutes</span><br><span class="line">@ IN SOA dns.host.com. dnsadmin.host.com. (</span><br><span class="line"> 2020021701 ; serial</span><br><span class="line"> 10800 ; refresh (3 hours)</span><br><span class="line"> 900 ; retry (15 minutes)</span><br><span class="line"> 604800 ; expire (1 week)</span><br><span class="line"> 86400 ; minimum (1 day)</span><br><span class="line"> )</span><br><span class="line"> NS dns.host.com.</span><br><span class="line">$TTL 60 ; 1 minute</span><br><span class="line">dns A 10.4.7.11</span><br><span class="line">HDSS7-11 A 10.4.7.11</span><br><span class="line">HDSS7-12 A 10.4.7.12</span><br><span class="line">HDSS7-21 A 10.4.7.21</span><br><span class="line">HDSS7-22 A 10.4.7.22</span><br><span class="line">HDSS7-200 A 10.4.7.200</span><br></pre></td></tr></table></figure><p>修改文件 /var/named/od.com.zone</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">$ORIGIN od.com.</span><br><span class="line">$TTL 600 ; 10 minutes</span><br><span class="line">@ IN SOA dns.od.com. dnsadmin.od.com. (</span><br><span class="line"> 2020021701 ; serial</span><br><span class="line"> 10800 ; refresh (3 hours)</span><br><span class="line"> 900 ; retry (15 minutes)</span><br><span class="line"> 604800 ; expire (1 week)</span><br><span class="line"> 86400 ; minimum (1 day)</span><br><span class="line"> )</span><br><span class="line"> NS dns.od.com.</span><br><span class="line">$TTL 60 ; 1 minute</span><br><span class="line">dns A 10.4.7.11</span><br></pre></td></tr></table></figure><p>启动 服务</p><p><code>systemctl start named</code></p><p>查看否启动了</p><p><code>netstat -luntp|grep 53</code></p><p>查看是否解析成功</p><p><code>dig -t A hdss7-21.host.com @10.4.7.11 +short</code></p><p>出现 10.4.7.21 即是成功</p><p>修改 文件 /etc/sysconfig/network-scripts/ifcfg-ens33</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># 将DNS1改成</span><br><span class="line">DNS1=10.4.7.11</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>kubernetes/Kubernetes学习</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/kubernetes/Kubernetes%E5%AD%A6%E4%B9%A0/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/kubernetes/Kubernetes%E5%AD%A6%E4%B9%A0/</url>
<content type="html"><![CDATA[<p>Kubernetes学习</p><h1 id="Kubernetes学习"><a href="#Kubernetes学习" class="headerlink" title="Kubernetes学习"></a>Kubernetes学习</h1><h2 id="Kubernetes概述"><a href="#Kubernetes概述" class="headerlink" title="Kubernetes概述"></a>Kubernetes概述</h2><ul><li>官网：<a href="https://kubernetes.io" target="_blank" rel="noopener">https://kubernetes.io</a></li><li>GitHub: <a href="https://github.com/kubernetes/kubernetes" target="_blank" rel="noopener">https://github.com/kubernetes/kubernetes</a></li><li>由来：谷歌的Borg系统，后经Go语言重写并捐献给CNCF基金会开源</li><li>含义：词根源于希腊语：舵手、飞行员，K8S</li><li>重要作用：开源的容器编排框架工具（生态极其丰富）</li></ul><h2 id="Kubernetes优点"><a href="#Kubernetes优点" class="headerlink" title="Kubernetes优点"></a>Kubernetes优点</h2><p>1、故障迁移：当某一个node节点关机或挂掉后，node节点上的服务会自动转移到另一个node节点上，这个过程所有服务不中断。这是docker或普通云主机是不能做到的</p><p>2、资源调度：当node节点上的cpu、内存不够用的时候，可以扩充node节点，新建的pod就会被kube-schedule调度到新扩充的node节点上</p><p>3、资源隔离：创建开发、运维、测试三个命名空间，切换上下文后，开发人员就只能看到开发命名空间的所有pod，看不到运维命名空间的pod，这样就不会造成影响，互不干扰</p><pre><code>传统的主机或只有docker环境中，登录进去就会看到所有的服务或者容器</code></pre><p>4、因为采用docker容器，进程之间互不影响，</p><p>5、安全：不同角色有不同的权限，查看pod、删除pod等操作；RBAC认证增加了k8s的安全</p><p>快速精准地部署应用程序</p><ul><li>限制硬件用量仅为所需资源</li></ul><p>Kubernetes 的优势</p><ul><li>可移动: 公有云、私有云、混合云、多态云</li><li>可扩展: 模块化、插件化、可挂载、可组合</li><li>自修复: 自动部署、自动重启、自动复制、自动伸缩</li></ul><p>负载均衡</p><pre><code>k8s可以更快的更新新版本，打包应用，更新的时候可以做到不用中断服务，服务器故障不用停机，从开发环境到测试环境到生产环境的迁移极其方便，一个配置文件搞定，一次生成image，到处运行。。。</code></pre><h3 id="Pod与Pod控制器"><a href="#Pod与Pod控制器" class="headerlink" title="Pod与Pod控制器"></a>Pod与Pod控制器</h3><h4 id="Pod"><a href="#Pod" class="headerlink" title="Pod"></a>Pod</h4><ul><li>Pod是K8S里能够被运行的最小的逻辑单元（原子单元）</li><li>1个Pod里面可以运行多个容器，它们共享UTS+NET+IPC名称空间</li><li>可以把Pod理解成豌豆荚，而同一个Pod内的每个容器是一颗颗豌豆</li><li>一个Pod里运行多个容器，又叫：边车（SideCar）模式</li></ul><h4 id="Pod控制器"><a href="#Pod控制器" class="headerlink" title="Pod控制器"></a>Pod控制器</h4><ul><li><p>Pod控制器是Pod启动的一种模块，用来保证在K8S里启动的Pod应始终按照人们的预期运行（副本数、生命周期、健康状态检查。。。）</p></li><li><p>K8S内提供了众多的Pod控制器，常用的有以下几种：</p><ul><li><p>Deployment</p></li><li><p>DaemonSet</p></li><li><p>ReplicaSet</p></li><li><p>StatefulSet</p></li><li><p>Job</p></li><li><p>Cronjob</p></li></ul></li></ul><h3 id="Nname与Namepace"><a href="#Nname与Namepace" class="headerlink" title="Nname与Namepace"></a>Nname与Namepace</h3><h4 id="Name"><a href="#Name" class="headerlink" title="Name"></a>Name</h4><ul><li>由于K8S内部，使用“资源”来定义每一种逻辑概念（功能）故每种“资源”，都应该有自己的“名称”</li><li>“资源”有api版本（apiVersion）类别（kind）、元数据（metadata）、定义清单（spec）、状态（status）等配置信息</li><li>“名称”通常定义在“资源”的“元数据”信息里</li></ul><h4 id="Namespace"><a href="#Namespace" class="headerlink" title="Namespace"></a>Namespace</h4><ul><li>随着项目增多、人员增加、集群规模的扩大，需要一种能够隔离KS内各种“资源”的方法，这就是名称空间</li><li>不同名称空间内的“资源”，名称可以相同，相同名称空间内的同种“资源”，“名称”不能相同</li><li>合理的使用K8S的名称空间，使得集群管理员能够更好的对交付到K8S里的服务进行分类管理和浏览</li><li>K8S里默认存在的名称空间有：default、kube-system、kub-public</li><li>查询K8S里特定“资源”要带上相应的名称空间</li></ul><h3 id="Label与Label选择器"><a href="#Label与Label选择器" class="headerlink" title="Label与Label选择器"></a>Label与Label选择器</h3><h4 id="Label"><a href="#Label" class="headerlink" title="Label"></a>Label</h4><ul><li>标签是K8S特色的管理方式，便于分类管理资源对象</li><li>一个标签可以对应多个资源，一个资源也可以有多个标签，它们是多对多的关系</li><li>一个资源拥有多个标签，可以实现不同维度的管理</li><li>标签的组成：key=value</li><li>与标签类似的，还有一种“注解”（annotations)</li></ul><h4 id="Label选择器"><a href="#Label选择器" class="headerlink" title="Label选择器"></a>Label选择器</h4><ul><li><p>给资源打上标签后，可以使用标签选择器过滤指定的标签</p></li><li><p>标签选择器目前有两个：基于等值关系（等于、不等于）和基于集合关系（属于、不属于、存在）</p></li><li><p>许多资源支持内嵌标签选择器字段</p><ul><li><p>matchLabels</p></li><li><p>matchExpressions</p></li></ul></li></ul><h3 id="Service与Ingress"><a href="#Service与Ingress" class="headerlink" title="Service与Ingress"></a>Service与Ingress</h3><h4 id="Service"><a href="#Service" class="headerlink" title="Service"></a>Service</h4><ul><li>在K8S的世界里，虽然每个Pod都会被分配一个单独的IP地址，但这个IP地址会随着Pod的销毁而消失</li><li>Service（服务）就是用来解决这个问题的核心理念</li><li>一个Service可以看做一组相同服务的Pod的对外访问接口</li><li>Service作用于哪些Pod是天天过标签选择器来定义的</li></ul><h4 id="Ingress"><a href="#Ingress" class="headerlink" title="Ingress"></a>Ingress</h4><ul><li>Ingress是K8S集群里工作在OSI网络参考模型下，第7层的应用，对外暴露的接口</li><li>Service只能进行L4流量调度，表现形式ip+port</li><li>Ingress则可以调度不同业务域、不同URL访问路径的业务流量</li></ul><h3 id="K8S的三条网络详解"><a href="#K8S的三条网络详解" class="headerlink" title="K8S的三条网络详解"></a>K8S的三条网络详解</h3><ul><li><p>核心组件</p><ul><li><p>配置存储中心->etcd服务</p></li><li><p>主控（master）节点</p><ul><li><p>kube-apiserver服务</p></li><li><p>kube-controller-manager服务</p></li><li><p>kube-scheduler服务</p></li></ul></li><li><p>运算（node）节点</p><ul><li><p>kube-kubelet服务</p></li><li><p>kube-proxy服务</p></li></ul></li></ul></li><li><p>Cli客户端</p><ul><li>kubectl</li></ul></li><li><p>核心附件</p><ul><li><p>CNI网络插件->flannel/calico</p></li><li><p>服务发现用插件->coredns</p></li><li><p>服务暴露用插件->traefik</p></li><li><p>GUI管理插件->Dashboard</p></li></ul></li></ul><p>下面对核心组件讲解</p><ul><li><p>apiserver</p><ul><li><p>提供了集群管理的REST API接口（包括鉴权、数据校验及集群状态变更）</p></li><li><p>负责其他模块之间的数据交互，承担通信枢纽功能</p></li><li><p>是资源配额控制的入口</p></li><li><p>提供完备的集群安全机制</p></li></ul></li><li><p>controller-manager</p><ul><li><p>由一系列控制器组成，通过apiserver监控整个集群的状态，并确保集群处于预期的工作状态</p></li><li><p>Node Controller</p></li><li><p>Deployment Controller</p></li><li><p>Service Controller</p></li><li><p>Volume Controller</p></li><li><p>Endpoint Controller</p></li><li><p>Garbage Controller</p></li><li><p>Namespace Controller</p></li><li><p>Job Controller</p></li><li><p>Resource quta Controller</p></li></ul></li><li><p>scheduler</p><ul><li><p>主要功能是接受调度pod到适合的运算节点上</p></li><li><p>预算策略（predict）</p></li><li><p>优选策略（priorities）</p></li></ul></li><li><p>kubelet</p><ul><li><p>简单地说，kubelet的主要功能就是定时从某个地方获取节点上pod的期望状态（运行什么容器、运行的副本数量、网络或者存储如何配置等等），并调用对应的容器平台接口达到这 状态</p></li><li><p>定时汇报当前节点的状态给apiserver，以供调度的时候使用</p></li><li><p>镜像和容器的清理工作，保证节点上镜像不会占满磁盘空间，退出容器不会占用太多资源</p></li></ul></li><li><p>kube-proxy</p><ul><li><p>是K8S在每个节点上运行网络代理，service资源的载体</p></li><li><p>建立了pod网络和集群网络的关系（clusterip->podip）</p></li><li><p>常用三种流量调度模式</p><ul><li><p>Userspace（废弃）</p></li><li><p>Iptables（濒临废弃）</p></li><li><p>Ipvs（推荐）</p></li></ul></li><li><p>负责建立和删除报货更新调度规则、通知apiserver自己的更新、或者从apiserver哪里获取其他kube-proxy的调度规则变化来更新自己的</p></li></ul></li></ul><p><img src="image.png" alt=""></p><h3 id="架构图"><a href="#架构图" class="headerlink" title="架构图"></a>架构图</h3><p><img src="image.png" alt=""></p>]]></content>
</entry>
<entry>
<title>K8s/K8s</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/K8s/K8s/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/K8s/K8s/</url>
<content type="html"><![CDATA[<h1 id="K8s"><a href="#K8s" class="headerlink" title="K8s"></a>K8s</h1><h2 id="概念："><a href="#概念：" class="headerlink" title="概念："></a>概念：</h2><ul><li><p>什么是k8s</p><ul><li>k8s是一组服务器集群</li><li>k8s所管理的集群节点上的容器</li></ul></li><li><p>k8s的功能</p><ul><li>自我修复</li><li>弹性伸缩：实时根据服务器的并发情况，增加或缩减容器数量</li><li>自动部署</li><li>回滚</li><li>服务发现和负载均衡</li><li>机密和配置共享管理</li></ul></li><li><p>k8s集群分为两类节点</p><ul><li>master node：主节点</li><li>worker node：工作节点</li></ul></li><li><p>master节点的组件（程序）</p><ul><li>apiserver: 接受客户端操作k8s的指令</li><li>schduler：从多个worker node节点的组件中选举一个来启动服务</li><li>controller manager：向worker节点的kublet发送指令</li></ul></li><li><p>node节点的组件（程序）</p><ul><li>kubelet：向docker发送指令管理docker容器</li><li>kubeproxy：管理dcoekr容器的网络</li></ul></li><li><p>etcd</p><ul><li>k8s的数据库</li><li>用来注册节点、服务、记录账户。。。</li></ul></li></ul><p><img src="J:%5Cnote%5CK8s%5Cimgs%5C%E6%9E%B6%E6%9E%84%E5%9B%BE.png" alt="架构图"></p><h2 id="Kubernetes核心概念"><a href="#Kubernetes核心概念" class="headerlink" title="Kubernetes核心概念"></a>Kubernetes核心概念</h2><h3 id="Pod"><a href="#Pod" class="headerlink" title="Pod"></a>Pod</h3><ul><li>最小部署单元</li><li>一组容器的集合</li><li>一个Pod中的容器共享网络命名空间</li><li>Pod是短暂的</li></ul><h3 id="Controllers"><a href="#Controllers" class="headerlink" title="Controllers"></a>Controllers</h3><ul><li>ReplicaSet：确保预期的Pod副本数量</li><li>Deployment：无状态应用部署</li><li>StatefulSet：有状态应用部署</li><li>DaemonSet：确保所有Node运行同一个Pod</li><li>Job：一次性任务</li><li>Cronjob：定时任务</li></ul><h3 id="Service"><a href="#Service" class="headerlink" title="Service"></a>Service</h3><ul><li>防止Pod关联</li><li>定义一组Pod的访问策略</li></ul><h3 id="Label"><a href="#Label" class="headerlink" title="Label"></a>Label</h3><ul><li>标签，添加到某个资源上，用于关联对象、查询和筛选</li></ul><h3 id="Namespace"><a href="#Namespace" class="headerlink" title="Namespace"></a>Namespace</h3><ul><li>命名空间，将对象逻辑上隔离</li><li>例如：为开发环境、测试环境、生产环境分别准备不同的名称空间，进行隔离</li></ul><h2 id="官方提供的三种部署方式"><a href="#官方提供的三种部署方式" class="headerlink" title="官方提供的三种部署方式"></a>官方提供的三种部署方式</h2><h3 id="minikube"><a href="#minikube" class="headerlink" title="minikube"></a>minikube</h3><ul><li>Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，仅用于尝试Kubernetes</li><li>部署地址：<a href="https://kubernetes.io/docs/setup/minikube/" target="_blank" rel="noopener">https://kubernetes.io/docs/setup/minikube/</a></li></ul><h3 id="kubeadm"><a href="#kubeadm" class="headerlink" title="kubeadm"></a>kubeadm</h3><ul><li>Kubeadm也是工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群</li><li>部署地址：<a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/" target="_blank" rel="noopener">https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/</a></li></ul><h3 id="二进制"><a href="#二进制" class="headerlink" title="二进制"></a>二进制</h3><ul><li>推荐，从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群</li><li>下载地址：<a href="https://github.com/kubernetes/kubernetes/releases" target="_blank" rel="noopener">https://github.com/kubernetes/kubernetes/releases</a></li></ul><h2 id="初始化服务器"><a href="#初始化服务器" class="headerlink" title="初始化服务器"></a>初始化服务器</h2><h3 id="部署单master集群"><a href="#部署单master集群" class="headerlink" title="部署单master集群"></a>部署单master集群</h3><h4 id="一：集群规划"><a href="#一：集群规划" class="headerlink" title="一：集群规划"></a>一：集群规划</h4><ul><li>master<ul><li>主机名：k8s-master1</li><li>IP: 192.168.0.100</li></ul></li><li>worker1<ul><li>主机名: k8s-node1</li><li>IP: 192.168.0.201</li></ul></li><li>worker2<ul><li>主机名: k8s-node2</li><li>IP: 192.168.0.202</li></ul></li></ul><p>k8s版本：1.16</p><p>安装方式：离线-二进制</p><p>操作系统 ：Centos 7</p><h3 id="二、初始化服务器"><a href="#二、初始化服务器" class="headerlink" title="二、初始化服务器"></a>二、初始化服务器</h3><ol><li><p>关闭防火墙</p><ul><li>systemctl stop firewalld</li><li>systemctl disable firewalld</li></ul></li><li><p>关闭selinux</p><ul><li><p>临时关闭</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">setenforce 0</span><br></pre></td></tr></table></figure></li><li><p>永久关闭</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/selinux/config</span><br><span class="line">将SELINUX修改成SELINUX=disabled</span><br></pre></td></tr></table></figure></li></ul></li></ol><ol start="3"><li><p>配置主机名</p><ul><li><p>配置主机名</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hostnamectl set-hostname 主机名</span><br></pre></td></tr></table></figure></li></ul></li><li><p>配置名称解析</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/hosts </span><br><span class="line">添加以下 </span><br><span class="line">192.168.0.101 k8s-master1</span><br><span class="line">192.168.0.201 k8s-node1</span><br><span class="line">192.168.0.202 k8s-node2</span><br></pre></td></tr></table></figure></li></ol><ol start="5"><li><p>配置时间同步</p><ul><li><p>选择一个节点作为服务端。剩下的作为客户端 master1作为时间服务器的服务端</p><p>(1) 配置k8s-master1</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">yum install chrony -y # 下载时间</span><br><span class="line">vi /etc/chrony.conf # 修改配置文件</span><br><span class="line"></span><br><span class="line">修改三项：</span><br><span class="line">server 127.0.0.1 iburst</span><br><span class="line">allow 192.168.0.0/24 </span><br><span class="line">local stratum 10</span><br><span class="line">修改完：</span><br><span class="line">systemctl start chronyd</span><br><span class="line">systemctl enable chronyd</span><br><span class="line">systemctl restart chronyd</span><br><span class="line">测试看看有没有成功启动：</span><br><span class="line">ss -unl | grep 123</span><br><span class="line">有这个就成功了：UNCONN 0 0 *:123</span><br></pre></td></tr></table></figure><p>(2) 配置 k8s-node1 和 k8s-node2</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/chrony.conf # 修改配置文件</span><br><span class="line">修改一项：</span><br><span class="line">server 时间服务器IP iburst</span><br><span class="line">systemctl start chronyd</span><br><span class="line">systemctl enable chronyd</span><br><span class="line">systemctl restart chronyd</span><br><span class="line">测试看看同步成功没有：</span><br><span class="line">chronyc sources</span><br><span class="line">出现了 ^* k8s-master1 有星号了就是成功了</span><br></pre></td></tr></table></figure></li></ul></li></ol><ol start="6"><li><p>关闭交换分区</p><ul><li><p>临时关闭分区：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">swapoff -a</span><br></pre></td></tr></table></figure></li><li><p>永久关闭分区:</p><p>修改文件 : </p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/fstab </span><br><span class="line">删除有 swap项的那一行</span><br><span class="line">:wq 保存退出</span><br><span class="line">free -m 查看 swap 为 0</span><br></pre></td></tr></table></figure></li></ul></li></ol><h2 id="三：部署etcd"><a href="#三：部署etcd" class="headerlink" title="三：部署etcd"></a>三：部署etcd</h2><h4 id="给etcd颁发证书"><a href="#给etcd颁发证书" class="headerlink" title="给etcd颁发证书"></a>给etcd颁发证书</h4><ul><li>创建证书颁发机构</li><li>填写表单–写明etcd所在节点的ip</li><li>向证书颁发机构申请证书</li></ul><h4 id="下载cfssl工具"><a href="#下载cfssl工具" class="headerlink" title="下载cfssl工具"></a>下载cfssl工具</h4><p>(在/home/etcd下)使用cfssl来生成自签证书，先下载cfssl工具：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> mv cfssl_linux-amd64 /usr/<span class="built_in">local</span>/bin/cfssl</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> mv cfssljson_linux-amd64 /usr/<span class="built_in">local</span>/bin/cfssljson</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> mv cfssl-certinfo_linux-amd64 /usr/<span class="built_in">local</span>/bin/cfssl-certinfo</span></span><br></pre></td></tr></table></figure><h4 id="生成证书-编写三个json文件"><a href="#生成证书-编写三个json文件" class="headerlink" title="生成证书:编写三个json文件"></a>生成证书:编写三个json文件</h4><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line"># 我的位置在/home/etcd下，三个json文件创建</span><br><span class="line"># cat ca-config.json</span><br><span class="line">{</span><br><span class="line"> <span class="attr">"signing"</span>: {</span><br><span class="line"> <span class="attr">"default"</span>: {</span><br><span class="line"> <span class="attr">"expiry"</span>: <span class="string">"87600h"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"profiles"</span>: {</span><br><span class="line"> <span class="attr">"www"</span>: {</span><br><span class="line"> <span class="attr">"expiry"</span>: <span class="string">"87600h"</span>,</span><br><span class="line"> <span class="attr">"usages"</span>: [</span><br><span class="line"> <span class="string">"signing"</span>,</span><br><span class="line"> <span class="string">"key encipherment"</span>,</span><br><span class="line"> <span class="string">"server auth"</span>,</span><br><span class="line"> <span class="string">"client auth"</span></span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># cat ca-csr.json</span><br><span class="line">{</span><br><span class="line"> <span class="attr">"CN"</span>: <span class="string">"etcd CA"</span>,</span><br><span class="line"> <span class="attr">"key"</span>: {</span><br><span class="line"> <span class="attr">"algo"</span>: <span class="string">"rsa"</span>,</span><br><span class="line"> <span class="attr">"size"</span>: <span class="number">2048</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"names"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"C"</span>: <span class="string">"CN"</span>,</span><br><span class="line"> <span class="attr">"L"</span>: <span class="string">"Beijing"</span>,</span><br><span class="line"> <span class="attr">"ST"</span>: <span class="string">"Beijing"</span></span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"># cat server-csr.json</span><br><span class="line">{</span><br><span class="line"> <span class="attr">"CN"</span>: <span class="string">"etcd"</span>,</span><br><span class="line"> <span class="attr">"hosts"</span>: [</span><br><span class="line"> <span class="string">"192.168.18.80"</span>,</span><br><span class="line"> <span class="string">"192.168.18.85"</span>,</span><br><span class="line"> <span class="string">"192.168.18.86"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"key"</span>: {</span><br><span class="line"> <span class="attr">"algo"</span>: <span class="string">"rsa"</span>,</span><br><span class="line"> <span class="attr">"size"</span>: <span class="number">2048</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"names"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"C"</span>: <span class="string">"CN"</span>,</span><br><span class="line"> <span class="attr">"L"</span>: <span class="string">"BeiJing"</span>,</span><br><span class="line"> <span class="attr">"ST"</span>: <span class="string">"BeiJing"</span></span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">生成证书命令行</span></span><br><span class="line">cfssl gencert -initca ca-csr.json | cfssljson -bare ca -</span><br><span class="line">cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server</span><br><span class="line"><span class="meta">#</span><span class="bash"> ls *pem（应该有4个pem结尾的文件）</span></span><br><span class="line">ca-key.pem ca.pem server-key.pem server.pem</span><br></pre></td></tr></table></figure><h4 id="部署etcd"><a href="#部署etcd" class="headerlink" title="部署etcd"></a>部署etcd</h4><p>这里使用的是二进制部署，不想用其他的方法。</p><p>二进制包下载地址：<a href="https://github.com/coreos/etcd/releases/" target="_blank" rel="noopener">https://github.com/coreos/etcd/releases/</a></p><p>华为加速：<a href="https://mirrors.huaweicloud.com/etcd/" target="_blank" rel="noopener">https://mirrors.huaweicloud.com/etcd/</a></p><p><strong>有etcd的node上都要部署，注意文件夹的名字和位置，不是随便命名和放置的</strong>。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">解压压缩包</span></span><br><span class="line">mkdir /opt/etcd/{bin,cfg,ssl} -p</span><br><span class="line">tar -zxvf etcd-v3.2.12-linux-amd64.tar.gz -C ./</span><br><span class="line">mv etcd-v3.2.12-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/</span><br></pre></td></tr></table></figure><p>创建etcd配置文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> touch etcd</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> chmod 777 etcd</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> cat /opt/etcd/cfg/etcd </span></span><br><span class="line"><span class="meta">#</span><span class="bash">[Member]</span></span><br><span class="line">ETCD_NAME="etcd-1"</span><br><span class="line">ETCD_DATA_DIR="/var/lib/etcd/default.etcd"</span><br><span class="line">ETCD_LISTEN_PEER_URLS="https://192.168.0.101:2380"</span><br><span class="line">ETCD_LISTEN_CLIENT_URLS="https://192.168.0.101:2379"</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">[Clustering]</span></span><br><span class="line">ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.101:2380"</span><br><span class="line">ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.101:2379"</span><br><span class="line">ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.0.101:2380,etcd-2=https://192.168.0.201:2380,etcd-3=https://192.168.0.202:2380"</span><br><span class="line">ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"</span><br><span class="line">ETCD_INITIAL_CLUSTER_STATE="new"</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">-------------------------------------解释-------------------------------------------</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_NAME 节点名称</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_DATA_DIR 数据目录</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_LISTEN_PEER_URLS 集群通信监听地址</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_LISTEN_CLIENT_URLS 客户端访问监听地址</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_INITIAL_ADVERTISE_PEER_URLS 集群通告地址</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_ADVERTISE_CLIENT_URLS 客户端通告地址</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_INITIAL_CLUSTER 集群节点地址</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_INITIAL_CLUSTER_TOKEN 集群Token</span></span><br><span class="line"><span class="meta">#</span><span class="bash">ETCD_INITIAL_CLUSTER_STATE 加入集群的当前状态，new是新集群，existing表示加入已有集群</span></span><br></pre></td></tr></table></figure><p>systemd管理etcd</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"># touch /usr/lib/systemd/system/etcd.service</span><br><span class="line"># cat /usr/lib/systemd/system/etcd.service </span><br><span class="line">[Unit]</span><br><span class="line">Description=Etcd Server</span><br><span class="line">After=network.target</span><br><span class="line">After=network-online.target</span><br><span class="line">Wants=network-online.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">Type=notify</span><br><span class="line">EnvironmentFile=/opt/etcd/cfg/etcd.conf</span><br><span class="line">ExecStart=/opt/etcd/bin/etcd \</span><br><span class="line">--name=${ETCD_NAME} \</span><br><span class="line">--data-dir=${ETCD_DATA_DIR} \</span><br><span class="line">--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \</span><br><span class="line">--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \</span><br><span class="line">--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \</span><br><span class="line">--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \</span><br><span class="line">--initial-cluster=${ETCD_INITIAL_CLUSTER} \</span><br><span class="line">--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \</span><br><span class="line">--initial-cluster-state=new \</span><br><span class="line">--cert-file=/opt/etcd/ssl/server.pem \</span><br><span class="line">--key-file=/opt/etcd/ssl/server-key.pem \</span><br><span class="line">--peer-cert-file=/opt/etcd/ssl/server.pem \</span><br><span class="line">--peer-key-file=/opt/etcd/ssl/server-key.pem \</span><br><span class="line">--trusted-ca-file=/opt/etcd/ssl/ca.pem \</span><br><span class="line">--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem</span><br><span class="line">Restart=on-failure</span><br><span class="line">LimitNOFILE=65536</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br></pre></td></tr></table></figure><p>将之前生成的证书拷贝到固定的目录</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cp ca*pem server*pem /opt/etcd/ssl</span><br></pre></td></tr></table></figure><p>启动并设置开机启动</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">systemctl start etcd</span><br><span class="line">systemctl enable etcd</span><br></pre></td></tr></table></figure><p>注意：以上步骤需要在所有要安装etcd的node上执行一次，不是只在master node上执行，可以通过scp命令发送到其他node，需要改一下etcd的配置文件，节点名和节点ip。</p><p>在都部署完成后，检查一下集群的状态(若是出现cluster is healthy，说明部署成功！)</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[root@k8s_master1 cfg]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.0.101:2379,https://192.168.0.201:2379,https://192.168.0.202:2379" cluster-health</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">member feb9f9de26a0d6d is healthy: got healthy result from https://192.168.18.80:2379</span><br><span class="line">member 94c59c0ee7401835 is healthy: got healthy result from https://192.168.18.85:2379</span><br><span class="line">member a5106f019eb73cbd is healthy: got healthy result from https://192.168.18.86:2379</span><br><span class="line">cluster is healthy</span><br></pre></td></tr></table></figure><h3 id="部署master服务"><a href="#部署master服务" class="headerlink" title="部署master服务"></a>部署master服务</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"># tar xvf k8s-master.tar.gz</span><br><span class="line"># mv kube-apiserver.service kube-controller-manager.service kube-scheduler.service /usr/lib/systemd/system/</span><br><span class="line"># mv kubernetes /opt/</span><br><span class="line"># cp /home/etcd/{ra*pem, server*pem} /opt/kubernetes/ssl/ -rvf</span><br></pre></td></tr></table></figure><ul><li>修改 apiserver的配置文件</li></ul>]]></content>
</entry>
<entry>
<title>Hadoop/MapReduce/MapRreduce示例</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/MapReduce/MapRreduce%E7%A4%BA%E4%BE%8B/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/MapReduce/MapRreduce%E7%A4%BA%E4%BE%8B/</url>
<content type="html"><![CDATA[<p>MapRreduce示例</p><h1 id="MapReduce示例"><a href="#MapReduce示例" class="headerlink" title="MapReduce示例"></a>MapReduce示例</h1><p>计算单词</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> com.imooc.bigdata.hadoop.mr.wc;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.IntWritable;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.LongWritable;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.Text;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Mapper;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.IOException;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> JackSon</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@version</span> 1.0</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@description</span></span></span><br><span class="line"><span class="comment"> * <span class="doctag">@create</span> 2020-01-28 15:54</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * KEYIN: Map任务数据的key类型，offset，是每行数据起始位置的偏移量，Long类型</span></span><br><span class="line"><span class="comment"> * VALUEIN：Map任务读数据的value类型，其实就是一行行的字符串，String</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * KEYOUT：Map方法自定义实现输出的key的类型</span></span><br><span class="line"><span class="comment"> * VALUEOUT：Map方法自定义实现输出的value的类型</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">WordCountMapper</span> <span class="keyword">extends</span> <span class="title">Mapper</span><<span class="title">LongWritable</span>, <span class="title">Text</span>, <span class="title">Text</span>, <span class="title">IntWritable</span>> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">map</span><span class="params">(LongWritable key, Text value, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>{</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 把value对应的行数据按照指定的分隔符拆开</span></span><br><span class="line"> String[] words = value.toString().split(<span class="string">"\t"</span>);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (String word : words) {</span><br><span class="line"> <span class="comment">//</span></span><br><span class="line"> context.write(<span class="keyword">new</span> Text(word.toLowerCase()), <span class="keyword">new</span> IntWritable(<span class="number">1</span>));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">package com.imooc.bigdata.hadoop.mr.wc;</span><br><span class="line"></span><br><span class="line">import org.apache.hadoop.io.IntWritable;</span><br><span class="line">import org.apache.hadoop.io.Text;</span><br><span class="line">import org.apache.hadoop.mapreduce.Reducer;</span><br><span class="line"></span><br><span class="line">import java.io.IOException;</span><br><span class="line">import java.util.Iterator;</span><br><span class="line"></span><br><span class="line">/**</span><br><span class="line"> * @author JackSon</span><br><span class="line"> * @version 1.0</span><br><span class="line"> * @description</span><br><span class="line"> * @create 2020-01-28 16:10</span><br><span class="line"> */</span><br><span class="line">public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {</span><br><span class="line"></span><br><span class="line"> @Override</span><br><span class="line"> protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {</span><br><span class="line"></span><br><span class="line"> int count = 0;</span><br><span class="line"></span><br><span class="line"> Iterator<IntWritable> iterator = values.iterator();</span><br><span class="line"></span><br><span class="line"> while (iterator.hasNext()) {</span><br><span class="line"> IntWritable value = iterator.next();</span><br><span class="line"> count += value.get();</span><br><span class="line"> }</span><br><span class="line"> context.write(key, new IntWritable(count));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br></pre></td><td class="code"><pre><span class="line">package com.imooc.bigdata.hadoop.mr.wc;</span><br><span class="line"></span><br><span class="line">import org.apache.hadoop.conf.Configuration;</span><br><span class="line">import org.apache.hadoop.fs.FileSystem;</span><br><span class="line">import org.apache.hadoop.fs.Path;</span><br><span class="line">import org.apache.hadoop.io.IntWritable;</span><br><span class="line">import org.apache.hadoop.io.Text;</span><br><span class="line"></span><br><span class="line">import org.apache.hadoop.mapred.FileInputFormat;</span><br><span class="line">import org.apache.hadoop.mapred.FileOutputFormat;</span><br><span class="line">import org.apache.hadoop.mapred.JobConf;</span><br><span class="line">import org.apache.hadoop.mapred.Reducer;</span><br><span class="line">import org.apache.hadoop.mapreduce.Job;</span><br><span class="line">import org.apache.hadoop.mapreduce.Mapper;</span><br><span class="line">import org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer;</span><br><span class="line"></span><br><span class="line">import java.io.IOException;</span><br><span class="line">import java.net.URI;</span><br><span class="line"></span><br><span class="line">/**</span><br><span class="line"> * @author JackSon</span><br><span class="line"> * @version 1.0</span><br><span class="line"> * @description</span><br><span class="line"> * @create 2020-01-28 16:24</span><br><span class="line"> */</span><br><span class="line">public class WordCountApp {</span><br><span class="line"></span><br><span class="line"> public static void main(String[] args) throws Exception {</span><br><span class="line"></span><br><span class="line"> System.setProperty("HADOOP_USER_NAME", "jackson");</span><br><span class="line"></span><br><span class="line"> Configuration configuration = new Configuration();</span><br><span class="line"> configuration.set("fs.defaultFS", "hdfs://192.168.146.138:8020");</span><br><span class="line"> try {</span><br><span class="line"> Job job = Job.getInstance(configuration);</span><br><span class="line"> JobConf jobConf = (JobConf)job.getConfiguration();</span><br><span class="line"> // Hadoop 2.0为Job设置 Jar class，否则任务执行过程中无法找到Map类和Reduce类</span><br><span class="line"> job.setJarByClass(WordCountApp.class);</span><br><span class="line"> job.setJobName("word count");</span><br><span class="line"> job.setMapperClass(WordCountMapper.class);</span><br><span class="line"> job.setReducerClass(IntSumReducer.class);</span><br><span class="line"> job.setOutputKeyClass(Text.class);</span><br><span class="line"> job.setOutputValueClass(IntWritable.class);</span><br><span class="line"></span><br><span class="line"> // 添加Combiner的设置即可 重点***** </span><br><span class="line"> job.setCombinerClass(WordCountReducer.class);</span><br><span class="line"></span><br><span class="line"> // 如果输出目录已经存在，则先删除</span><br><span class="line"> FileSystem fileSystem = FileSystem.get(new URI("hdfs://192.168.146.138:8020"), configuration, "jackson");</span><br><span class="line"> Path outputPath = new Path("/wordcount/output");</span><br><span class="line"> if (fileSystem.exists(outputPath)) {</span><br><span class="line"> fileSystem.delete(outputPath, true);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> FileInputFormat.setInputPaths(jobConf, new Path("/wordcount/input/"));</span><br><span class="line"> FileOutputFormat.setOutputPath(jobConf, new Path("/wordcount/output/"));</span><br><span class="line"> boolean result = job.waitForCompletion(true);</span><br><span class="line"> System.exit(result ? 0 : -1);</span><br><span class="line"> } catch (ClassNotFoundException e) {</span><br><span class="line"> // TODO Auto-generated catch block</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> // TODO Auto-generated catch block</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } catch (InterruptedException e) {</span><br><span class="line"> // TODO Auto-generated catch block</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>Hadoop/MapReduce/MapReduce自定义分区</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/MapReduce/MapReduce%E8%87%AA%E5%AE%9A%E4%B9%89%E5%88%86%E5%8C%BA/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/MapReduce/MapReduce%E8%87%AA%E5%AE%9A%E4%B9%89%E5%88%86%E5%8C%BA/</url>
<content type="html"><![CDATA[<p>MapReduce自定义分区</p><p>因为默认分区只有一个，而且算法是官方自带的，由此我们自定义分区</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">package com.imooc.bigdata.hadoop.mr.access;</span><br><span class="line"></span><br><span class="line">import org.apache.hadoop.io.Text;</span><br><span class="line">import org.apache.hadoop.mapreduce.Partitioner;</span><br><span class="line"></span><br><span class="line">/**</span><br><span class="line"> * @author JackSon</span><br><span class="line"> * @version 1.0</span><br><span class="line"> * @description MapReduce自定义分区规则</span><br><span class="line"> * @create 2020-01-28 22:10</span><br><span class="line"> */</span><br><span class="line">public class AccessPartitioner extends Partitioner<Text, Access> {</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> @Override</span><br><span class="line"> public int getPartition(Text phone, Access access, int numPartitions) {</span><br><span class="line"> // reutrn 返回的number指的是分区的位置</span><br><span class="line"> if (phone.toString().startsWith("13")) {</span><br><span class="line"> return 1;</span><br><span class="line"> } else if (phone.toString().startsWith("15")) {</span><br><span class="line"> return 2;</span><br><span class="line"> } else {</span><br><span class="line"> return 0;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>然后在工作区添加</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 自定义分区规则</span></span><br><span class="line">job.setPartitionerClass(AccessPartitioner<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"><span class="comment">// 设置reduce个数 个数超过自定义设置的返回分区数是不会报错的，但是多余的分区是没有东西的</span></span><br><span class="line">job.setNumReduceTasks(<span class="number">3</span>);</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>Hadoop/MapReduce/MapReduce概述</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/MapReduce/MapReduce%E6%A6%82%E8%BF%B0/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/MapReduce/MapReduce%E6%A6%82%E8%BF%B0/</url>
<content type="html"><![CDATA[<p>MapReduce概述</p><h1 id="分布式计算框架MapReduce"><a href="#分布式计算框架MapReduce" class="headerlink" title="分布式计算框架MapReduce"></a>分布式计算框架MapReduce</h1><h2 id="MapReduce概述"><a href="#MapReduce概述" class="headerlink" title="MapReduce概述"></a>MapReduce概述</h2><ul><li>源自于Google的MapReduce论文，论文发表于2004年12月</li><li>Hadoop MapReduce是Google MapReduce的克隆版</li><li>MapReduce优点：海量数据离线处理&易开发&易运行</li><li>MapReduce缺点：实时流式计算</li></ul><h2 id="MapReduce-编程模型之执行步骤"><a href="#MapReduce-编程模型之执行步骤" class="headerlink" title="MapReduce 编程模型之执行步骤"></a>MapReduce 编程模型之执行步骤</h2><ul><li>准备map处理的输入数据</li><li>Mapper处理</li><li>Shuffle</li><li>Reduce处理</li><li>结果输出</li></ul><h2 id="MapReduce-编程模型之核心概念"><a href="#MapReduce-编程模型之核心概念" class="headerlink" title="MapReduce 编程模型之核心概念"></a>MapReduce 编程模型之核心概念</h2><ul><li>Split</li><li>InputFormat</li><li>OutputFormat</li><li>Combiner</li><li>Partitioner</li></ul>]]></content>
</entry>
<entry>
<title>Hadoop/hdfs/java API 使用hdfs</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/hdfs/java%20API%20%E4%BD%BF%E7%94%A8hdfs/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/hdfs/java%20API%20%E4%BD%BF%E7%94%A8hdfs/</url>
<content type="html"><![CDATA[<p>java API 使用hdfs</p><h1 id="Java-API-使用-HDFS"><a href="#Java-API-使用-HDFS" class="headerlink" title="Java API 使用 HDFS"></a>Java API 使用 HDFS</h1><ul><li>pom.xml 文件</li></ul><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">hadoop.version</span>></span>3.2.1<span class="tag"></<span class="name">hadoop.version</span>></span></span><br><span class="line"><span class="tag"></<span class="name">properties</span>></span></span><br><span class="line"><span class="tag"><<span class="name">dependencies</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.hadoop<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>hadoop-client<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${hadoop.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>junit<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>junit<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>4.11<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>test<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.slf4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>slf4j-simple<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.7.25<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>compile<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">dependencies</span>></span></span><br></pre></td></tr></table></figure><p>引入依赖后</p><p>使用API</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br></pre></td><td class="code"><pre><span class="line">```</span><br><span class="line"><span class="keyword">package</span> com.imooc.bigdata.hadoop.hdfs.hadoop.hdfs;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> jdk.nashorn.internal.ir.Block;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.conf.Configuration;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.*;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.IOUtils;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.util.Progressable;</span><br><span class="line"><span class="keyword">import</span> org.junit.After;</span><br><span class="line"><span class="keyword">import</span> org.junit.Before;</span><br><span class="line"><span class="keyword">import</span> org.junit.Test;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.*;</span><br><span class="line"><span class="keyword">import</span> java.net.URI;</span><br><span class="line"><span class="keyword">import</span> java.net.URISyntaxException;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> *<span class="doctag">@description</span></span></span><br><span class="line"><span class="comment"> *<span class="doctag">@author</span> JackSon</span></span><br><span class="line"><span class="comment"> *<span class="doctag">@create</span> 2020-01-24 20:33</span></span><br><span class="line"><span class="comment"> *<span class="doctag">@version</span> 1.0</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">HDFSApp</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String HDFS_PATH = <span class="string">"hdfs://192.168.146.138:8020"</span>;</span><br><span class="line"> FileSystem fileSystem = <span class="keyword">null</span>;</span><br><span class="line"> Configuration configuration = <span class="keyword">null</span>;</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Before</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">setUp</span><span class="params">()</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> configuration = <span class="keyword">new</span> Configuration();</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 构造一个访问指定HDFS系统的客户端对象</span></span><br><span class="line"><span class="comment"> * 第一个参数：HDFS的URI</span></span><br><span class="line"><span class="comment"> * 第二个参数：客户端指定的配置参数</span></span><br><span class="line"><span class="comment"> * 第三个参数：客户端的身份，说白了就是用户名</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> fileSystem = FileSystem.get(<span class="keyword">new</span> URI(HDFS_PATH), configuration, <span class="string">"jackson"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 创建文件夹</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> Exception</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">mkdir</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> fileSystem.mkdirs(<span class="keyword">new</span> Path(<span class="string">"/test1"</span>));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 查看内容</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">text</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> FSDataInputStream in = fileSystem.open(<span class="keyword">new</span> Path(<span class="string">"/README.txt"</span>));</span><br><span class="line"> IOUtils.copyBytes(in, System.out, <span class="number">1024</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 创建文件</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> Exception</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">create</span><span class="params">()</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> FSDataOutputStream out = fileSystem.create(<span class="keyword">new</span> Path(<span class="string">"/test1/hello2.txt"</span>));</span><br><span class="line"> out.writeUTF(<span class="string">"hello"</span>);</span><br><span class="line"> out.flush();</span><br><span class="line"> out.close();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 修改文件名</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">rename</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> Path oldPath = <span class="keyword">new</span> Path(<span class="string">"/test1/a.txt"</span>);</span><br><span class="line"> Path newPath = <span class="keyword">new</span> Path(<span class="string">"/test1/b.txt"</span>);</span><br><span class="line"> <span class="keyword">boolean</span> result = fileSystem.rename(oldPath, newPath);</span><br><span class="line"> System.out.println(<span class="string">"result = "</span> + result);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@After</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">tearDown</span><span class="params">()</span></span>{</span><br><span class="line"> configuration = <span class="keyword">null</span>;</span><br><span class="line"> fileSystem = <span class="keyword">null</span>;</span><br><span class="line"> System.out.println(<span class="string">"-----------------tearDown--------------------"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">copyFromLocalBigFile</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> InputStream in = <span class="keyword">new</span> BufferedInputStream(<span class="keyword">new</span> FileInputStream(<span class="keyword">new</span> File(<span class="string">"J:\\学习软件及工具包\\Linux系统\\jdk-8u231-linux-x64.tar.gz"</span>)));</span><br><span class="line"> FSDataOutputStream out = fileSystem.create(<span class="keyword">new</span> Path(<span class="string">"/hdfs/test3/jdk.tgz"</span>),</span><br><span class="line"> <span class="keyword">new</span> Progressable() {</span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">progress</span><span class="params">()</span> </span>{</span><br><span class="line"> System.out.println(<span class="string">"."</span>);</span><br><span class="line"> }</span><br><span class="line"> });</span><br><span class="line"> IOUtils.copyBytes(in, out, <span class="number">4096</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">copyToLocalFile</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> Path oldPath = <span class="keyword">new</span> Path(<span class="string">"/LICENSE.txt"</span>);</span><br><span class="line"> Path newPath = <span class="keyword">new</span> Path(<span class="string">"D:\\"</span>);</span><br><span class="line"> fileSystem.copyToLocalFile(oldPath, newPath);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 查看目录下的文件</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> Exception</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">listFiles</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> FileStatus[] fileStatuses = fileSystem.listStatus(<span class="keyword">new</span> Path(<span class="string">"/"</span>));</span><br><span class="line"> <span class="keyword">for</span> (FileStatus file : fileStatuses) {</span><br><span class="line"> String isDir = file.isDirectory() ? <span class="string">"文件夹"</span> : <span class="string">"文件"</span>;</span><br><span class="line"> String permission = file.getPermission().toString();</span><br><span class="line"> <span class="keyword">short</span> replication = file.getReplication();</span><br><span class="line"> <span class="keyword">long</span> len = file.getLen();</span><br><span class="line"> String path = file.getPath().toString();</span><br><span class="line"></span><br><span class="line"> System.out.println(isDir + <span class="string">"\t"</span> + permission + <span class="string">"\t"</span> + replication + <span class="string">"\t"</span> + len + <span class="string">"\t"</span> + path);</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 递归查看目录下的文件</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> Exception</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">listFilesRecursive</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> RemoteIterator<LocatedFileStatus> files = fileSystem.listFiles(<span class="keyword">new</span> Path(<span class="string">"/"</span>), <span class="keyword">true</span>);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">while</span> (files.hasNext()) {</span><br><span class="line"> LocatedFileStatus file = files.next();</span><br><span class="line"> String isDir = file.isDirectory() ? <span class="string">"文件夹"</span> : <span class="string">"文件"</span>;</span><br><span class="line"> String permission = file.getPermission().toString();</span><br><span class="line"> <span class="keyword">short</span> replication = file.getReplication();</span><br><span class="line"> <span class="keyword">long</span> len = file.getLen();</span><br><span class="line"> String path = file.getPath().toString();</span><br><span class="line"></span><br><span class="line"> System.out.println(isDir + <span class="string">"\t"</span> + permission + <span class="string">"\t"</span> + replication + <span class="string">"\t"</span> + len + <span class="string">"\t"</span> + path);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 获取文件块信息数据</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">getFileBlockLocations</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> FileStatus fileStatus = fileSystem.getFileStatus(<span class="keyword">new</span> Path(<span class="string">"/hdfs/test3/jdk.tgz"</span>));</span><br><span class="line"> BlockLocation[] blocks = fileSystem.getFileBlockLocations(fileStatus, <span class="number">0</span>, fileStatus.getLen());</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (BlockLocation block : blocks){</span><br><span class="line"> <span class="keyword">for</span> (String name : block.getNames()) {</span><br><span class="line"> System.out.println(name + <span class="string">":"</span> + block.getOffset());</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 删除文件</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> Exception</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">delete</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> <span class="keyword">boolean</span> delete = fileSystem.delete(<span class="keyword">new</span> Path(<span class="string">"/hdfs/test3/jdk.tgz"</span>), <span class="keyword">true</span>);</span><br><span class="line"> System.out.println(<span class="string">"delete = "</span> + delete);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"><span class="comment">// public static void main(String[] args) throws Exception {</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// Configuration configuration = new Configuration();</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// FileSystem fileSystem = FileSystem.get(new URI("hdfs://192.168.146.138:8020"), configuration, "jackson");</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// Path path = new Path("/hdfsapi/test");</span></span><br><span class="line"><span class="comment">// boolean mkdirs = fileSystem.mkdirs(path);</span></span><br><span class="line"><span class="comment">// System.out.println("mkdirs = " + mkdirs);</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// }</span></span><br><span class="line"></span><br><span class="line">}```</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>Hadoop/hdfs/hdfs学习</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/hdfs/hdfs%E5%AD%A6%E4%B9%A0/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/hdfs/hdfs%E5%AD%A6%E4%B9%A0/</url>
<content type="html"><![CDATA[<p>hdfs学习</p><h1 id="HDFS学习"><a href="#HDFS学习" class="headerlink" title="HDFS学习"></a>HDFS学习</h1><h2 id="HDFS概述"><a href="#HDFS概述" class="headerlink" title="HDFS概述"></a>HDFS概述</h2><ol><li>分布式</li><li>commodity hardware</li><li>fault-tolerant 容错</li><li>high throughput</li><li>large data sets</li></ol><h2 id="HDFS前提和设计目标"><a href="#HDFS前提和设计目标" class="headerlink" title="HDFS前提和设计目标"></a>HDFS前提和设计目标</h2><pre><code>Hardware Failure 硬件错误 每个机器只存储文件的部分数据， blocksize = 128M block存放在不同的机器上，由于容错Streaming Data Access 流式数据访问Large Data Sets 大规模数据集</code></pre><h2 id="HDFS的架构"><a href="#HDFS的架构" class="headerlink" title="HDFS的架构"></a>HDFS的架构</h2><ol><li>NameNode(master) and DataNodes(slave)</li><li>master/slave <del>架构</del></li><li>一个blocksize最大是128M，如果文件大于128M则拆成2个block</li></ol>]]></content>
</entry>
<entry>
<title>Hadoop/安装SSH无密码登录</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/%E5%AE%89%E8%A3%85SSH%E6%97%A0%E5%AF%86%E7%A0%81%E7%99%BB%E5%BD%95/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/%E5%AE%89%E8%A3%85SSH%E6%97%A0%E5%AF%86%E7%A0%81%E7%99%BB%E5%BD%95/</url>
<content type="html"><![CDATA[<p>安装SSH无密码登录</p><h1 id="安装SSH无密码登录"><a href="#安装SSH无密码登录" class="headerlink" title="安装SSH无密码登录"></a>安装SSH无密码登录</h1><h2 id="一：连接服务器"><a href="#一：连接服务器" class="headerlink" title="一：连接服务器"></a>一：连接服务器</h2><p>ssh 用户名@服务器ip</p><p>看到要授权ssh钥匙则yes</p><h2 id="二：查看根目录是否有ssh文件夹"><a href="#二：查看根目录是否有ssh文件夹" class="headerlink" title="二：查看根目录是否有ssh文件夹"></a>二：查看根目录是否有ssh文件夹</h2><ul><li><code>ls -a</code> 或者 <code>ls -la</code> 查看隐藏文件</li></ul><h2 id="三：安装"><a href="#三：安装" class="headerlink" title="三：安装"></a>三：安装</h2><ul><li><p><code>ssh-keygen -t rsa</code> 然后一路回车</p></li><li><p>查看 <code>cd ~/.ssh</code> </p></li><li><p>-rw-------. 1 jackson jackson 411 Jan 23 23:01 authorized_keys<br>-rw-------. 1 jackson jackson 1675 Jan 23 22:58 id_rsa<br>-rw-r--r--. 1 jackson jackson 411 Jan 23 22:58 id_rsa.pub<br>-rw-r--r--. 1 jackson jackson 171 Jan 23 22:57 known_hosts </p></li><li><p><code>cat idrsa.pub >> authorized_keys</code> </p></li><li><p><code>chmod 600 authorized_keys</code> </p></li></ul>]]></content>
</entry>
<entry>
<title>Hadoop/Hadoop概述</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/Hadoop%E6%A6%82%E8%BF%B0/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/Hadoop%E6%A6%82%E8%BF%B0/</url>
<content type="html"><![CDATA[<p>Hadoop 学习</p><h1 id="Hadoop-学习"><a href="#Hadoop-学习" class="headerlink" title="Hadoop 学习"></a>Hadoop 学习</h1><h2 id="初识Hadoop"><a href="#初识Hadoop" class="headerlink" title="初识Hadoop"></a>初识Hadoop</h2><h3 id="Hadoop核心组件之分布式文件系统HDFS"><a href="#Hadoop核心组件之分布式文件系统HDFS" class="headerlink" title="Hadoop核心组件之分布式文件系统HDFS"></a>Hadoop核心组件之分布式文件系统HDFS</h3><ul><li><p>源自于Google的GFS论文，论文发表于2003年10月</p></li><li><p>HDFS是GFS的克隆版</p></li><li><p>HDFS特点：扩展性&容错性&海量数量存储</p></li><li><p>将文件切分成指定大小的数据块并以多副本的存储在多个机器上</p></li><li><p>数据切分、多副本、容错等操作对用户是透明的</p></li></ul><h3 id="Hadoop核心组件之分布式计算框架MapReduce"><a href="#Hadoop核心组件之分布式计算框架MapReduce" class="headerlink" title="Hadoop核心组件之分布式计算框架MapReduce"></a>Hadoop核心组件之分布式计算框架MapReduce</h3><ul><li>源自于Google的MapReduce论文，论文发表于2004年12月</li><li>MapReduce是Google，MapReduce的克隆版</li><li>MapReduce特点：扩展性&容错性&海量数据离线处理</li></ul><h3 id="Hadoop核心组件之资源调度系统YARN"><a href="#Hadoop核心组件之资源调度系统YARN" class="headerlink" title="Hadoop核心组件之资源调度系统YARN"></a>Hadoop核心组件之资源调度系统YARN</h3><ul><li>YARN：Yet Another Resource Negotiator</li><li>负责整个集群资源的管理和调度</li><li>YARN特点：扩展性&容错性&多框架统资源统一调度</li></ul><h3 id="Hadoop优势之高可靠性"><a href="#Hadoop优势之高可靠性" class="headerlink" title="Hadoop优势之高可靠性"></a>Hadoop优势之高可靠性</h3><ul><li>数据存储：数据块多副本</li><li>数据计算：重新调度作业计算</li></ul><h3 id="Hadoop优势之高扩展性"><a href="#Hadoop优势之高扩展性" class="headerlink" title="Hadoop优势之高扩展性"></a>Hadoop优势之高扩展性</h3><ul><li>存储/计算资源不够时，可以横向的线性扩展机器</li><li>一个集群中可以包含数以千计的节点</li></ul><h2 id="Hadoop-常用命令"><a href="#Hadoop-常用命令" class="headerlink" title="Hadoop 常用命令"></a>Hadoop 常用命令</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">hadoop fs -ls /</span><br><span class="line">hadoop fs -put</span><br><span class="line">hadoop fs -copyFromLocal</span><br><span class="line">hadoop fs -moveFromLocal</span><br><span class="line">hadoop fs -cat</span><br><span class="line">hadoop fs -text</span><br><span class="line">hadoop fs -get</span><br><span class="line">hadoop fs -mkdir</span><br><span class="line">hadoop fs -mv</span><br><span class="line">hadoop fs -getmerge</span><br><span class="line">hadoop fs -rm</span><br><span class="line">hadoop fs -rmdir</span><br><span class="line">hadoop fs -rm -r</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>Hadoop/Hadoop安装</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/Hadoop%E5%AE%89%E8%A3%85/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Hadoop/Hadoop%E5%AE%89%E8%A3%85/</url>
<content type="html"><![CDATA[<p>Hadoop安装</p><h1 id="Hadopp安装"><a href="#Hadopp安装" class="headerlink" title="Hadopp安装"></a>Hadopp安装</h1><h2 id="下载解压："><a href="#下载解压：" class="headerlink" title="下载解压："></a>下载解压：</h2><p>wget <a href="http://mirror.bit.edu.cn/apache/hadoop/common/hadoop\-3.2.1/hadoop\-3.2.1.tar.gz" target="_blank" rel="noopener">http://mirror.bit.edu.cn/apache/hadoop/common/hadoop\-3.2.1/hadoop\-3.2.1.tar.gz</a></p><p>这是3.2.1版本的</p><p>tar -xzvf 解压即可</p><h2 id="hadoop软件包常见目录说明"><a href="#hadoop软件包常见目录说明" class="headerlink" title="hadoop软件包常见目录说明"></a>hadoop软件包常见目录说明</h2><ul><li><p>bin : hadoop 客户端名单</p></li><li><p>etc/hadoop ： hadoop相关的配置文件存放目录</p></li><li><p>sbin：启动hadoop相关进程的脚本</p></li><li><p>share：常用例子</p></li></ul><h2 id="hadoop-配置"><a href="#hadoop-配置" class="headerlink" title="hadoop 配置"></a>hadoop 配置</h2><p>环境变量：</p><p> <code>vim /etc/profile</code> 修改</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">```</span><br><span class="line"> `source /etc/profile` 立即生效</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">修改hadoop配置文件</span><br><span class="line"></span><br><span class="line">vim /home/jackson/app/hadoop/etc/hadoop/hadoop\-env.sh</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">修改JAVA\_HOME</span><br><span class="line"></span><br><span class="line">export JAVA\_HOME=/home/jackson/app/jdk1.8</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">修改core\-size.xml</span><br><span class="line"></span><br><span class="line">vim /home/jackson/app/hadoop/etc/hadoop/core\-size.xml</span><br><span class="line"></span><br><span class="line">```xml</span><br><span class="line"><configuration></span><br><span class="line"> <property> </span><br><span class="line"> <name>fs.defaultFS</name> </span><br><span class="line"> <value>hdfs://localhost:8020</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><p>修改hdfs-site.xml</p><p>vim /home/jackson/app/hadoop/etc/hadoop/hdfs-site.xml</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.replication<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.tmp.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/home/jackson/app/tmp<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><p>编辑</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop000 # 这个是你hosts的名称</span><br></pre></td></tr></table></figure><h2 id="启动HDFS："><a href="#启动HDFS：" class="headerlink" title="启动HDFS："></a>启动HDFS：</h2><p>第一次执行的时候一定要格式化文件系统，不要重复执行：</p><p> <code>hdfs namenode -format</code> </p><p>启动集群：</p><p> <code>$HADOOP_HOME/sbin/start-dfs.sh</code> </p><p>web查看状态：</p><p><a href="http://192.168.146.138:9870" target="_blank" rel="noopener">http://192.168.146.138:9870</a> 如果连接不上，可能是防火墙原因，关闭防火墙或者开启端口 </p>]]></content>
</entry>
<entry>
<title>GitLab/GitLab</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/GitLab/GitLab/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/GitLab/GitLab/</url>
<content type="html"><![CDATA[<h1 id="GitLab使用"><a href="#GitLab使用" class="headerlink" title="GitLab使用"></a>GitLab使用</h1><h2 id="创建-gitlab（使用docker）"><a href="#创建-gitlab（使用docker）" class="headerlink" title="创建 gitlab（使用docker）"></a>创建 gitlab（使用docker）</h2><ul><li>创建文件 docker-compose.yaml 编辑如下内容</li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">'3'</span></span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"> <span class="attr">gitlab:</span></span><br><span class="line"> <span class="attr">image:</span> <span class="string">'twang2218/gitlab-ce-zh:11.1.4'</span></span><br><span class="line"> <span class="attr">restart:</span> <span class="string">always</span></span><br><span class="line"> <span class="attr">hostname:</span> <span class="string">'gitlab.example.com'</span></span><br><span class="line"> <span class="attr">environment:</span></span><br><span class="line"> <span class="attr">TZ:</span> <span class="string">'Asia/Shanghai'</span></span><br><span class="line"> <span class="attr">GITLAB_OMNIBUS_CONFIG:</span> <span class="string">|</span></span><br><span class="line"> <span class="string">external_url</span> <span class="string">'http://192.168.111.138:8080'</span></span><br><span class="line"> <span class="string">gitlab_rails['gitlab_shell_ssh_port']</span> <span class="string">=</span> <span class="number">2222</span></span><br><span class="line"> <span class="string">unicorn['port']</span> <span class="string">=</span> <span class="number">8888</span></span><br><span class="line"> <span class="string">nginx['listen_port']</span> <span class="string">=</span> <span class="number">8080</span></span><br><span class="line"> <span class="comment"># 需要配置到 gitlab.rb 中的配置可以在这里配置，每个配置一行，注意缩进。</span></span><br><span class="line"> <span class="comment"># 比如下面的电子邮件的配置：</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_enable'] = true</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_address'] = "smtp.exmail.qq.com"</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_port'] = 465</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_user_name'] = "xxxx@xx.com"</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_password'] = "password"</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_authentication'] = "login"</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_enable_starttls_auto'] = true</span></span><br><span class="line"> <span class="comment"># gitlab_rails['smtp_tls'] = true</span></span><br><span class="line"> <span class="comment"># gitlab_rails['gitlab_email_from'] = 'xxxx@xx.com'</span></span><br><span class="line"> <span class="attr">ports:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">'8080:8080'</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">'8443:443'</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">'2222:22'</span></span><br><span class="line"> <span class="attr">volumes:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/usr/local/docker/gitlab/config:/etc/gitlab</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/usr/local/docker/gitlab/data:/var/opt/gitlab</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/usr/local/docker/gitlab/logs:/var/log/gitlab</span></span><br></pre></td></tr></table></figure><ul><li>使用 docker-compose 来启动 gitlab</li></ul><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker-compose up -d</span><br></pre></td></tr></table></figure><ul><li><p>查看 启动是否完成</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">docker ps # 查看容器</span><br><span class="line">docker logs 容器id # 查看运行日志</span><br></pre></td></tr></table></figure></li></ul><h2 id="访问-gitlab"><a href="#访问-gitlab" class="headerlink" title="访问 gitlab"></a>访问 gitlab</h2><p>去浏览器输入地址: 192.168.111.1:8080 这是我的ip地址，请输入自己的服务器的ip</p><ul><li>访问后一开始要输入 root 用户的密码</li><li>登录进去</li></ul><p>修改配置 三个地方</p><h3 id="第一个"><a href="#第一个" class="headerlink" title="第一个"></a>第一个</h3><p><img src="J:%5Cnote%5CGitLab%5Cimgs%5C%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20200417221749.png" alt="微信图片_20200417221749"></p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417221913340.png" alt="image-20200417221913340"></p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417221941455.png" alt="image-20200417221941455"></p><p>拉倒下面</p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417222017889.png" alt="image-20200417222017889"></p><p>修改成 SSH and HTTPS</p><h3 id="第二个"><a href="#第二个" class="headerlink" title="第二个"></a>第二个</h3><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417222107580.png" alt="image-20200417222107580"></p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417222134824.png" alt="image-20200417222134824"></p><p>关掉打钩</p><h3 id="第三个"><a href="#第三个" class="headerlink" title="第三个"></a>第三个</h3><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417222153268.png" alt="image-20200417222153268"></p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417222204487.png" alt="image-20200417222204487"></p><p>关掉打钩</p><h2 id="使用SSH密匙"><a href="#使用SSH密匙" class="headerlink" title="使用SSH密匙"></a>使用SSH密匙</h2><p>在命令窗口打开 填写如下命令</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ssh-keygen -t rsa -C "自己在gitlab的邮箱"</span><br></pre></td></tr></table></figure><p>如果是windows 找到 C:\Users\用户.ssh 下面 找到</p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417223008141.png" alt="image-20200417223008141"></p><p>去到 gitlab</p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417223841435.png" alt="image-20200417223841435"></p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417223859831.png" alt="image-20200417223859831"></p><p><img src="C:%5CUsers%5CJackSon%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20200417224100083.png" alt="image-20200417224100083"></p>]]></content>
</entry>
<entry>
<title>GitLab/docker-compose</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/GitLab/docker-compose/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/GitLab/docker-compose/</url>
<content type="html"><![CDATA[{"version":"3","services":{"gitlab":{"image":"twang2218/gitlab-ce-zh:11.1.4","restart":"always","hostname":"gitlab.example.com","environment":{"TZ":"Asia/Shanghai","GITLAB_OMNIBUS_CONFIG":"external_url 'http://192.168.111.138:8080'\ngitlab_rails['gitlab_shell_ssh_port'] = 2222\nunicorn['port'] = 8888\nnginx['listen_port'] = 8080\n# 需要配置到 gitlab.rb 中的配置可以在这里配置，每个配置一行，注意缩进。\n# 比如下面的电子邮件的配置：\n# gitlab_rails['smtp_enable'] = true\n# gitlab_rails['smtp_address'] = \"smtp.exmail.qq.com\"\n# gitlab_rails['smtp_port'] = 465\n# gitlab_rails['smtp_user_name'] = \"xxxx@xx.com\"\n# gitlab_rails['smtp_password'] = \"password\"\n# gitlab_rails['smtp_authentication'] = \"login\"\n# gitlab_rails['smtp_enable_starttls_auto'] = true\n# gitlab_rails['smtp_tls'] = true\n# gitlab_rails['gitlab_email_from'] = 'xxxx@xx.com'\n"},"ports":["8080:8080","8443:443","2222:22"],"volumes":["/usr/local/docker/gitlab/config:/etc/gitlab","/usr/local/docker/gitlab/data:/var/opt/gitlab","/usr/local/docker/gitlab/logs:/var/log/gitlab"]}}}]]></content>
</entry>
<entry>
<title>Docker/Docker新一轮学习 </title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Docker/Docker%E6%96%B0%E4%B8%80%E8%BD%AE%E5%AD%A6%E4%B9%A0%20/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Docker/Docker%E6%96%B0%E4%B8%80%E8%BD%AE%E5%AD%A6%E4%B9%A0%20/</url>
<content type="html"><![CDATA[<p>Docker新一轮学习</p><h1 id="Docker学习"><a href="#Docker学习" class="headerlink" title="Docker学习"></a>Docker学习</h1><h2 id="Docker命令"><a href="#Docker命令" class="headerlink" title="Docker命令"></a>Docker命令</h2><h3 id="Docker-基本命令"><a href="#Docker-基本命令" class="headerlink" title="Docker 基本命令"></a>Docker 基本命令</h3><ul><li>启动Docker <code>systemctl start docker</code> </li><li>停止Docker <code>systemctl stop docker</code> </li><li>重启Docker <code>systemctl restart docker</code> </li><li>开机启动Docker <code>systemctl enable docker</code> </li><li>查看Docker概要信息 <code>docker info</code> </li><li>查看Docker帮助文档 <code>docker --help</code> </li><li>查看Docker版本信息 <code>docker version</code> </li></ul><h3 id="Docker-镜像命令"><a href="#Docker-镜像命令" class="headerlink" title="Docker 镜像命令"></a>Docker 镜像命令</h3><ul><li><p>列出本机所有镜像 <code>docker images</code> </p><ul><li><p>可选参数</p><ul><li><p>-a ：显示所有镜像（包括中间层）</p></li><li><p>-q ：只显示镜像ID</p></li><li><p>-qa ：可以组合</p></li><li><p>--digests ：显示镜像的摘要信息</p></li><li><p>--no-trunc ：显示完整的镜像信息</p></li></ul></li></ul></li><li><p>搜索镜像 <code>docker search</code> </p><ul><li><p>可选参数</p><ul><li><p>--no-trunc ：显示完整的镜像描述</p></li><li><p>-s ：列出收藏数不小于指定值的镜像</p></li><li><p>--automated ：只列出Docker Hub自动构建类型的镜像</p></li></ul></li></ul></li><li><p>下载镜像 <code>docker pull</code> </p></li><li><p>删除镜像 <code>docker rmi</code> </p><ul><li><p>可选参数</p><ul><li>-f ：用来强制停止正在运行的容器并且删除</li></ul></li></ul></li></ul><h3 id="Docker-容器操作基本命令"><a href="#Docker-容器操作基本命令" class="headerlink" title="Docker 容器操作基本命令"></a>Docker 容器操作基本命令</h3><ul><li><p>创建并启动容器： <code>docker run [OPTIONS] IMAGE [COMMAND] [ARG...]</code> </p><ul><li><p>OPTIONS 参数</p><ul><li><p>--name=”容器新名字” :为容器指定一个名称</p></li><li><p>-i：以交互模式运行容器，通常与-t同时使用</p></li><li><p>-t：为容器重新分配一个伪输入终端，通常与-i同时使用</p></li><li><p>-d：后台运行容器，并返回容器ID</p></li><li><p>-p：端口映射 例子: -p 8080:8080 ，前一个8080是宿主机的端口，后一个是容器的端口</p></li><li><p>-v :容器目录挂载</p><ul><li><p>例子：docker run -it -v 宿主机文件:容器文件 容器名称</p></li><li><p>多个目录挂载：docker run -it -v 宿主机文件:容器文件 -v 宿主机文件2:容器文件2 容器名称</p></li><li><p>注意如果权限不足表示：需要添加 <code>--privileged=true</code> 来解决</p></li></ul></li></ul></li><li><p>常用：docker run -it --name 别名 镜像ID 来运行一个容器，取别名，交互模式运行，以及分配一个伪终端</p></li></ul></li><li><p>列出容器：docker ps [OPTIONS] </p><ul><li><p>OPTIONS 参数</p><ul><li><p>-a ：查看所有容器</p></li><li><p>-n 2 ：显示最近创建的2个容器</p></li><li><p>-f status=exited ：查看停止的容器</p></li></ul></li></ul></li><li><p>退出容器：exit或者Ctrl+P+Q</p></li><li><p>进入容器：<code>docker attach 容器ID or 容器名</code></p></li><li><p>启动容器：<code>docker start 容器ID or 容器名</code> </p></li><li><p>重启容器： <code>docker restart 容器ID or 容器名</code> </p></li><li><p>停止容器： <code>docker stop 容器ID or 容器名</code> <code>docker kill 容器ID or 容器名</code> </p></li><li><p>删除容器： <code>docker rm 容器ID or 容器名</code> </p><ul><li><p>参数 </p><ul><li>-f : 强制删除正在运行的容器</li></ul></li><li><p>组合语句</p><ul><li>docker rm -f ${docker ps -qa} ：删除所有容器</li></ul></li></ul></li><li><p>进入容器执行命令： <code>docker exec -it 容器名称 or 容器ID 执行命令</code></p></li><li><p>查看容器日志： <code>docker logs [OPTIONS] CONTAINER</code> </p><ul><li><p>OPTIONS</p><ul><li>--details 显示更多的信息</li></ul></li></ul></li></ul><pre><code>* \-f, \-\-follow 跟踪实时日志* \-\-since string 显示自某个timestamp之后的日志，或相对时间，如42m（即42分钟）* \-\-tail string 从日志末尾显示多少行日志， 默认是all* \-t, \-\-timestamps 显示时间戳* \-\-until string 显示自某个timestamp之前的日志，或相对时间，如42m（即42分钟）</code></pre><ul><li><p>查看容器进程： docker top [OPTIONS] CONTAINER [ps OPTIONS]</p></li><li><p>宿主机和容器之间的文件拷贝：</p><ul><li><p>从容器里拷贝文件到宿主机</p><ul><li><code>docker cp 容器名:要拷贝的文件在容器里面的路径 要拷贝到宿主机的相应路径</code> </li></ul></li><li><p>从宿主机里拷贝文件到容器</p><ul><li><code>docker cp 要拷贝的文件路径 容器名:要拷贝到容器里面对应的路径</code> </li></ul></li></ul></li><li><p>提交运行时容器称为镜像： <code>docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]</code> </p><ul><li><p>OPTIONS</p><ul><li>-a :提交的镜像作者；</li></ul></li></ul></li></ul><pre><code>* \-c :使用Dockerfile指令来创建镜像；* \-m :提交时的说明文字；* \-p :在commit时，将容器暂停。</code></pre><ul><li><p>例子</p><ul><li>docker commit -a “runoob.com” -m “my apache” a404c6c174a2 mymysql:v1 </li></ul></li></ul><h3 id="容器备份与恢复"><a href="#容器备份与恢复" class="headerlink" title="容器备份与恢复"></a>容器备份与恢复</h3><p>备份:<br> <code>docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]</code> </p><ul><li><p>OPTIONS</p><ul><li>-a :提交的镜像作者；</li></ul></li></ul><pre><code>* \-c :使用Dockerfile指令来创建镜像；* \-m :提交时的说明文字；* \-p :在commit时，将容器暂停。</code></pre><p>恢复:<br> <code>docker load [OPTIONS]</code> </p><ul><li><p>OPTIONS</p><ul><li>--input , -i : 指定导入的文件，代替 STDIN。</li></ul></li></ul><pre><code>* \-\-quiet , \-q : 精简输出信息</code></pre><h3 id="DockerFile"><a href="#DockerFile" class="headerlink" title="DockerFile"></a>DockerFile</h3><p>常用命令：</p><ul><li><p>FROM img_name:tag ：定义了使用哪个基础镜像启动构建流程</p></li><li><p>MAINTAINER user_info ：声明镜像维护者信息</p></li><li><p>LABEL key value ：镜像描述元信息（可以写多条）</p></li><li><p>ENV key value ：设置环境变量（可以写多条）</p></li><li><p>RUN command ： 构建容器时需要运行的命令（可以写多条）</p></li><li><p>WORKDIR path_dir ：设置终端默认登录进来的工作目录</p></li><li><p>EXPOSE port ：当前容器对外暴露出的端口</p></li><li><p>ADD sourcedir/file dest _dir/file ：将宿主机的文件复制到容器内，如果是一个压缩文件，将会在复制后自动解压</p></li><li><p>COPY sourcedir/file dest_dir/file ：和ADD相似，但是如果有压缩文件是不能解压</p></li><li><p>VOLUME ：创建一个可用从本地主机或其他容器挂载的挂载点，一般用来存放数据库和需要保持的数据等</p></li><li><p>CMD ：指定容器启动时需要运行的命令，假如有多个CMD，最后一个生效</p></li><li><p>ENTRYPOINT ：指定容器启动时要运行的命令</p></li><li><p>ONBUILD ：当构建一个被继承的Dockerfile时运行的命令，父镜像在被子镜像继承后父镜像的onbuild被触发。可以把onbuild理解为一个触发器</p></li></ul><h3 id="Docker-搭建-Registry仓库"><a href="#Docker-搭建-Registry仓库" class="headerlink" title="Docker 搭建 Registry仓库"></a>Docker 搭建 Registry仓库</h3><p>拉取镜像:<br> <code>docker pull registry</code><br>运行容器:<br> <code>docker run -itd -v /data/registry:/var/lib/registry -p 5000:5000 --restart=always --name registry</code><br>查看：<br><a href="http://127.0.0.1:5000/v2/\_catalog" target="_blank" rel="noopener">http://127.0.0.1:5000/v2/\_catalog</a><br>修改文件 daemon.json</p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> ""insecure-registries":["192.168.146.138:5000"]"</span><br><span class="line">}</span><br><span class="line">```重启docker</span><br><span class="line"> `systemctl restart docker` </span><br><span class="line"></span><br><span class="line">推送：</span><br><span class="line"> `docker push 自己的IP:5000/容器名称` </span><br><span class="line"></span><br><span class="line">拉取：</span><br><span class="line"> `docker pull 自己的IP:5000/容器名称` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line">### Docker 推送镜像到 Docker Hub</span><br><span class="line">登录：</span><br><span class="line"> `docker login` </span><br><span class="line"></span><br><span class="line">输入好用户名和密码后</span><br><span class="line"></span><br><span class="line">上传：</span><br><span class="line">* \`docker push 用户名/镜像名称:版本号</span><br><span class="line"></span><br><span class="line">### Docker 推送到阿里云</span><br><span class="line">官方地址：[https://cr.console.aliyun.com/](https://cr.console.aliyun.com/)</span><br><span class="line">* 步骤</span><br><span class="line"></span><br><span class="line"> * 创建命名空间</span><br><span class="line"></span><br><span class="line"> * 创建镜像仓库</span><br><span class="line"></span><br><span class="line"> * 进入镜像仓库</span><br><span class="line">* 登录阿里云docker</span><br><span class="line"></span><br><span class="line"> * `sudo docker login --username=程序猿辉哥 registry.cn-hangzhou.aliyuncs.com` </span><br><span class="line">* 推送镜像</span><br><span class="line"></span><br><span class="line"> * `sudo docker tag [ImageId] registry.cn-hangzhou.aliyuncs.com/jackson1/tomcat:[镜像版本号]` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> * `sudo docker push registry.cn-hangzhou.aliyuncs.com/jackson1/tomcat:[镜像版本号]` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">### Docker 推送到 harbor</span><br><span class="line"></span><br><span class="line">地址下载：[https://storage.googleapis.com/harbor\-releases/](https://storage.googleapis.com/harbor-releases/)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">首先下载 docker\-compose</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">版本号可以改</span><br><span class="line"></span><br><span class="line">添加权限</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"></span><br><span class="line">将下载完的harbor解压</span><br><span class="line"></span><br><span class="line">修改文件 harbor.yml</span><br><span class="line"></span><br><span class="line">* hostname = A.B.C.D \# 写你自己的网址或IP，公网访问要写公网IP</span><br><span class="line">* 支持http访问: customize\_crt = false</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">运行： `./prepare` </span><br><span class="line"></span><br><span class="line">成功后再运行 `./install.sh` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">#### 常用管理命令</span><br><span class="line"></span><br><span class="line">* 停止服务： </span><br><span class="line">* 开始服务： </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">修改 daemon.json文件</span><br></pre></td></tr></table></figure><p>{<br> “insecure-registries”: [<br> “A.B.C.D”]<br> }</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"></span><br><span class="line">修改完后再重启docker `systemctl restart docker` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">上传镜像：</span><br><span class="line"></span><br><span class="line"> `docker login A.B.C.D` </span><br><span class="line"></span><br><span class="line"> `docker push A.B.C.D/新建的项目名称/镜像名称:tag` </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">拉取镜像:</span><br><span class="line"></span><br><span class="line">`docker pull A.B.C.D/新建的项目名称/镜像名称:tag`</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">docker中 启动所有的容器命令</span><br><span class="line"></span><br><span class="line">docker start $(docker ps -a | awk '{ print $1}' | tail -n +2)</span><br><span class="line">docker中 关闭所有的容器命令</span><br><span class="line"></span><br><span class="line">docker stop $(docker ps -a | awk '{ print $1}' | tail -n +2)</span><br><span class="line">docker中 删除所有的容器命令</span><br><span class="line"></span><br><span class="line">docker rm $(docker ps -a | awk '{ print $1}' | tail -n +2)</span><br><span class="line">docker中 删除所有的镜像</span><br><span class="line"></span><br><span class="line">docker rmi $(docker images | awk '{print $3}' |tail -n +2)</span><br><span class="line">tail -n +2 表示从第二行开始读取</span><br></pre></td></tr></table></figure>]]></content>
</entry>
<entry>
<title>Docker/Docker 日志爆满清理</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/Docker/Docker%20%E6%97%A5%E5%BF%97%E7%88%86%E6%BB%A1%E6%B8%85%E7%90%86/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/Docker/Docker%20%E6%97%A5%E5%BF%97%E7%88%86%E6%BB%A1%E6%B8%85%E7%90%86/</url>
<content type="html"><![CDATA[<h1 id="Docker-日志爆满清理"><a href="#Docker-日志爆满清理" class="headerlink" title="Docker 日志爆满清理"></a>Docker 日志爆满清理</h1><ul><li><p>查看 docker 那些日志占用多</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">du -d1 -h /var/lib/docker/containers | sort -h</span><br></pre></td></tr></table></figure><p>例子:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">36K /var/lib/docker/containers/4d91f92dd7604216f2e9e123572e9a80d7bbee3d8c8ce7be2ed241c572816fb6</span><br><span class="line">40K /var/lib/docker/containers/374aa0ba92b37d829012282ff15c1bb838d95dedb54589874c4285991be2d4aa</span><br><span class="line">40K /var/lib/docker/containers/7cfdbc453b2f7109b52e974061754266e6cdc0ccaee62ab4a5887865113e1144</span><br><span class="line">40K /var/lib/docker/containers/84ee24989ad52383c6e99eaa4d236e600095aa7f855e81fbafe10416b75ceefb</span><br><span class="line">40K /var/lib/docker/containers/aeced3ef3e23df27e52f65743bb05448b46a2c660acc5b0aab12604e060779b4</span><br><span class="line">40K /var/lib/docker/containers/c36722baf0d2e1c22b7dde9979665ab62cd8ab85c3f1d0f427bb7a34e0fd977a</span><br><span class="line">44K /var/lib/docker/containers/62477b332d18e192d70c7420435d47a379e6bbd8de13da8a8762e0fd95b341ca</span><br><span class="line">44K /var/lib/docker/containers/78da0cf9743b6940fabbbd8c574b99dc5deb642fa998a8f819a6c6978fc875d7</span><br><span class="line">44K /var/lib/docker/containers/9f63daf7caa7c469385bed4b178fbfe662e15b8c569c6644081af090f8e40426</span><br><span class="line">44K /var/lib/docker/containers/e2d1286119a45aac7e58d6dac6e4b44b1d1288799b735943be45abed50244e56</span><br><span class="line">56K /var/lib/docker/containers/ebd1bd211a1b9d02bb39bfb80eec3d0960a5b25e18f54d7371781ec456e7a1e8</span><br><span class="line">176K /var/lib/docker/containers/1fe0a241e5ce9726c547c68739793633f9dd906768a36fe80e8fb80373aa3bfb</span><br><span class="line">17M /var/lib/docker/containers/ac30e68d454b37d22b3964053a2b52ba043baa1add13556a09c0e3e05589104f</span><br><span class="line">25M /var/lib/docker/containers/872ca4e3d005594591ca2df0e832d36eef448981ab2820c69df4ff1399f8423e</span><br><span class="line">25M /var/lib/docker/containers/bd49a0a0368b99a9f69981d8b921ea1830957451577b635a07d5425d48e1144b</span><br><span class="line">30M /var/lib/docker/containers/8f732390a020a6ef647fabb04da32c87d6341b72ac2af6bb4a1cf5743fda54db</span><br><span class="line">88M /var/lib/docker/containers/648e883aa0a93f696f64e4ab76434657f4845769fe1eaaad49c2dc1d7960f2b0</span><br><span class="line">171M /var/lib/docker/containers/8de7ff9f0276586a6ab346c2be1c9dc879bbb0d795fa7776c1d8d1568ea2794a</span><br><span class="line">354M /var/lib/docker/containers</span><br></pre></td></tr></table></figure></li><li><p>选择你要清理的容器进行清理</p><p>进入到某个容器的日志文件夹里面</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cd /var/lib/docker/containers/8de7ff9f0276586a6ab346c2be1c9dc879bbb0d795fa7776c1d8d1568ea2794a</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cat /dev/null > /var/lib/docker/containers/container_id/container_log_name</span><br></pre></td></tr></table></figure></li></ul>]]></content>
</entry>
<entry>
<title>SpringBoot ELK 的日志收集</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/29/ELK/SpringBoot%20ELK%20%E7%9A%84%E6%97%A5%E5%BF%97%E6%94%B6%E9%9B%86/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/29/ELK/SpringBoot%20ELK%20%E7%9A%84%E6%97%A5%E5%BF%97%E6%94%B6%E9%9B%86/</url>
<content type="html"><![CDATA[<h3 id="SpringBoot-ELK-的日志收集"><a href="#SpringBoot-ELK-的日志收集" class="headerlink" title="SpringBoot ELK 的日志收集"></a>SpringBoot ELK 的日志收集</h3><p>引入依赖:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><dependency></span><br><span class="line"> <groupId>net.logstash.logback</groupId></span><br><span class="line"> <artifactId>logstash-logback-encoder</artifactId></span><br><span class="line"> <version>6.3</version></span><br><span class="line"></dependency></span><br></pre></td></tr></table></figure><p>在resource文件夹下创建文件 logback-spring.xml</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta"><?xml version="1.0" encoding="UTF-8"?></span></span><br><span class="line"><span class="tag"><<span class="name">configuration</span> ></span></span><br><span class="line"> <span class="comment"><!-- 控制台输出 --></span></span><br><span class="line"> <span class="tag"><<span class="name">appender</span> <span class="attr">name</span>=<span class="string">"LOGSTASH"</span> <span class="attr">class</span>=<span class="string">"net.logstash.logback.appender.LogstashTcpSocketAppender"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">destination</span>></span>111.230.248.15:5000<span class="tag"></<span class="name">destination</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">encoder</span>></span></span><br><span class="line"> <span class="comment"><!--格式化输出：%d表示日期，%thread表示线程名，%-5level：级别从左显示5个字符宽度%msg：日志消息，%n是换行符--></span></span><br><span class="line"> <span class="tag"><<span class="name">pattern</span>></span>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n<span class="tag"></<span class="name">pattern</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">charset</span>></span>UTF-8<span class="tag"></<span class="name">charset</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">encoder</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">appender</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">include</span></span></span><br><span class="line"><span class="tag"> <span class="attr">resource</span>=<span class="string">"org/springframework/boot/logging/logback/base.xml"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- 日志输出级别 --></span></span><br><span class="line"> <span class="tag"><<span class="name">root</span> <span class="attr">level</span>=<span class="string">"INFO"</span>></span>-</span><br><span class="line"> <span class="tag"><<span class="name">appender-ref</span> <span class="attr">ref</span>=<span class="string">"CONSOLE"</span> /></span></span><br><span class="line"> <span class="tag"><<span class="name">appender-ref</span> <span class="attr">ref</span>=<span class="string">"LOGSTASH"</span> /></span></span><br><span class="line"> <span class="tag"></<span class="name">root</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><p>去github下载 docker-compose </p><p><a href="https://github.com/deviantony/docker-elk" target="_blank" rel="noopener">https://github.com/deviantony/docker-elk</a></p><p>下载后执行 docker-compose up （当然里面有了三个 elasticsearch 和 logstash 以及 kibana，里面的文件夹conf配置文件需要根据你个人需要去修改）</p><p>修改 logstash下的pipeline文件下的logstash.conf</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line">tcp {</span><br><span class="line">port => 5000</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">## Add your filters / logstash plugins configuration here</span><br><span class="line">filter {</span><br><span class="line">grok {</span><br><span class="line">match => {</span><br><span class="line">"message" => "%{TIMESTAMP_ISO8601:time} %{GREEDYDATA:logThread} %{LOGLEVEL:logLevel} %{GREEDYDATA:loggerClass} - %{GREEDYDATA:content}"</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">output {</span><br><span class="line">elasticsearch {</span><br><span class="line">hosts => "elasticsearch:9200"</span><br><span class="line">user => "elastic"</span><br><span class="line">password => "changeme"</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>然后启动 springboot 再去查看</p><p>用户密码默认conf文件中: elastic changeme</p>]]></content>
</entry>
<entry>
<title>Hello World</title>
<link href="/hejiahuichengxuyuan.github.io/2020/07/28/hello-world/"/>
<url>/hejiahuichengxuyuan.github.io/2020/07/28/hello-world/</url>
<content type="html"><![CDATA[<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="noopener">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/" target="_blank" rel="noopener">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html" target="_blank" rel="noopener">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues" target="_blank" rel="noopener">GitHub</a>.</p><h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo new <span class="string">"My New Post"</span></span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/writing.html" target="_blank" rel="noopener">Writing</a></p><h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo server</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/server.html" target="_blank" rel="noopener">Server</a></p><h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo generate</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/generating.html" target="_blank" rel="noopener">Generating</a></p><h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ hexo deploy</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/one-command-deployment.html" target="_blank" rel="noopener">Deployment</a></p>]]></content>
</entry>
</search>