Skip to content

Commit 76bc730

Browse files
twotwotwotwo
committed
add config for platform
1 parent 3dfc14f commit 76bc730

File tree

4 files changed

+148
-10
lines changed

4 files changed

+148
-10
lines changed

‎log_to_graphs/README.md

+65-7
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# 日志转图工具
22

3-
## 依赖
3+
## Environment
4+
5+
[Installing Python 2.7 on Centos 6.5](../python27_on_centos65.md)
46

57
`pip install matplotlib numpy cairocffi`
68
* matplotlib: http://wiki.li3huo.com/matplotlib
@@ -104,14 +106,70 @@
104106

105107
### Others: ../smtp/mail_tool.py 把报告以邮件形式进行发送
106108

109+
110+
## 生成日报表的过程(./gen-report.sh)
111+
112+
### 1. 日志汇总&合并
113+
114+
```bash
107115
#!/bin/sh
108-
today=`date +%Y-%m-%d`
109-
for ip in 10.75.2.15 10.75.0.16; do
110-
rsync -avzp -e "ssh -p 2188" feiliu-user@${ip}:/data/www/gas.feiliu.com/logs/activate/all/${today}.log ${today}_${ip}.log
116+
117+
yesterday=`date -d last-day +%Y%m%d`
118+
logfile="/var/log/platform/access.log-${yesterday}"
119+
store_dir=/data0/monitor/daily_logs/platform_log
120+
121+
# load logs to ${store_dir}
122+
for ip in 10.163.61.176 10.163.57.246; do
123+
rsync -avzp -e "ssh -p 22" ${ip}:${logfile} ${store_dir}/${yesterday}_${ip}.log
111124
done
112-
if [ -n "$1" ]; then
113-
sort -m ${today}_*.log |grep $1
125+
126+
# merge logs to ${store_dir}/${yesterday}.log
127+
sort -m ${store_dir}/${yesterday}_*.log -o ${store_dir}/${yesterday}.log
128+
129+
merge_filter=$1
130+
if [ -n "${merge_filter}" ]; then
131+
sort -m ${store_dir}/${yesterday}_*.log |grep ${merge_filter} > ${store_dir}/${yesterday}.log
114132
else
115-
sort -m ${today}_*.log
133+
sort -m ${store_dir}/${yesterday}_*.log -o ${store_dir}/${yesterday}.log
116134
fi
117135

136+
```
137+
138+
### 2. log to data(numpy_helper.py)
139+
140+
```bash
141+
# 准备python脚本执行环境
142+
pip install virtualenv
143+
virtualenv env
144+
source env/bin/activate
145+
# 安装需要的类库
146+
pip install matplotlib numpy cairocffi
147+
```
148+
149+
```bash
150+
src_dir=/data0/monitor/daily_logs
151+
ini_file=/data0/monitor/daily_logs/helper.ini
152+
pic_file="/data0/monitor/daily_logs/report/daily-report-${yesterday}.png";
153+
PATH=${src_dir}/env/bin:$PATH
154+
python ${src_dir}/numpy_helper.py -c parse -i ${ini_file} -s platform
155+
```
156+
157+
### 3. data to graphs(daily_log_plot.py)
158+
159+
```bash
160+
python ${src_dir}/daily_log_plot.py -n /tmp/pay_v4.npz -p ${pic_file} -t "Web Platform Daily Report(${yesterday})" --not-show
161+
cp ${pic_file} ${src_dir}/platform_daily_report.png
162+
```
163+
164+
165+
### 4. add to crontab
166+
167+
```bash
168+
crontab -e
169+
# generate report on 2:00 am
170+
0 2 * * * sh /data0/monitor/daily_logs/gen-report.sh
171+
#30 11 * * * sh /data0/monitor/daily_logs/gen-report.sh
172+
# send mail on 2:20 am
173+
20 2 * * * python /data0/monitor/daily_logs/mail_tool.py platform
174+
#05 17 * * * python /data0/monitor/daily_logs/mail_tool.py platform
175+
```

‎log_to_graphs/helper.ini

+13-1
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,16 @@ log_file=logs/pay_perform.log.20170919
2828
#log_file=/data/daily_report/pay-log/report/pay_perform.log.%(yesterday)s
2929
request_cmd=awk '{print substr($0,12,2)":"substr($0,15,2)":"substr($0,18,2) }' %(log_file)s |sort | uniq -c
3030
##hour, cost, code
31-
resp_hour_cmd=awk -F '\\\\x02' '{print substr($0,12,2), $8, $9}' %(log_file)s
31+
resp_hour_cmd=awk -F '\\\\x02' '{print substr($0,12,2), $8, $9}' %(log_file)s
32+
33+
[platform]
34+
##log sample: [2018-08-16T23:59:03.013] [INFO] http - 111.200.239.186 - http://monitor.feiliu.com/ - 0 - "GET /article/home/10053?max=1 HTTP/1.0" 200 755 - 16 ms - "" "axios/0.18.0"
35+
timestamp=%Y%m%d
36+
cost_type=ms
37+
npz_file=platform.npz
38+
log_file=logs/platform.log
39+
#log_file=/data0/monitor/daily_logs/platform_log/%(yesterday)s.log
40+
# 每秒的请求数: HH:mm:ss count
41+
request_cmd=awk '{print substr($0,13,8) }' %(log_file)s |sort | uniq -c
42+
##hour, cost, code
43+
resp_hour_cmd=awk -F ' - ' '{split($6,cost," " );split($5,status," " );print substr($0,13,2), cost[1], status[4]}' %(log_file)s

‎log_to_graphs/numpy_helper.py

+3-2
Original file line numberDiff line numberDiff line change
@@ -40,12 +40,13 @@ def __init__(self, config_file, section):
4040
self.cost_type = config.get(section, 'cost_type')
4141
self.request_cmd = config.get(section, 'request_cmd', 0, {'log_file': log_file})
4242
self.resp_hour_cmd = config.get(section, 'resp_hour_cmd', 0, {'log_file': log_file})
43-
self.auth_resp_groupby_ch_cmd = config.get(section, 'auth_resp_groupby_ch_cmd', 0, {'log_file': log_file})
4443

4544
log( 'npz_file = %s, log_file = %s'% (self.npz_file, log_file) )
4645
log( 'request_cmd = %s'% self.request_cmd )
4746
log( 'resp_hour_cmd = %s'% self.resp_hour_cmd )
48-
log( 'auth_resp_groupby_ch_cmd = %s'% self.auth_resp_groupby_ch_cmd )
47+
if config.has_option(section, 'auth_resp_groupby_ch_cmd'):
48+
self.auth_resp_groupby_ch_cmd = config.get(section, 'auth_resp_groupby_ch_cmd', 0, {'log_file': log_file})
49+
log( 'auth_resp_groupby_ch_cmd = %s'% self.auth_resp_groupby_ch_cmd )
4950

5051
def parse(self, save=False):
5152
"""根据加载的配置信息，按指定方式读取日志数据

‎python27_on_centos65.md

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Installing Python 2.7 on Centos 6.5
2+
3+
Centos 6.\* comes with Python 2.6, but we can't just replace it with v2.7 because it's used by the OS internally (apparently) so you will need to install v2.7 (or 3.x, for that matter) along with it. Fortunately, CentOS made this quite painless with their [Software Collections Repository](http://wiki.centos.org/AdditionalResources/Repositories/SCL)
4+
5+
```bash
6+
# yum update # update yum
7+
# yum install centos-release-scl # install SCL
8+
# yum install python27 # install Python 2.7
9+
# /opt/rh/python27/root/usr/bin/python2.7 -V
10+
/opt/rh/python27/root/usr/bin/python2.7: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
11+
```
12+
13+
To make 2.7 work, config `/etc/ld.so.conf.d/python27.conf`:
14+
15+
```bash
16+
# /sbin/ldconfig -v |grep python
17+
/sbin/ldconfig: /etc/ld.so.conf.d/kernel-2.6.32-642.el6.x86_64.conf:6: duplicate hwcap 1 nosegneg
18+
/sbin/ldconfig: Path `/lib' given more than once
19+
/sbin/ldconfig: Path `/usr/lib' given more than once
20+
libpyglib-2.0-python.so.0 -> libpyglib-2.0-python.so.0.0.0
21+
libpython2.6.so.1.0 -> libpython2.6.so.1.0
22+
# find / -name libpython2.7.so.1.0
23+
/opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
24+
# echo '/opt/rh/python27/root/usr/lib64/' > /etc/ld.so.conf.d/python27.conf
25+
# /sbin/ldconfig -v |grep python
26+
/sbin/ldconfig: /etc/ld.so.conf.d/kernel-2.6.32-642.el6.x86_64.conf:6: duplicate hwcap 1 nosegneg
27+
/sbin/ldconfig: Path `/lib' given more than once
28+
/sbin/ldconfig: Path `/usr/lib' given more than once
29+
/opt/rh/python27/root/usr/lib64:
30+
libpython2.7.so.1.0 -> libpython2.7.so.1.0
31+
libpyglib-2.0-python.so.0 -> libpyglib-2.0-python.so.0.0.0
32+
libpython2.6.so.1.0 -> libpython2.6.so.1.0
33+
/opt/rh/python27/root/usr/lib64/tls: (hwcap: 0x8000000000000000)
34+
/opt/rh/python27/root/usr/lib64/sse2: (hwcap: 0x0000000004000000)
35+
# /opt/rh/python27/root/usr/bin/python2.7 -V
36+
Python 2.7.13
37+
```
38+
39+
To install additional libraries, you will need to install PIP:
40+
41+
```bash
42+
# wget https://bootstrap.pypa.io/get-pip.py
43+
# /opt/rh/python27/root/usr/bin/python2.7 get-pip.py
44+
```
45+
46+
once installed, you can install PIP using `pip2.7`, e.g.:
47+
48+
```bash
49+
# pip2.7 install requests
50+
# /opt/rh/python27/root/usr/bin/pip -V
51+
pip 18.0 from /opt/rh/python27/root/usr/lib/python2.7/site-packages/pip (python 2.7)
52+
```
53+
54+
(Hack)Add in `~/.bash_profile` or `~/.zshrc`
55+
56+
```bash
57+
alias python=/opt/rh/python27/root/usr/bin/python
58+
alias pip=/opt/rh/python27/root/usr/bin/pip
59+
export PATH=/opt/rh/python27/root/usr/bin:$PATH
60+
```
61+
62+
`pip install -U pip # Update pip`
63+
64+
## Reference
65+
66+
- [Installing Python 2.7 on Centos 6.5](https://gist.github.com/dalegaspi/dec44117fa5e7597a559)
67+
- [CentOS6系统安装Python2.7](https://www.jianshu.com/p/b8792a7b5350)

0 commit comments

Comments
 (0)