最近在学习ELK，做日志分析。采集apache2日志时需要json格式,随手写篇博客记录一下操作。

apache2.4

apache2配置文件和1之前的有点不一样,之前yum安装好是在/etc/httpd/httpd.conf。
apache2的在/etc/apache2/apache2.conf，日志格式也在这里面配置

#vim /etc/apache2/apache2.conf

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

输入/LogFormat查找日志格式字段。我的是apache2.4 是上面这样的
前面LogFormat日志格式，后面vhost_combined是自定义标识，中间是apache提供的格式字符串，具体参考apache2官网，我这里列举一些常用的。

来源

配置

参照上表根据需要将日志格式化

一个标准的json如下

{
    "@timestamp":"2020-06-24T15:21:29+0800",
    "time":"[24/Jun/2020:15:21:29 +0800]",
    "clientip":"183.157.11.166",
    "remote_user":"-",
    "request":"GET /static/img/icons/favicon-32x32.png HTTP/1.1",
    "status":"200",
    "http_vhost":"file.showa.fun:443",
    "URL":"/static/img/icons/favicon-32x32.png",
    "http_referrer":"https://file.showa.fun:85/login?redirect=%2Ffiles%2F",
    "http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36 Edg/83.0.478.54",
    "message":"183.157.11.166------[24/Jun/2020:15:21:29 +0800]-GET /static/img/icons/favicon-32x32.png HTTP/1.1-2001250"
}

推荐直接再加一条日志格式配置，然后后面写上自定义标识。
我的日志格式如下，标识是apache-json。

#vim /etc/apache2/apache2.conf
LogFormat '{"time": "%t","clientip": "%h","remote_user": "%u","request": "%r","status": "%s","http_vhost": "%v:%p","URL": "%U","http_referer": "%{Referer}i","http_user_agent": "%{User-Agent}i","message": "%h--%l-%u-%t-%r-%>s%b"}'   apache-json
:x

保存并退出后,编辑配置文件/etc/apache2/sites-available/000-default.conf 和default-ssl.conf，如果有其他虚拟主机的配置文件也要改。

#vim 000-default.conf
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log apache-json
\\将原先的combined改成自定义标识
:x
保存并退出
#systemctl reload apache2
\\重载apache2
#tail -100f /var/logs/apache2/access.log
\\验证看下日志输出

{"time": "[24/Jun/2020:16:36:14 +0800]","clientip": "183.157.11.166","remote_user": "-","request": "GET /123.txt HTTP/1.1","status": "200","http_vhost": "127.0.1.1:443","URL": "/123.txt","http_referer": "https://file.showa.fun:88/","http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36 Edg/83.0.478.54","message": "183.157.11.166------[24/Jun/2020:16:36:14 +0800]-GET /123.txt HTTP/1.1-2003"}
{"time": "[24/Jun/2020:16:36:32 +0800]","clientip": "183.157.11.166","remote_user": "-","request": "-","status": "408","http_vhost": "127.0.1.1:443","URL": "-","http_referer": "-","http_user_agent": "-","message": "183.157.11.166------[24/Jun/2020:16:36:32 +0800]---408-"}

如果不确定自己json格式是否合法，可以去https://www.json.cn/#这里验证一下。

配置filebeat

apache2这边日志已经配置好了，再设置一下日志收集程序就好了

#vim /opt/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/access.log
  json.keys_under_root: true		\\注意这俩行是开启json格式输出
  json.add_error_key: true
  fields:
    appname: "apache2-access"

- type: log
  enabled: true
  paths:
    - /var/log/apache2/other_vhosts_access.log
  json.keys_under_root: true
  json.add_error_key: true
  fields:
          appname: "apache2-vhosts-access"

output.logstash:
  hosts: ["centos8:5055"]		\\logstash地址

:x保存并退出

完成配置后重启filebeat，等待一会，就可以去kibana看下效果了。

注意:如果以前kibana索引添加过，请先刷新一下索引字段。
刷新后，访问一下服务器生成日志，然后查看下最近文档。如下就没有问题了。

前面添加的字段都能被正常识别了。

{
  "_index": "logstash-apache2-access-2020-06-24",
  "_type": "_doc",
  "_id": "5Ox45XIBE-2ICHvYeDbz",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-06-24T08:36:12.019Z",
    "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36 Edg/83.0.478.54",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "agent": {
      "ephemeral_id": "c74c66c6-7f6f-4026-8576-aea9d4e3cb8f",
      "type": "filebeat",
      "id": "83833263-7653-45b0-bfb0-4716e04b9d0e",
      "version": "7.5.0",
      "hostname": "wisnuc"
    },
    "geoip": {
      "ip": "183.157.11.166",
      "longitude": 120.1619,
      "latitude": 30.294,
      "country_code2": "CN",
      "country_code3": "CN",
      "location": {
        "lon": 120.1619,
        "lat": 30.294
      },
      "region_code": "ZJ",
      "city_name": "Hangzhou",
      "region_name": "Zhejiang",
      "country_name": "China",
      "timezone": "Asia/Shanghai",
      "continent_code": "AS"
    },
    "URL": "/",
    "host": {
      "name": "wisnuc"
    },
    "ecs": {
      "version": "1.1.0"
    },
    "http_referer": "-",
    "status": "200",
    "input": {
      "type": "log"
    },
    "log": {
      "file": {
        "path": "/var/log/apache2/access.log"
      },
      "offset": 6544
    },
    "request": "GET / HTTP/1.1",
    "http_vhost": "127.0.1.1:443",
    "@version": "1",
    "time": "[24/Jun/2020:16:36:11 +0800]",
    "clientip": "183.157.11.166",
    "fields": {
      "appname": "apache2-access"
    },
    "message": "183.157.11.166------[24/Jun/2020:16:36:11 +0800]-GET / HTTP/1.1-20012407",
    "remote_user": "-"
  },
  "fields": {
    "@timestamp": [
      "2020-06-24T08:36:12.019Z"
    ]
  },
  "sort": [
    1592987772019
  ]
}

参考资料

apache2.4-doc
1mod_remoteip
apache2mod_log_config