环境

  • ubuntu 16.04
  • rails: 5.0.2
  • ruby: 2.3.3
  • docker: 17.03.1-ce
  • docker-compose: 1.11.2
  • ELK: 5.4.0

之前有记录过一篇如何安装和使用Elasticstack的日志,整个安装步骤过于繁琐,这次尝试使用docker构建的ELK组合来收集管理Rails的日志。

安装docker和docker-compose

# 安装依赖
apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
# 导入官方GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# 添加官方源
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
# 更新源
apt-get update
# 安装docker
apt-get install docker-ce

# 安装docker-compose(可能需要管理员权限下载安装)
curl -L https://github.com/docker/compose/releases/download/1.11.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
# 设置执行权限
chmod +x /usr/local/bin/docker-compose

构建ELK

镜像基于 sebp/elk 来构建,在此基础上修改Logstash的input配置,并开放12201/udp端口用来接收日志数据。

镜像名为 selk

# 创建镜像目录
mkdir ~/workspace/selk
cd ~/workspace/selk

创建文件: Dockerfile_selk

FROM sebp/elk

ADD ./03-input.conf /etc/logstash/conf.d/03-input.conf
ADD ./30-output.conf /etc/logstash/conf.d/30-output.conf

EXPOSE 12201/udp

创建: 03-input.conf

input {
  udp {
    host => "0.0.0.0"
      port => 12201
      codec => json_lines
  }
}

创建: 30-output.conf

output {
  elasticsearch {
    hosts => ["localhost"]
      codec => json_lines
  }
  stdout {
    codec => json_lines
  }
}

创建文件 Dockerfile_web, 用来使用nginx转发kibana的管理面板,实现认证访问:

FROM nginx

RUN apt-get update -qq && apt-get -y install apache2-utils
RUN mkdir -p /usr/data

ADD ./default.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

创建 default.conf 用来设置nginx转发规则:

upstream kibana {
  server selk:5601;
  keepalive 64;
}
server {
  listen 80;
  server_name <ip>;
  location / {
    auth_basic "kibana_secret";
    auth_basic_user_file /usr/data/passwords;
    proxy_pass http://kibana;
    proxy_redirect off;
    proxy_buffering off;

    proxy_http_version 1.1;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";
  }
}

创建认证文件,实现kibana的认证访问:

# 此目录将被映射到容器内的/usr/data目录
mkdir -p ~/docker/selk_web
# 创建user1的密码
htpasswd -c passwords user1

创建: docker-compose.yml, 用来管理elk和nginx服务,并将 /var/lib/elasticsearch 目录挂载到 elk-data 数据卷

version: '2'

volumes:
  elk-data: {}

services:
  selk:
    image: bindiry/selk
    ports:
      # 这里将9200端口暴露给宿主机是为了定期删除任务
      - "127.0.0.1:9200:9200"
      # 这里可以为宿主机端口绑定到内网ip, 如 "192.168.0.1:12201:12201/udp"
      - "12201:12201/udp"
    volumes:
      - elk-data:/var/lib/elasticsearch

  web:
    image: bindiry/selk_web
    ports:
      - "80:80"
    # 这里使用links的方式连接elk服务,并使nginx能正常转发请求到kibana
    links:
      - selk:selk
    volumes:
      - ~/docker/selk_web:/usr/data

# 构建镜像
docker build -f Dockerfile_selk -t bindiry/selk .
docker build -f Dockerfile_web -t bindiry/selk_web .
# 根据 docker-compose 配置启用容器
docker-compose up
# 后台启动容器
docker-compose up -d

使用浏览器打开 http://<ip> 查看Kibana是否工作正常。

发布image到hub.docker.com

docker push bindiry/selk

使用logstasher将Rails应用日志提交到Logstash

Gemfile增加:

gem 'logstasher'

安装gem:

bundle install

修改配置文件 config/environments/[environment].rb, 增加:

config.logstasher.enabled = true
# Each of the following lines are optional. If you want to selectively disable log subscribers.
config.logstasher.controller_enabled = true
config.logstasher.mailer_enabled = false
config.logstasher.record_enabled = true
config.logstasher.view_enabled = false
config.logstasher.job_enabled = true
# This line is optional if you do not want to suppress app logs in your <environment>.log
config.logstasher.suppress_app_log = false
# This line is optional, it allows you to set a custom value for the @source field of the log event
config.logstasher.source = Rails.env
# This line is optional if you do not want to log the backtrace of exceptions
config.logstasher.backtrace = false
# This line is optional, defaults to log/logstasher_<environment>.log
config.logstasher.logger_path = "log/logstasher_#{Rails.env}.log"

安装socat,配合tail,实时将产生的日志通过udp发送给处于内网中的ELK服务器

apt-get install socat

后台运行

tail -F /home/deploy/app/current/log/logstasher_production.log | socat STDIN UDP-SENDTO:<内网ip>:12201 2>&1 &

使用lograge将Rails应用日志提交到Logstash

此方法经测试,在production环境中会出现日志不能通过 logstash-logger 发送的情况。

官方说明:

Logging eventually stops in production

This is most likely not a problem with LogStashLogger, but rather a different gem changing the log level of Rails.logger. This is especially likely if you’re using a threaded server such as Puma, since gems often change the log level of Rails.logger in a non thread-safe way. See #17 for more information.

相关讨论: Logging Suddenly Stops on Production

Gemfile增加如下gems:

gem "lograge"
gem 'logstash-event'
gem 'logstash-logger'

安装gems:

bundle install

修改开发环境配置文件 config/environments/[environment].rb

# 开启lograge
config.lograge.enabled = true
# 配置自定义日志数据
config.lograge.custom_options = lambda do |event|
  exceptions = %w(controller action format id)
  {
    params:      event.payload[:params].except(*exceptions).to_json,
    type:        :rails,
    environment: Rails.env,
    request_ip: event.payload[:request_ip]
  }
end
# 设置日志格式
config.lograge.formatter = Lograge::Formatters::Logstash.new
# 利用LogStashLogger将日志数据发送到指定端口
config.lograge.logger = LogStashLogger.new uri: 'udp://localhost:12201'

修改 app/controllers/application_controller.rb,覆写 append_info_to_payload方法,帮助自定义日志数据配置获取客户端IP

def append_info_to_payload(payload)
  super
  payload[:request_ip] = request.ip
end

定期清理旧日志索引

# 安装 Elasticsearch Curator 5.0.4
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elastic.co/curator/5/debian stable main' > /etc/apt/sources.list.d/curator.list
apt-get update && apt-get install elasticsearch-curator
# 创建配置文件
mkdir -p ~/.curator
touch ~/.curator/curator.yml
# 创建action配置文件,用于定时删除45天前的索引 delete_indices.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 45

# 测试删除
curator delete_indices.yml
# 将命令加入cron,每天凌晨20分执行
20 0 * * * curator /root/delete_indices.yml

其他常用命令

# 查看指定容器的详细信息
docker inspect [容器名/id]
# 查看已创建的数据卷
docker volume ls
# 回收空间(小心使用)
docker system prune

常见问题

  • mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu–vg-root)]], net usable_space [xxx gb], net total_space [xxxgb], spins?

遇到这个错误需要设置一下max_map_count的值(via: elasticsearch #111):

  • 修改 /etc/sysctl.conf 文件,加入 vm.max_map_count=262144
  • 或者执行命令: sysctl -w vm.max_map_count=262144

参考