読者です 読者をやめる 読者になる 読者になる

td-agentのインストールでハマった件

*.posファイルが存在するとtd-agentが起動できない現象に遭遇

サーバは
Amazon Linux 2014.09 (hvm)

td-agentのバージョン

[root@server1 td-agent]# td-agent --version
td-agent 0.10.55

[root@server1 td-agent]# rpm -qi td-agent
Name        : td-agent
Version     : 1.1.21
Release     : 0
Architecture: x86_64
Install Date: 20150205115141秒
Group       : System Environment/Daemons
Size        : 103551538
License     : APL2
Signature   : DSA/SHA1, 20141020130739秒, Key ID 1093db45a12e206f
Source RPM  : td-agent-1.1.21-0.src.rpm
Build Date  : 20141020083113秒
Build Host  : ip-10-123-31-198.ec2.internal
Relocations : (not relocatable)
Vendor      : Treasure Data, Inc.
URL         : http://treasure-data.com/
Summary     : td-agent
Description :

confファイルの設定は

  <source>
    type config_expander
    <config>
      type tail
      format syslog
      path /var/log/messages
      pos_file /var/log/td-agent/syslog.messages.pos
      tag syslog.messages
    </config>
  </source>

  <match *.**>
    type forest
    subtype s3
    <template>
      s3_bucket hoge
      s3_region fuga
      s3_object_key_format %{path}%{time_slice}.%{file_extension}
      path ${tag}/
      buffer_path /var/log/td-agent/buffer/${tag}
      time_slice_format %Y/%m/%d/%H/%H%M.${hostname}
      flush_interval 1m
    </template>
  </match>
  <source>
    type forward
    port 24224
  </source>

td-agentが落ちてしまう時のログ

2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'config_expander'
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="forward"
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:124:block in configure: adding match pattern="*.**" type="forest"
2015-02-05 12:19:11 +0000 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'forest'
2015-02-05 12:19:11 +0000 [info]: plugin/in_tail.rb:475:initialize: following tail of /var/log/messages
2015-02-05 12:19:11 +0000 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 's3'
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:237:ensure in run: shutting down fluentd
2015-02-05 12:19:11 +0000 [info]: fluent/supervisor.rb:240:supervise: process finished code=256
2015-02-05 12:19:11 +0000 [warn]: fluent/supervisor.rb:243:supervise: process died within 1 second. exit.

gem達

[root@server1 td-agent]# ll /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/
合計 244
drwxr-xr-x 3 root root 4096  25 11:51 aws-sdk-1.52.0
drwxr-xr-x 5 root root 4096  25 12:09 aws-sdk-v1-1.61.0
drwxr-xr-x 4 root root 4096  25 11:51 bson-1.11.1
drwxr-xr-x 3 root root 4096  25 11:51 bson_ext-1.8.6
drwxr-xr-x 6 root root 4096  25 11:51 bundler-1.3.6
drwxr-xr-x 6 root root 4096  25 11:52 cool.io-1.3.0
drwxr-xr-x 7 root root 4096  25 11:51 diff-lcs-1.2.5
drwxr-xr-x 5 root root 4096  25 11:51 fluent-logger-0.4.9
drwxr-xr-x 4 root root 4096  25 11:51 fluent-mixin-config-placeholders-0.3.0
drwxr-xr-x 4 root root 4096  25 11:51 fluent-mixin-plaintextformatter-0.2.6
drwxr-xr-x 4 root root 4096  25 11:52 fluent-plugin-config-expander-0.1.5
drwxr-xr-x 5 root root 4096  25 11:51 fluent-plugin-flume-0.1.1
drwxr-xr-x 4 root root 4096  25 11:52 fluent-plugin-forest-0.3.0
drwxr-xr-x 5 root root 4096  25 11:51 fluent-plugin-mongo-0.7.3
drwxr-xr-x 4 root root 4096  25 11:51 fluent-plugin-rewrite-tag-filter-1.4.1
drwxr-xr-x 4 root root 4096  25 12:09 fluent-plugin-s3-0.5.2
drwxr-xr-x 5 root root 4096  25 11:51 fluent-plugin-scribe-0.10.12
drwxr-xr-x 4 root root 4096  25 11:51 fluent-plugin-td-0.10.22
drwxr-xr-x 4 root root 4096  25 11:51 fluent-plugin-td-monitoring-0.1.3
drwxr-xr-x 4 root root 4096  25 11:51 fluent-plugin-webhdfs-0.3.1
drwxr-xr-x 7 root root 4096  25 11:51 fluentd-0.10.55
drwxr-xr-x 3 root root 4096  25 11:51 git-1.2.8
drwxr-xr-x 4 root root 4096  25 11:51 hirb-0.7.2
drwxr-xr-x 7 root root 4096  25 11:51 http_parser.rb-0.5.1
drwxr-xr-x 6 root root 4096  25 11:51 httpclient-2.4.0
drwxr-xr-x 4 root root 4096  25 11:51 iobuffer-1.1.2
drwxr-xr-x 4 root root 4096  25 11:51 ipaddress-0.8.0
drwxr-xr-x 6 root root 4096  25 11:51 jeweler-1.6.2
drwxr-xr-x 9 root root 4096  25 11:51 json-1.7.7
drwxr-xr-x 4 root root 4096  25 11:51 ltsv-0.1.0
drwxr-xr-x 4 root root 4096  25 11:51 mini_portile-0.6.0
drwxr-xr-x 4 root root 4096  25 11:51 mixlib-cli-1.5.0
drwxr-xr-x 4 root root 4096  25 11:51 mixlib-config-2.1.0
drwxr-xr-x 4 root root 4096  25 11:51 mixlib-log-1.6.0
drwxr-xr-x 3 root root 4096  25 11:51 mixlib-shellout-1.6.0
drwxr-xr-x 5 root root 4096  25 11:51 mongo-1.8.6
drwxr-xr-x 6 root root 4096  25 11:52 msgpack-0.5.11
drwxr-xr-x 7 root root 4096  25 11:51 nokogiri-1.5.10
drwxr-xr-x 6 root root 4096  25 11:51 ohai-6.20.0
drwxr-xr-x 4 root root 4096  25 11:51 parallel-0.6.5
drwxr-xr-x 3 root root 4096  25 11:51 rake-0.9.2.2
drwxr-xr-x 3 root root 4096  25 11:51 rdoc-3.9.5
drwxr-xr-x 3 root root 4096  25 11:51 rspec-3.0.0
drwxr-xr-x 4 root root 4096  25 11:51 rspec-core-3.0.4
drwxr-xr-x 3 root root 4096  25 11:51 rspec-expectations-3.0.4
drwxr-xr-x 3 root root 4096  25 11:51 rspec-mocks-3.0.4
drwxr-xr-x 3 root root 4096  25 11:51 rspec-support-3.0.4
drwxr-xr-x 4 root root 4096  25 11:51 rubyzip-0.9.9
drwxr-xr-x 3 root root 4096  25 11:51 sigdump-0.2.2
drwxr-xr-x 5 root root 4096  25 11:51 string-scrub-0.0.3
drwxr-xr-x 5 root root 4096  25 11:51 systemu-2.5.2
drwxr-xr-x 9 root root 4096  25 11:51 td-0.11.5
drwxr-xr-x 5 root root 4096  25 11:51 td-client-0.8.66
drwxr-xr-x 5 root root 4096  25 11:51 td-logger-0.3.23
drwxr-xr-x 6 root root 4096  25 11:52 thread_safe-0.3.4
drwxr-xr-x 7 root root 4096  25 11:51 thrift-0.8.0
drwxr-xr-x 4 root root 4096  25 11:52 tzinfo-1.2.2
drwxr-xr-x 3 root root 4096  25 11:52 tzinfo-data-1.2015.1
drwxr-xr-x 6 root root 4096  25 11:51 uuidtools-2.1.5
drwxr-xr-x 4 root root 4096  25 11:51 webhdfs-0.5.5
drwxr-xr-x 8 root root 4096  25 11:51 yajl-ruby-1.1.0

ツイッターで@repeatedlyに相談した所、td-agent2を入れてみたところこの現象が出なくなった!
ざっくりな質問に真摯に対応してくださったrepeatedly に感謝です!!

今回は

wget http://packages.treasuredata.com.s3.amazonaws.com/2/redhat/7/x86_64/td-agent-2.1.3-1.x86_64.rpm
rpm -ivh td-agent-2.1.3-1.x86_64.rpm
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-config-expander
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-forest
service td-agent start

で無事解決