td-agentのインストールでハマった件
*.posファイルが存在するとtd-agentが起動できない現象に遭遇
サーバは
Amazon Linux 2014.09 (hvm)
td-agentのバージョン
[root@server1 td-agent]# td-agent --version td-agent 0.10.55 [root@server1 td-agent]# rpm -qi td-agent Name : td-agent Version : 1.1.21 Release : 0 Architecture: x86_64 Install Date: 2015年02月05日 11時51分41秒 Group : System Environment/Daemons Size : 103551538 License : APL2 Signature : DSA/SHA1, 2014年10月20日 13時07分39秒, Key ID 1093db45a12e206f Source RPM : td-agent-1.1.21-0.src.rpm Build Date : 2014年10月20日 08時31分13秒 Build Host : ip-10-123-31-198.ec2.internal Relocations : (not relocatable) Vendor : Treasure Data, Inc. URL : http://treasure-data.com/ Summary : td-agent Description :
confファイルの設定は
<source> type config_expander <config> type tail format syslog path /var/log/messages pos_file /var/log/td-agent/syslog.messages.pos tag syslog.messages </config> </source> <match *.**> type forest subtype s3 <template> s3_bucket hoge s3_region fuga s3_object_key_format %{path}%{time_slice}.%{file_extension} path ${tag}/ buffer_path /var/log/td-agent/buffer/${tag} time_slice_format %Y/%m/%d/%H/%H%M.${hostname} flush_interval 1m </template> </match> <source> type forward port 24224 </source>
td-agentが落ちてしまう時のログ
2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'config_expander' 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="config_expander" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:107:block in configure: adding source type="forward" 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:124:block in configure: adding match pattern="*.**" type="forest" 2015-02-05 12:19:11 +0000 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'forest' 2015-02-05 12:19:11 +0000 [info]: plugin/in_tail.rb:475:initialize: following tail of /var/log/messages 2015-02-05 12:19:11 +0000 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 's3' 2015-02-05 12:19:11 +0000 [info]: fluent/engine.rb:237:ensure in run: shutting down fluentd 2015-02-05 12:19:11 +0000 [info]: fluent/supervisor.rb:240:supervise: process finished code=256 2015-02-05 12:19:11 +0000 [warn]: fluent/supervisor.rb:243:supervise: process died within 1 second. exit.
gem達
[root@server1 td-agent]# ll /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/ 合計 244 drwxr-xr-x 3 root root 4096 2月 5 11:51 aws-sdk-1.52.0 drwxr-xr-x 5 root root 4096 2月 5 12:09 aws-sdk-v1-1.61.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 bson-1.11.1 drwxr-xr-x 3 root root 4096 2月 5 11:51 bson_ext-1.8.6 drwxr-xr-x 6 root root 4096 2月 5 11:51 bundler-1.3.6 drwxr-xr-x 6 root root 4096 2月 5 11:52 cool.io-1.3.0 drwxr-xr-x 7 root root 4096 2月 5 11:51 diff-lcs-1.2.5 drwxr-xr-x 5 root root 4096 2月 5 11:51 fluent-logger-0.4.9 drwxr-xr-x 4 root root 4096 2月 5 11:51 fluent-mixin-config-placeholders-0.3.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 fluent-mixin-plaintextformatter-0.2.6 drwxr-xr-x 4 root root 4096 2月 5 11:52 fluent-plugin-config-expander-0.1.5 drwxr-xr-x 5 root root 4096 2月 5 11:51 fluent-plugin-flume-0.1.1 drwxr-xr-x 4 root root 4096 2月 5 11:52 fluent-plugin-forest-0.3.0 drwxr-xr-x 5 root root 4096 2月 5 11:51 fluent-plugin-mongo-0.7.3 drwxr-xr-x 4 root root 4096 2月 5 11:51 fluent-plugin-rewrite-tag-filter-1.4.1 drwxr-xr-x 4 root root 4096 2月 5 12:09 fluent-plugin-s3-0.5.2 drwxr-xr-x 5 root root 4096 2月 5 11:51 fluent-plugin-scribe-0.10.12 drwxr-xr-x 4 root root 4096 2月 5 11:51 fluent-plugin-td-0.10.22 drwxr-xr-x 4 root root 4096 2月 5 11:51 fluent-plugin-td-monitoring-0.1.3 drwxr-xr-x 4 root root 4096 2月 5 11:51 fluent-plugin-webhdfs-0.3.1 drwxr-xr-x 7 root root 4096 2月 5 11:51 fluentd-0.10.55 drwxr-xr-x 3 root root 4096 2月 5 11:51 git-1.2.8 drwxr-xr-x 4 root root 4096 2月 5 11:51 hirb-0.7.2 drwxr-xr-x 7 root root 4096 2月 5 11:51 http_parser.rb-0.5.1 drwxr-xr-x 6 root root 4096 2月 5 11:51 httpclient-2.4.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 iobuffer-1.1.2 drwxr-xr-x 4 root root 4096 2月 5 11:51 ipaddress-0.8.0 drwxr-xr-x 6 root root 4096 2月 5 11:51 jeweler-1.6.2 drwxr-xr-x 9 root root 4096 2月 5 11:51 json-1.7.7 drwxr-xr-x 4 root root 4096 2月 5 11:51 ltsv-0.1.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 mini_portile-0.6.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 mixlib-cli-1.5.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 mixlib-config-2.1.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 mixlib-log-1.6.0 drwxr-xr-x 3 root root 4096 2月 5 11:51 mixlib-shellout-1.6.0 drwxr-xr-x 5 root root 4096 2月 5 11:51 mongo-1.8.6 drwxr-xr-x 6 root root 4096 2月 5 11:52 msgpack-0.5.11 drwxr-xr-x 7 root root 4096 2月 5 11:51 nokogiri-1.5.10 drwxr-xr-x 6 root root 4096 2月 5 11:51 ohai-6.20.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 parallel-0.6.5 drwxr-xr-x 3 root root 4096 2月 5 11:51 rake-0.9.2.2 drwxr-xr-x 3 root root 4096 2月 5 11:51 rdoc-3.9.5 drwxr-xr-x 3 root root 4096 2月 5 11:51 rspec-3.0.0 drwxr-xr-x 4 root root 4096 2月 5 11:51 rspec-core-3.0.4 drwxr-xr-x 3 root root 4096 2月 5 11:51 rspec-expectations-3.0.4 drwxr-xr-x 3 root root 4096 2月 5 11:51 rspec-mocks-3.0.4 drwxr-xr-x 3 root root 4096 2月 5 11:51 rspec-support-3.0.4 drwxr-xr-x 4 root root 4096 2月 5 11:51 rubyzip-0.9.9 drwxr-xr-x 3 root root 4096 2月 5 11:51 sigdump-0.2.2 drwxr-xr-x 5 root root 4096 2月 5 11:51 string-scrub-0.0.3 drwxr-xr-x 5 root root 4096 2月 5 11:51 systemu-2.5.2 drwxr-xr-x 9 root root 4096 2月 5 11:51 td-0.11.5 drwxr-xr-x 5 root root 4096 2月 5 11:51 td-client-0.8.66 drwxr-xr-x 5 root root 4096 2月 5 11:51 td-logger-0.3.23 drwxr-xr-x 6 root root 4096 2月 5 11:52 thread_safe-0.3.4 drwxr-xr-x 7 root root 4096 2月 5 11:51 thrift-0.8.0 drwxr-xr-x 4 root root 4096 2月 5 11:52 tzinfo-1.2.2 drwxr-xr-x 3 root root 4096 2月 5 11:52 tzinfo-data-1.2015.1 drwxr-xr-x 6 root root 4096 2月 5 11:51 uuidtools-2.1.5 drwxr-xr-x 4 root root 4096 2月 5 11:51 webhdfs-0.5.5 drwxr-xr-x 8 root root 4096 2月 5 11:51 yajl-ruby-1.1.0
ツイッターで@repeatedlyに相談した所、td-agent2を入れてみたところこの現象が出なくなった!
ざっくりな質問に真摯に対応してくださったrepeatedly に感謝です!!
今回は
wget http://packages.treasuredata.com.s3.amazonaws.com/2/redhat/7/x86_64/td-agent-2.1.3-1.x86_64.rpm rpm -ivh td-agent-2.1.3-1.x86_64.rpm /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-config-expander /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-forest service td-agent start
で無事解決