java 通过tcp向logstash发送数据
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/35143576/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Sending data to logstash via tcp
提问by user1553248
I'm running into some issues sending log data to my logstash instance from a simple java application. For my use case, I'm trying to avoid using log4j logback and instead batch json events on separate lines through a raw tcp socket. The reason for that is I'm looking to send data through a aws lambda function to logstash which means storing logs to disk may not work out.
我在从一个简单的 Java 应用程序向我的 logstash 实例发送日志数据时遇到了一些问题。对于我的用例，我试图避免使用 log4j logback，而是通过原始 tcp 套接字在单独的行上批处理 json 事件。原因是我希望通过 aws lambda 函数将数据发送到 logstash，这意味着将日志存储到磁盘可能无法解决。
My logstash configuration file looks like the following:
我的 logstash 配置文件如下所示：
input {
tcp {
port => 5400
codec => json
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
host => "localhost"
protocol => "http"
index => "someIndex"
}
}
My java code right now is just opening a tcp socket to the logstash server and sending an event directly.
我现在的 java 代码只是打开一个到 logstash 服务器的 tcp 套接字并直接发送一个事件。
Socket socket = new Socket("logstash-hostname", 5400);
DataOutputStream os = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
os.writeBytes("{\"message\": {\"someField\":\"someValue\"} }");
os.flush();
socket.close();
The application is connecting to the logstash host properly (if logstash is not up an exception is thrown when connecting), but no events are showing up in our ES cluster. Any ideas on how to do this are greatly appreciated!
应用程序正确连接到logstash 主机（如果logstash 未启动，连接时会抛出异常），但我们的ES 集群中没有显示任何事件。非常感谢有关如何执行此操作的任何想法！
I don't see any relevant logs in logstash.err, logstash.log, or logstash.stdout pointing to what may be going wrong.
我在 logstash.err、logstash.log 或 logstash.stdout 中没有看到任何相关日志指出可能出错的地方。
回答by alfredocambera
The problem is that your data is already deserialized on your input and you are trying to deserialize it again on your filter. Simply remove the json filter.
问题是您的数据已经在您的输入上反序列化，而您正试图在您的过滤器上再次反序列化它。只需删除 json 过滤器。
Here is how I recreated your scenario:
这是我重新创建您的场景的方式：
# the json input
root@monitoring:~# cat tmp.json
{"message":{"someField":"someValue"}}
# the logstash configuration file
root@monitoring:~# cat /etc/logstash/conf.d/test.conf
input {
tcp {
port => 5400
codec => json
}
}
filter{
}
output {
stdout {
codec => rubydebug
}
}
# starting the logstash server
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf
# sending the json to logstash with netcat
nc localhost 5400 < tmp.json
# the logstash output via stdout
{
"message" => {
"someField" => "someValue"
},
"@version" => "1",
"@timestamp" => "2016-02-02T13:31:18.703Z",
"host" => "0:0:0:0:0:0:0:1",
"port" => 56812
}
Hope it helps,
希望能帮助到你，
回答by Dmytro Pashko
Don't forget to append \nat the end of your JSON:
不要忘记\n在 JSON 的末尾附加：
os.writeBytes("{\"bossMessage\": {\"someField\":\"someValue\"} }" + '\n');
回答by JakesIV
I found that when using the codec on the TCP Input node then the JSON parsing fails on unknown reason. Removing the codec on TCP input and adding filter only resolved my JSON parsing issues.
我发现在 TCP Input 节点上使用编解码器时，JSON 解析因未知原因失败。删除 TCP 输入上的编解码器并添加过滤器仅解决了我的 JSON 解析问题。
input {
tcp {
port => 5400
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
host => "localhost"
protocol => "http"
index => "someIndex"
}
}
Also I write the whole string at once and do not send the "\n" in the message else Logstash gets an empty record with code "\r" only.
此外，我一次写入整个字符串，并且不会在消息中发送“\n”，否则 Logstash 会获得仅带有代码“\r”的空记录。
PrintWriter pw = new PrintWriter(os);
pw.println(stringOut);
pw.flush();
This will append "\r" to the end of the stream.
这会将“\r”附加到流的末尾。
