在 Java 中使用 Lambda 函数的 AWS S3 事件通知
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/41235593/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
AWS S3 Event notification using Lambda function in Java
提问by NGR
I am trying to use Lambda function for S3 Put event notification. My Lambda function should be called once I put/add any new JSON file in my S3 bucket. The challenge I have is there are not enough documents for this to implement such Lambda function in Java. Most of doc I found are for Node.js
我正在尝试将 Lambda 函数用于 S3 Put 事件通知。一旦我在 S3 存储桶中放置/添加任何新的 JSON 文件，就应该调用我的 Lambda 函数。我面临的挑战是没有足够的文档来在 Java 中实现这样的 Lambda 函数。我发现的大部分文档都是针对 Node.js
I want, my Lambda function should be called and then inside that Lambda function, I want to consume that added json and then send that JSON to AWS ES Service.
我想，应该调用我的 Lambda 函数，然后在该 Lambda 函数中，我想使用添加的 json，然后将该 JSON 发送到 AWS ES 服务。
But what all classes I should use for this? Anyone has any idea about this? S3 abd ES are all setup and running. The auto generated code for lambda is `
但是我应该为此使用什么所有类？有人对此有任何想法吗？S3 abd ES 都已设置并正在运行。lambda 的自动生成代码是`
@Override
public Object handleRequest(S3Event input, Context context) {
context.getLogger().log("Input: " + input);
// TODO: implement your handler
return null;
}
What next??
接下来是什么？？
采纳答案by NGR
Finally, here are the steps for S3 --> Lambda --> ES integration using Java.
最后，这里是 S3 --> Lambda --> ES 使用 Java 集成的步骤。
- Have your S3, Lamba and ES created on AWS. Steps are here.
Use below Java code in your lambda function to fetch a newly added object in S3 and send it to ES service.
public Object handleRequest(S3Event input, Context context) { AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain()); for (S3EventNotificationRecord record : input.getRecords()) { String s3Key = record.getS3().getObject().getKey(); String s3Bucket = record.getS3().getBucket().getName(); context.getLogger().log("found id: " + s3Bucket+" "+s3Key); // retrieve s3 object S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key)); InputStream objectData = object.getObjectContent(); //Start putting your objects in AWS ES Service String esInput = "Build your JSON string here using S3 objectData"; HttpClient httpClient = new DefaultHttpClient(); HttpPut putRequest = new HttpPut(AWS_ES_ENDPOINT + "/{Index_name}/{product_name}/{unique_id}" ); StringEntity input = new StringEntity(esInput); input.setContentType("application/json"); putRequest.setEntity(input); httpClient.execute(putRequest); httpClient.getConnectionManager().shutdown(); } return "success";}Use either Postman or Sense to create Actual index & corresponding mapping in ES.
Once done, download and run proxy.json your machine. Make sure you setup ES Security steps suggested in this post
Test setup and Kibana by running http://localhost:9200/_plugin/kibana/URL from your machine.
All is set. Go ahead and set your dashboard in Kibana. Test it by adding new objects in your S3 bucket
- 在 AWS 上创建您的 S3、Lamba 和 ES。步骤在这里。
在 lambda 函数中使用以下 Java 代码来获取 S3 中新添加的对象并将其发送到 ES 服务。
public Object handleRequest(S3Event input, Context context) { AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain()); for (S3EventNotificationRecord record : input.getRecords()) { String s3Key = record.getS3().getObject().getKey(); String s3Bucket = record.getS3().getBucket().getName(); context.getLogger().log("found id: " + s3Bucket+" "+s3Key); // retrieve s3 object S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key)); InputStream objectData = object.getObjectContent(); //Start putting your objects in AWS ES Service String esInput = "Build your JSON string here using S3 objectData"; HttpClient httpClient = new DefaultHttpClient(); HttpPut putRequest = new HttpPut(AWS_ES_ENDPOINT + "/{Index_name}/{product_name}/{unique_id}" ); StringEntity input = new StringEntity(esInput); input.setContentType("application/json"); putRequest.setEntity(input); httpClient.execute(putRequest); httpClient.getConnectionManager().shutdown(); } return "success";}使用 Postman 或 Sense 在 ES 中创建实际索引和相应的映射。
通过从您的机器运行http://localhost:9200/_plugin/kibana/URL 来测试设置和 Kibana 。
一切就绪。继续在 Kibana 中设置您的仪表板。通过在 S3 存储桶中添加新对象来测试它
回答by jens walter
Handling S3 events in Lambda can be done, but you have to keep in mind, the the S3Event object only transports the reference to the object and not the object itself. To get to the actual object you have to invoke the AWS SDK yourself. Requesting a S3 Object within a lambda function would look like this:
可以在 Lambda 中处理 S3 事件，但您必须记住，S3Event 对象仅传输对对象的引用，而不是对象本身。要获取实际对象，您必须自己调用 AWS 开发工具包。在 lambda 函数中请求 S3 对象如下所示：
public Object handleRequest(S3Event input, Context context) {
AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain());
for (S3EventNotificationRecord record : input.getRecords()) {
String s3Key = record.getS3().getObject().getKey();
String s3Bucket = record.getS3().getBucket().getName();
context.getLogger().log("found id: " + s3Bucket+" "+s3Key);
// retrieve s3 object
S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key));
InputStream objectData = object.getObjectContent();
//insert object into elasticsearch
}
return null;
}
Now the rather difficult part to insert this object into ElasticSearch. Sadly the AWS SDK does not provide any functions for this. The default approach would be to do a REST call against the AWS ES endpoint. There are various samples out their on how to proceed with calling an ElasticSearch instance.
现在是将这个对象插入到 ElasticSearch 中相当困难的部分。遗憾的是，AWS SDK 没有为此提供任何功能。默认方法是对 AWS ES 端点进行 REST 调用。有关于如何继续调用 ElasticSearch 实例的各种示例。
Some people seem to go with the following project:
有些人似乎采用以下项目：
