Problem with apache camel reading a file from s3

0
votes

I am implementing a route that takes a file from s3 reads it and deletes it. The problem is that after deleting it continues to search for the file and goes into error, I don't understand why the route does not stop.

This is my from

var uri = "aws-s3://$s3BucketName?amazonS3Client=#s3Client&fileName=$fileName&deleteAfterRead=true"
from(uri)

2021-02-23 23:53:23.226 WARN 2452 --- [ftp-bucket/test] o.a.camel.component.aws.s3.S3Consumer : Consumer S3Consumer[aws-s3://bucket/test?amazonS3Client=%23s3Client&deleteAfterRead=true&fileName=testAlex.CSV] failed polling endpoint: aws-s3://bucket/test?amazonS3Client=%23s3Client&deleteAfterRead=true&fileName=testAlex.CSV. Will try again at next poll. Caused by: [com.amazonaws.services.s3.model.AmazonS3Exception - The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 Extended Request ID: /=)] com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 Extended Request ID: /=) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052) ~[aws-java-sdk-s3-1.11.714.jar:na] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998) ~[aws-java-sdk-s3-1.11.714.jar:na] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486) ~[aws-java-sdk-s3-1.11.714.jar:na] at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:74) ~[camel-aws-s3-3.2.0.jar:3.2.0] at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) ~[camel-support-3.2.0.jar:3.2.0] at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106) ~[camel-support-3.2.0.jar:3.2.0] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

1
javaamazon-web-serviceskotlinamazon-s3apache-camel
Look into stack trace - com.amazonaws.services.s3.model.AmazonS3Exception - The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 – Victor Gubin
I think this happens because I delete the file. – Alessandro
I read the file correctly and delete it from s3, but it is as if after the route goes in loop giving me that error – Alessandro
Is your logic trying to find an S3 object that does not exist? – smac2020
No, the route must stop after the file is deleted – Alessandro

1 Answers

1
votes

You're using a consumer, so it will poll on period basis. You have multiple options to solve the problem:

  1. use the repeatCount option on your S3 Endpoint

    from("aws-s3://$s3BucketName amazonS3Client=#s3Client&fileName=$fileName&deleteAfterRead=true&repeatCount=1")

  2. Use timer + producer AWS-S3 on the producer side has the deleteObject operation [1]

So you can do something like:

from(timer)
to(aws-s3)

and specify the deleteObject operation required options and headers.

One important point, camel 3.2.0 was a development release. We suggest to use the last supported LTS release of 3.7.x family, which is 3.7.2.

Another point, AWS-S3 component has been deprecated in 3.8.0 and will be removed in 3.9.0, so please switch to Camel-AWS2-S3.

[1] https://github.com/apache/camel/blob/camel-3.2.0/components/camel-aws-s3/src/main/java/org/apache/camel/component/aws/s3/S3Producer.java#L98