ElasticSearch backup to S3 AWS

Question

I'm trying to backup my cluster of ElasticSearch on AWS to an S3 bucket.

I've followed the following 'tutorial' : Use Amazon S3 to Store a Single Amazon Elasticsearch Service Index

These are the steps I have taken:

Create a S3 bucket (called cb-search-es-backup).

Create a new policy (called P_ES_SNAPSHOT_TO_S3):

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::cb-search-es-backup"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::cb-search-es-backup/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Create a Service Role, attached the previous created policy to it arn:aws:iam::12345678910:role/Role_ES_TO_S3

Trust policy of the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

In Kibana I've tried to use the following in the Dev Tools:

PUT /_snapshot/ES_BACKUP
{
"type": "s3",
    "settings": {
        "bucket": "cb-search-es-backup",
        "region": "eu-west-1",
        "role_arn": "arn:aws:iam::423628447134:role/Role_ES_TO_S3"
    }
}

but I receive the following response from kibana:

{ "Message": "User: anonymous is not authorized to perform: iam:PassRole on resource: arn:aws:iam::12345678910:role/Role_ES_TO_S3" }

AeonM AeonM · Accepted Answer · 2018-08-06T06:04:33

I've got same error with kibana. Try to backup with aws-es-proxy.

If you are going to use docker image, please pass the AWS access credentials as below:

docker run --rm -it -p 9200:9200 -e "AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXX" -e "AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYYYYYYYYYYYYYYY" abutaha/aws-es-proxy ./aws-es-proxy -verbose -listen 0.0.0.0:9200 -endpoint https://enpoint-url

ElasticSearch backup to S3 AWS

1 Answers