aws cli dynamo db (ValidationException) Error

Question

I'm looking to batch write item to dynamodb using python's boto3 module and i'm getting this. This is the first time i've ever worked with aws cli or boto3. The documentation says validation exception errors occur when there are empty values and possible incorrect data types, but i've played with all those and it doesn't seem to be working.

Does dynamodb only like to have 25 items written to it at a time? how can i control those batches if so?

My request:

client = boto3.client('dynamodb')
response = client.batch_write_item(RequestItems=batch_dict)

Top of batch_dict:

{'scraper_exact_urls': [{'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},
 {'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},....

Schema:

attributes: "pps_id"=>\Aws\DynamoDb\Enum\Type::NUMBER, "sku"=>\Aws\DynamoDb\Enum\Type::STRING, "scraper_class_name"=>\Aws\DynamoDb\Enum\Type::STRING, "store_id"=>\Aws\DynamoDb\Enum\Type::NUMBER, "updated_url"=>\Aws\DynamoDb\Enum\Type::STRING, "updated_by"=>\Aws\DynamoDb\Enum\Type::STRING, "updated_on"=>\Aws\DynamoDb\Enum\Type::NUMBER, fields: "pps_id", "scraper_class_name",

The Error:

ClientError: An error occurred (ValidationException) when calling the    BatchWriteItem operation: 1 validation error detected: Value .... Map value   must satisfy constraint: [Member must have length less than or equal to 25,   Member must have length greater than or equal to 1]

Answer 1

The BatchWriteItem API works on 25 items at a time. You could use the following code, adapted from the non-copying batching question, to call BatchWriteItem on 25 item chunks

def batch(iterable, n=1):
    l = len(iterable)
    for ndx in range(0, l, n):
        yield iterable[ndx:min(ndx + n, l)]

client = boto3.client('dynamodb')

for x in batch(batch_dict['scraper_exact_urls'], 25):
    subbatch_dict = {'scraper_exact_urls': x}
    response = client.batch_write_item(RequestItems=subbatch_dict)

Answer 2

And here is the Javascript version of batch writing/deleting many items, in case someone needs it:

const batchWriteManyItems = async (tableName, itemObjs, chunkSize = 25) => {

        const buildParams = (table) => JSON.parse(`{"RequestItems": {"${table}": []}}`)

        const queryChunk = (arr, size) => {
            const tempArr = []
            for (let i = 0, len = arr.length; i < len; i += size) {
                tempArr.push(arr.slice(i, i + size));
            }

            return tempArr
        }

        await Promise.all(queryChunk(itemObjs, chunkSize).map(async chunk => {
            let itemParams = buildParams(tableName);
            itemParams.RequestItems[tableName] = chunk
            await dynamoDB.batchWriteItem(itemParams).promise()
        }))
    }

NOTE: the itemObjs attribute can take {PutRequest: {Item: ...} for write items and {DeleteRequest: {Item: ...} to delete items