AWS DynamoDB - Boto3 get all attributes, fieldnames, column headers from a dynamoDB table / structure

Question

Best

At this moment I'm using Boto3 in python 2.7 and what I would like to have is: The column headers of my specific DynamoDB table.

At this moment,I'm dealing with a very large dynamoDB table, with 80 columns and + 1.00O.000 records. And the task which I've is, to manipulate these data. To do this, I'm making use of chunks. This means that I'm retrieving each time 1000 rows, from my data-table and manipulate them and write the new result to a csv. (This is required, by some reasons).

But because I'm using chunks, it can be that not every chunk contains 80 columns, this means that it sometimes can contain 79 or 78 columns. This happens when there are no values available for a specific column in a chunk.

And this isn't desirable because at the end of the day, all those csv's should been concatenated again to each other, and therefore, each csv should contain the equal amount of columns.

Thus my idea, which I've is : Add empty columns to the chunk-csv's who doesn't contain all the required columns.

But therefore, I've to know, what the headers, attributes, field names are from my table (or the structure) + The thing is, those column headers are dynamic, there can't be a static list of headers And it can be, that suddenly new records are added with a unique column, (which means that the next time, I would receive 81 columns for each of my csv's) - Thus those header knowledge should come from my table / amazon aws

Kind regards

Answer 1

You can get some of the info you are looking for by using the attribute_definitions attribute of a Table object, like this:

import boto3
ddb = boto3.resource('dynamodb')
table = ddb.Table('MyTable')
attrs = table.attribute_definitions

The variable attrs would now contain a dictionary of all of the attributes you explicitly defined when creating the table which normally is only the attributes that are used as keys in some index.

However, since DynamoDB is schema less you can store any combination of other attributes in an item in DynamoDB. So, as the comment above states, the only way to know all attributes used in all items is to iterate through all of the items and build a set of attributes found in each item.