Regarding Scanning record based on datetime

Question

Have a dynamo db table which has a column called 'created' which stores datetime in epoch format which i need to scan for archival.

I need to archive the records which are more than 30 days older from a DynamoDB table. However, date time field is in epoch format.

i used below code filterkey is the column 'created' & filter value os 'epoch' value of current date and time. connection.scan_table(FilterExpression=Attr(filter_key).lte(filter_value))

when I tried that above code although records are there its returning no records. may be the scanning method is not correct because I am comparing epoch date and time . can someone help me on this. scanning table with epoch date time formt in DynamoDB?

Answer 1

Don't do scans at all. Try TTL instead.

Time To Live (TTL) for DynamoDB allows you to define when items in a table expire so that they can be automatically deleted from the database.

TTL is provided at no extra cost as a way to reduce storage usage and reduce the cost of storing irrelevant data without using provisioned throughput. With TTL enabled on a table, you can set a timestamp for deletion on a per-item basis, allowing you to limit storage usage to only those records that are relevant.

TTL is useful if you have continuously accumulating data that loses relevance after a specific time period. For example: session data, event logs, usage patterns, and other temporary data. If you have sensitive data that must be retained only for a certain amount of time according to contractual or regulatory obligations, TTL helps you ensure that it is removed promptly and as scheduled.

Seems exactly like your case. TTL should be a Number attribute with a time in epoch format (just like you have). After the TTL your records will be deleted from the database when TTL comes. You can then listen for deleted records with DynamoDB streams and archive them. Read more about enabling the TTL.

Such event based, push approach will scale much better then scans.