DynamoDB ScanFIlter with BETWEEN comparator

Question

I have a requirement whereby I need to scan a DynamoDB table using a filter that allows me to filter documents on the table based on a Date field. Specifically, the filter needs to return the documents whose lastAccessTime (not a hash or range key) field is between the current scan and the last scan. The field is defined, on DynamoDB, as a string and in my Java code I build the condition in the following way:

Condition betweenDateCondition = new Condition()
.withComparisonOperator(ComparisonOperator.BETWEEN.toString())
.withAttributeValueList(
    new AttributeValue().withS(dateFormatter.print(getLastScanTime())),
    new AttributeValue().withS(dateFormatter.print(getCurrentScanTime())));

Map<String, Condition> conditions = new HashMap<String, Condition>();
    keyConditions.put("lastAccessTime", betweenDateCondition);

ScanRequest scanRequest = new ScanRequest()
    .withTableName("myTable")
    .withScanFilter(conditions)
    .withLimit(itemLimit)   
    .withReturnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
    .withExclusiveStartKey(exclusiveStartKey)
    .withTotalSegments(totalSegments)
    .withSegment(segment);

ScanResult scanResult = ddbClient.scan(scanRequest);

When I execute this, however, I notice that the filter does not seem to work in that scan operations a few hours apart return the same data. Obviously, data on the DynamoDB table is added only, so I'd expect to receive only the new additions.

Documentation does really spend an incredible amount of words on the use of comparison operators, especially ComparisonOperator.BETWEEN and string dates, and I haven't found similar problems around.

1) did you try a more simple scenario without segments / limits / start key? 2) are you sure the attribute is named lastAccessTime ? — Chen Harel
Hi and thanks. Filters do not seem to work regardless the use of parallel scan/limits/start key, in my first version I tried a pretty basic scan and had identical results. I also can confirm the attribute name is correct. — Diferdin
another thing that pops to mind, are you sure the date is actually formatted in the same way you are conditioning it? (i.e. milliseconds since epoch / human readable string) — Chen Harel
Hi, I think dates are formatted in the same way, but is there a format dates should be adhering to in order for BETWEEN to work? — Diferdin
I personally use milliseconds since epoch as my raw database representations — Chen Harel

Geshfaci Geshfaci · Accepted Answer · 2014-12-18T19:23:39

Map<String, Condition> conditions = new HashMap<String, Condition>();
keyConditions.put("lastAccessTime", betweenDateCondition);

Seems like you're accidentally modifying a different Map then the one you actually want, not sure if that's the issue or this was just a typo when you posted this to StackOverflow.

DynamoDB ScanFIlter with BETWEEN comparator

2 Answers