Get all the table items from DynamoDB table using Java High Level API

Question

I implemented scan operation using in dynamodb table using dynamodbmapper, but I'm not getting all the results. Scan returns different number of items, whenever I run my program.

Code snippet :

DyanmoDBScanExpression scanExpression = new DynamoDBScanExpression();
List<Books> scanResult = mapper.scan(Books.class, scanExpression);

I investigated into it, and found out about the limit of the items scan returns. But I couln't find a way to get all the items from the table using mapper! Is there a way so I can loop through all the items of the table. I have set enough heap memory in JVM so there won't be memory issues.

Answer 1

In java use the DynamoDBScanExpression without any filter,

// Change to your Table_Name (you can load dynamically from lambda env as well)
DynamoDBMapperConfig mapperConfig = new DynamoDBMapperConfig.Builder().withTableNameOverride(DynamoDBMapperConfig.TableNameOverride.withTableNameReplacement("Table_Name")).build();

DynamoDBMapper mapper = new DynamoDBMapper(client, mapperConfig);

DynamoDBScanExpression scanExpression = new DynamoDBScanExpression();

// Change to your model class   
List < ParticipantReport > scanResult = mapper.scan(ParticipantReport.class, scanExpression);



// Check the count and iterate the list and perform as desired.
scanResult.size();

Answer 2

the scan should return all the items.
the catch is that the collection returned is lazily loaded. you need to iterate through the List and when it consumes all the items that are fetched additional calls will be made behind the scenes to bring in more items (until everything is brought in).

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/JavaQueryScanORMModelExample.html

In that example it's:

List<Book> scanResult = mapper.scan(Book.class, scanExpression);

for (Book book : scanResult) {
    System.out.println(book);
}

Answer 3

You need to iterate until LastEvaluatedKey is no longer returned. Check how is done in one of the official examples from the SDK:

https://github.com/awslabs/aws-dynamodb-examples/blob/23837f36944f4166c56988452475edee99868166/src/main/java/com/amazonaws/codesamples/lowlevel/LowLevelQuery.java#L70

Answer 4

A little bit late, but

import java.util.HashMap;
import java.util.Map;
import java.util.List;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBQueryExpression;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;

public final class LogFetcher {

    static AmazonDynamoDBClient client = new AmazonDynamoDBClient();
    static String tableName = "SystemLog";

    public static List<SystemLog> findLogsForDeviceWithMacID(String macID) {

        client.setRegion(Region.getRegion(Regions.EU_WEST_1));

        DynamoDBMapper mapper = new DynamoDBMapper(client);
        Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
        eav.put(":val1", new AttributeValue().withS(macID));

        DynamoDBQueryExpression<SystemLog> queryExpression = new DynamoDBQueryExpression<SystemLog>()
                .withKeyConditionExpression("parentKey = :val1")
                .withExpressionAttributeValues(eav);

         List<SystemLog> requestedLogs = mapper.query(SystemLog.class, queryExpression);

        return requestedLogs;
    }
}

And sample class

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBAttribute;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBRangeKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;

@DynamoDBTable(tableName="SystemLog")
public final class SystemLog {
    public Integer pidValue;
    public String uniqueId;
    public String parentKey;

    //DynamoDB

    //Partition (hash) key
    @DynamoDBHashKey(attributeName="parentKey")
    public String getParentKey() { return parentKey; }
    public void setParentKey(String parentKey) { this.parentKey = parentKey; }

    //Range key
    @DynamoDBRangeKey(attributeName="uniqueId")
    public String getUniqueId() { return uniqueId; }
    public void setUniqueId(String uniqueId) { this.uniqueId = uniqueId;} 

    @DynamoDBAttribute(attributeName="pidValue")
    public Integer getPidValue() { return pidValue; }    
    public void setPidValue(Integer pidValue) { this.pidValue = pidValue; }
}

Answer 5

By default, the DynamoDBMapper#scan method returns a "lazy-loaded" collection. It initially returns only one page of results, and then makes a service call for the next page if needed. To obtain all the matching items, iterate over the paginated results collection.

However, PaginatedScanList comes with out of box PaginatedScanList#loadAllResults method which helps to eagerly load all results for this list.

NOTE: loadAllResults method is not supported in ITERATION_ONLY mode.

List<Books> scanResult = mapper.scan(Books.class, new DynamoDBScanExpression());
scanResult.loadAllResults();//Eagerly loads all results for this list.
//Total results loaded into the list
System.out.println(scanResult.size());