How to use Google Datastore CompositeFilter with Dataflow?

Question

When reading data from Datastore as input for a pipeline I am able to create a "simple" query, where I set a filter on a single property like so:

Pipeline p = ...
Filter filter = Filter.newBuilder()
        .setPropertyFilter(PropertyFilter.newBuilder()
                .setProperty(PropertyReference.newBuilder()
                        .setName("propertyA"))
                .setOp(PropertyFilter.Operator.EQUAL)
                .setValue(Value.newBuilder().setStringValue("valueA").build())
                .build())
        .build();

Query query = Query.newBuilder()
    .addKind(KindExpression.newBuilder().setName("myKind").build())
                .setFilter(filter)
                .build();
p.apply("read", DatastoreIO.v1().read().withProjectId("myProjectId")
                .withNamespace("myNamespace").withQuery(query)).apply(.....

When I tried to apply multiple Filters on the query by concatenating the "setFilter()"-calls, only the last set Filter was applied to the query.

Upon some research I found a CompositeFilter which is supposed to enable the combination of multiple filters. I can build a composite filter, but when i want to set it on the query as a filter, the IDE complains that the types don't match and there doesn't seem to be another method to apply filters.

I managed to use a query with multiple filters by using GQL and can see in the logs that it is transformed into a CompositeFilter. I am not here to complain as the usage of GQL is much nicer than building the queries/filters by hand, but wanted to ask how the composite filter of the library WOULD be used in the context of DatastoreIO.v1().read(....) or if it isn't possible.

I am using com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-all/2.2.0

Thank you for your help.

Can you edit the post, to include the CompositeFilter that you have tried (and the IDE warns you about)? — amport
Holy **** I just saw a method on Filter.Builder thats called "setCompositeFilter". Thats going to be the way to go. Then you can set that filter on Query.Builder... Like stated above, I am now using GQL and am happy with that solution. but nice to know now how it would be done btw. the CompositeFilter is from com.google.datastore.v1.CompositeFilter — user2122552
According to the docs, you can. You can use CompositeFilter newBuilder method, and create a filter like the following: Filter composeFilter = Filter.newBuilder() .setCompositeFilter(CompositeFilter.newBuilder().addFilters(filter1).addFilters(filter2).build()).build(); Filter1 and Filter2 here are filters like the ones you have created. My IDE allows me to do this (I'm using IntelliJ IDEA, last version). — amport
Yes like I said before, I didn't see that method :-) Thanky anyways! If you want you can answer and I will accept it — user2122552

amport amport · Accepted Answer · 2018-06-25T16:00:41

According to the docs, you can. You can use CompositeFilter newBuilder method, and create a filter like the following:

Filter composeFilter = Filter.newBuilder() .setCompositeFilter(CompositeFilter.newBuilder().addFilters(filter1).addFilters(filter2).build()).build();

Filter1 and Filter2 here are filters like the ones you have created. My IDE allows me to do this (I'm using IntelliJ IDEA, last version).

How to use Google Datastore CompositeFilter with Dataflow?

1 Answers