0
votes

We are trying to use a simple user-defined function (UDF) in the where clause of a query in Azure Cosmos DB, but it's not working correctly. The end goal is to query all results where the timestamp _ts is greater than or equal to yesterday, but our first step is to get a UDF working.

The abbreviated data looks like this:

[
    {
        "_ts": 1500000007
    }
    {
        "_ts": 1500000005
    }
]

Using the Azure Portal's Cosmos DB Data Explorer, a simple query like the following will correctly return one result:

SELECT * FROM c WHERE c._ts >= 1500000006

A simple query using our UDF incorrectly returns zero results. This query is below:

SELECT * FROM c WHERE c._ts >= udf.getHardcodedTime()

The definition of the function is below:

function getHardcodedTime(){
    return 1500000006;
}

And here is a screenshot of the UDF for verification:

Screenshot of UDF in Azure Portal

As you can see, the only difference between the two queries is that one query uses a hard-coded value while the other query uses a UDF to get a hard-coded value. The problem is that the query using the UDF returns zero results instead of returning one result.

Are we using the UDF correctly?

Update 1 When the UDF is updated to return the number 1, then we get a different count of results each time.

New function:

function getHardcodedTime(){
    return 1;
}

New query: SELECT count(1) FROM c WHERE c._ts >= udf.getHardcodedTime()

Results vary with 7240, 7236, 7233, 7264, etc. (This set is the actual order of responses from Cosmos DB.)

1
I don't see problems with the use of udf, and I cannot reproduce the problem with Cosmos DB Emulator. What would you get if you run SELECT udf.getHardcodedTime() FROM c?Chun Liu
I get many objects returned where the object contains the valuefrom the function. Here is a snippet. [ {"$1": 1500000006 }, ... { "$1": 1500000006} ]Toby Artisan
Could it be that you get continuation token in the select-using-udf case? If cosmosDB does not deduce your udf is deterministic then it can no longer use an index and hence it would do a full scan.Imre Pühvel
Is there a way to indicate to Cosmos DB that the function is deterministic? I found some examples on how to do that with SQL databases, but I didn't find anything for Cosmos DB.Toby Artisan
I don't think you can. But you can work around this issue by using SP, taking parameter from UDF and passing it to query as value. cumbersome, but should work.Imre Pühvel

1 Answers

1
votes

By your description, the most likely cause is that the UDF version is slow and returns a partial result with continuation token instead of final result.

The concept of continuation is explained here as:

If a query's results cannot fit within a single page of results, then the REST API returns a continuation token through the x-ms-continuation-token response header. Clients can paginate results by including the header in subsequent results.

The observed count variation could be caused by the query stopping for next page at slightly different times. Check the x-ms-continuation header to know if that's the case.

Why is UDF slow?

UDF returning a constant is not the same as constant for the query. Correct me if you know better, but from CosmosDB side, it does not know that the specific UDF is actually deterministic, but assumes it could evaluate to any value and hence has to be executed for each document to check for a match. It means it cannot use an index and has to do a slow full scan.

What can you do?

Option 1: Follow continuations

If you don't care about the performance, you could keep using the UDF and just follow the continuation until all rows are processed.

Some DocumentDB clients can do this for you (ex: .net API), so this may be the fastest fix if you are in a hurry. But beware, this does not scale (keeps getting slower and cost more and more RU) and you should not consider this a long-term solution.

Option 2: Drop UDF, use parameters

You could pass the hardcodedTime instead as parameter. This way the query execution engine would know the value, could use a matching index and give you correct results without the hassle of continuations.

I don't know which API you use, but related reading just in case: Parameterized SQL queries.

Option 3: Wrap in stored proc

If you really-really must control the hardcodedTime in UDF then you could implement a server-side procedure which would:

  1. query the hardcodedTime from UDF
  2. query the documents, passing the hardcodedTime as parameter
  3. return results as SP output.

It would use the UDF AND index, but brings a hefty overhead in the amount of code required. Do your math if keeping UDF is worth the extra effort in dev and maintenance.

Related documentation about SPs: Azure Cosmos DB server-side programming: Stored procedures, database triggers, and UDFs