ndb not querying by date correctly

Question

I'm unit testing a method on the Expense model. The relevant parts of the model look like this:

class Expense(ndb.Model):

    user = ndb.KeyProperty(User, required=True)
    timestamp = ndb.DateTimeProperty(required=True)
    ...

    @staticmethod
    def new(user, amount, comment, description, timestamp):
        """creates a new expense and stores it. It returns the Expense instance"""
        ... validation ...
        exp = Expense(user=user, timestamp=timestamp, ...)
        exp.put()
        return exp

    @staticmethod
    def list(user, from_date=None, to_date=None):
        ...  
        query = Expense.query(Expense.user == user)
        if from_date is not None:
            if not isinstance(from_date, datetime.datetime):
                raise ValueError("from_date must be a datetime.datetime instance")
            query.filter(Expense.timestamp >= from_date)
        if to_date is not None:
            if not isinstance(to_date, datetime.datetime):
                raise ValueError("to_date must be a datetime.datetime instance")
            query.filter(Expense.timestamp <= to_date)
        query.order(- Expense.timestamp)
        return query

I have the following unit test to test Expense.list:

The first 4 lines of the test create instances of the Expense model instance.

one on january 3rd 2013
one on november 1st 2013
one on november 2nd 2013
one on january 1st 2014

home that saves you some horizontal scrolling.

def test_list_expenses_filter_by_date(user):
    expense1 = Expense.new(user=user, amount=200.12, description="a netbook", comment="", timestamp=datetime.datetime(year=2013, month=1, day=3, hour=12, minute=30))
    expense2 = Expense.new(user=user, amount=1.0, description="A can of coke", comment="Really cheap!",timestamp=datetime.datetime(year=2013, month=11, day=1, hour=12, minute=30))
    expense3 = Expense.new(user=user, amount=3.05, description="A burger", comment="Tasty!", timestamp=datetime.datetime(year=2013, month=11, day=2, hour=12))
    expense4 = Expense.new(user=user, amount=0.30, description="Toilet visit", comment="I normally don't pay for these things, but it was urgent.", timestamp=datetime.datetime(year=2014, month=1, day=1, hour=1, minute=15))

    from_date = datetime.datetime(year=2013, month=5, day=1)
    to_date = datetime.datetime(year=2013, month=12, day=31)
    exps  = Expense.list(user=user, from_date=from_date, to_date=to_date)
    for expense in exps:
        assert from_date <= expense.timestamp <= to_date

But when I run this test from pytest, I get an AssertionError on the following line, as displayed by pytest:

assert datetime.datetime(2013, 5, 1, 0, 0) <= datetime.datetime(2013, 1, 3, 12, 30)

E + where datetime.datetime(2013, 1, 3, 12, 30) = Expense(key=Key('Expense', 5066549580791808), amount=200.12, comment=u'', desc...tamp=datetime.datetime(2013, 1, 3, 12, 30), user=Key('User', 5629499534213120)).timestamp

So why is my Expense.list method returning instances with a dat that is out of range?

I've tried this both in the app engine testbed, and in a requesthandler, viewing the output in a browser, so it's not a bug in the datastore stub.

I've also tried building the query differently. Instead of calling Query.filter repeatedly, I added filter expressions to a list, passed that list as *args to ndb.AND and passed that to a single Query.filter but that didn't work either.

I've ruled out the possibility of a bug in ndb. The following test passes:

def test_ndb_date():
    from google.appengine.ext import ndb

    class Time(ndb.Model):
        timestamp = ndb.DateTimeProperty(required=True)

    t1 = Time(timestamp=datetime.datetime.now() - datetime.timedelta(days=365))
    t2 = Time(timestamp=datetime.datetime.now())
    t3 = Time(timestamp=datetime.datetime.now() + datetime.timedelta(days=365))
    ndb.put_multi([t1, t2, t3])
    from_date = datetime.datetime.now() - datetime.timedelta(days=1)
    to_date = datetime.datetime.now() + datetime.timedelta(days = 1)
    result = Time.query(Time.timestamp <= from_date, Time.timestamp >= to_date).fetch()
    for time in result:
        assert from_date <= time.timestamp >= to_date

bigblind bigblind · Accepted Answer · 2014-01-06T18:05:06

I misunderstood how Query.filter works. It doesn't alter the current Query instance, but returns a new one.

ndb not querying by date correctly

1 Answers