islice from right-hand side

Question

For lists one can use slicing my_list[-10:] to get the last (up to) 10 elements

I want to do the same with itertools.islice. In this case I have a collections.deque.

Is there a more efficient way of doing this than the following?

from collections import deque
import itertools
my_deque = deque()
my_list = list()
for i in range(100):
    my_deque.append(i)
    my_list.append(i)

And then we do the slicing:

start = max(0, len(my_deque) - 10)
for i in list(itertools.islice(my_deque, start, None)):
    pass

My timings:

deque: 1000000 loops, best of 3: 962 ns per loop

list slicing: 10000000 loops, best of 3: 95.9 ns per loop

Show the full, exact, complete, runnable code to get those timeit results. — John Zwinck
itertools is designed for working with arbitrary, possibly infinite, iterators. An infinite iterator doesn't have a "last" 10 items. Also, for an arbitrary iterator, the only way to get an item is to call its next method to get a single item; more efficient methods simply don't exist. — chepner
About for i in list(itertools.islice(my_deque, start, None)): the conversion into a list is useless and may use memory for nothing. — glenfant
Why not simply using deque.rotate() then yield values till your desired range ? — Chiheb Nexus
@Chris_Rands I don't think there's any iterator that has a __reversed__ method, only some containers have that method. — MSeifert

MSeifert MSeifert · Accepted Answer · 2017-08-08T09:11:51

As you found out there's no way to slice a collections.deque. But it supports rotation which could be used in this case:

last_n = 10
my_deque.rotate(last_n)
for i in itertools.islice(my_deque, last_n):
    pass

The problem with itertools.islice in this case is that it needs to iterate over all elements until it reaches the stop. That's because it needs to work with iterators in general not only with random-access-containers like lists and deques. So in your case the islice really had to iterate over all elements in the deque. With rotate and then islice it only has to iterate over 10 elements.

As for timings, it's hard to know what you compared but using this setup:

from collections import deque
import itertools
my_deque = deque(range(10000))
my_list = list(range(10000))

I get the following timings:

%%timeit

my_deque.rotate(10)
for i in itertools.islice(my_deque, 10):
    pass
my_deque.rotate(-10)  # so the next timing operates on the orginal deque again

2.76 µs ± 41.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%%timeit

start = max(0, len(my_deque) - 10)
for i in itertools.islice(my_deque, start, None):
    pass

136 µs ± 8.08 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%%timeit

start = max(0, len(my_list) - 10)
for i in itertools.islice(my_list, start, None):
    pass

119 µs ± 1.64 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit my_list[-10:]

434 ns ± 12.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

So it can't beat list slicing (still ~5 times slower) but it's definitely much faster (~50 times) than using your islice approaches.

islice from right-hand side

My timings:

1 Answers