unique - Python: Uniqueness for list of lists

Question

I am curious what would be an efficient way of uniquefying such data objects:

testdata =[ ['9034968', 'ETH'], ['14160113', 'ETH'], ['9034968', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15724032', 'ETH'], ['15481740', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['10307528', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['15481740', 'ETH'], ['15379365', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15379365', 'ETH']
]

For each data pair, left numeric string PLUS the type at the right tells the uniqueness of a data element. The return value should be a list of lists as same as the testdata, but with only the unique values kept.

Answer 1

You can use a set:

unique_data = [list(x) for x in set(tuple(x) for x in testdata)]

You can also see this page which benchmarks a variety of methods that either preserve or don't preserve order.

Answer 2

I tried @Mark's answer and got an error. Converting the list and each elements into a tuple made it work. Not sure if this the best way though.

list(map(list, set(map(lambda i: tuple(i), testdata))))

Of course the same thing can be expressed using a list comprehension instead.

[list(i) for i in set(tuple(i) for i in testdata)]

I am using Python 2.6.2.

Update

@Mark has since changed his answer. His current answer uses tuples and will work. So will mine :)

Update 2

Thanks to @Mark. I have changed my answer to return a list of lists rather than a list of tuples.

Answer 3

import sets
testdata =[ ['9034968', 'ETH'], ['14160113', 'ETH'], ['9034968', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15724032', 'ETH'], ['15481740', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['10307528', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['15481740', 'ETH'], ['15379365', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15379365', 'ETH']]
conacatData = [x[0] + x[1] for x in testdata]
print conacatData
uniqueSet = sets.Set(conacatData)
uniqueList = [ [t[0:-3], t[-3:]] for t in uniqueSet]
print uniqueList

Answer 4

Expanding a bit on @Mark Byers solution, you can also just do one list comprehension and typecast to get what you need:

testdata = list(set(tuple(x) for x in testdata))

Also, if you don't like list comprehensions as many find them confusing, you can do the same in a for loop:

for i, e in enumerate(testdata):
    testdata[i] = tuple(e)
testdata = list(set(testdata))

Answer 5

if you have a list of objects than you can modify @Mark Byers answer to:

unique_data = [list(x) for x in set(tuple(x.testList) for x in testdata)]

where testdata is a list of objects which has a list testList as attribute.

Answer 6

Use unique in numpy to solve this:

import numpy as np

np.unique(np.array(testdata), axis=0)

Note that the axis keyword needs to be specified otherwise the list is first flattened.

Alternatively, use vstack:

np.vstack({tuple(row) for row in testdata})