8
votes

I have a 2-d array

 xx=[[a,1],[b,2],[c,3]]

Now I'm trying to remove duplicate entries from it. For simple 1-D array, simple code like

xx=list(set(xx))

would work. But trying set on 2-d elements gives an error

temp = set(xx)
TypeError: unhashable type: 'list'

One workaround would be to serialize xx elements, and then do a list(set()) on new array and then unserialize all the elements back again.

Is there any solution in python?

4

4 Answers

21
votes

Convert elements to tuple and then use set.

>>> xx=[['a',1],['b',2],['c',3],['c',3]]
>>> set(tuple(element) for element in xx)
set([('a', 1), ('b', 2), ('c', 3)])
>>> 

Tuples, unlike lists, can be hashed. Hence. And once you are done, convert the elements back to list. Putting everything together:

>>> [list(t) for t in set(tuple(element) for element in xx)]
[['a', 1], ['b', 2], ['c', 3]]
2
votes

One year after the excellent answer of Manoj Govindan, I'm adding my piece of advice:

Floating points numbers are just a pain if you want to compare things...

For instance,

>>>0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1 == 0.1*10

False

That's because your computer can't accurately represent decimal floating points as binary numbers (computers handle binary/base 2 numbers only, not decimals/base 10).

So be really careful when comparing floats!

1
votes

This is my solution, I've left the a[i][0] intentionally this way so you can change the member regarding your need.

ab= [['2.71.122.116', 'test_sys_-fw.test_sys_.us'],
     ['10.10.100.26', 'test_sys_5k1'],
     [None, 'Azure'],
     [None, 'test-server'],
     ['2.71.122.119', 'asa-5506-fw'],
     ['33.151.18.23', 'netscaler1'],
     ['33.151.18.23', 'netscaler2'],
     ['33.151.18.23', 'Palo Alto'],
     ['33.151.18.23', 'Arbor CP'],
     ['44.221.2.100', 'fw-la5515'],
     ['44.221.2.101', 'fw-la2-5515'],
     ['44.221.2.99', 'NexusLA2'],
     ['44.221.2.103', 'ASALA5510'],
     ['2.71.122.120', 'asa-5506-fw2'],
     ['2.71.122.106', '2928_SW2']]

def deduplicate_by_ip(a):
    """
    Clears Empty ip address records from list
    removes duplicates by
    :param a:
    :return:
    """

    source_ips = []
    new_list = []
    for i in range(len(a)):
        if a[i][0] != None:
            if a[i][0] not in source_ips:
                source_ips.append(a[i][0])
                new_list.append(a[i])
    return new_list

list = deduplicate_by_ip(ab)
print("Total items in original list :", len(ab))
print("Total items after deduplication :", len(list))
print("The list", list)
0
votes

If the order doesn't matter, I believe the most concise way (for code-golf etc.) is to use the built-in map, list, and tuple with extended iterable unpacking and Python 3.5's additional unpacking generalizations:

x = [["a", 1], ["b", 2], ["c", 3], ["c", 3], ["a", 1]]

*y,=map(list,{*map(tuple,x)})

print(y)

Output:

[['a', 1], ['b', 2], ['c', 3]]

See it live