Get all unique values in a JavaScript array (remove duplicates)

1968

votes

I have an array of numbers that I need to make sure are unique. I found the code snippet below on the internet and it works great until the array has a zero in it. I found this other script here on Stack Overflow that looks almost exactly like it, but it doesn't fail.

So for the sake of helping me learn, can someone help me determine where the prototype script is going wrong?

Array.prototype.getUnique = function() {
 var o = {}, a = [], i, e;
 for (i = 0; e = this[i]; i++) {o[e] = 1};
 for (e in o) {a.push (e)};
 return a;
}

Use object `{ }` to prevent duplicates

function uniqueArray1( ar ) {
  var j = {};

  ar.forEach( function(v) {
    j[v+ '::' + typeof v] = v;
  });

  return Object.keys(j).map(function(v){
    return j[v];
  });
}

Use helper array `[ ]`

function uniqueArray2(arr) {
    var a = [];
    for (var i=0, l=arr.length; i<l; i++)
        if (a.indexOf(arr[i]) === -1 && arr[i] !== '')
            a.push(arr[i]);
    return a;
}

Use `filter + indexOf`

function uniqueArray3(a) {
  function onlyUnique(value, index, self) { 
      return self.indexOf(value) === index;
  }

  // usage
  var unique = a.filter( onlyUnique ); // returns ['a', 1, 2, '1']

  return unique;
}

Use ES6 `[...new Set(a)]`

function uniqueArray4(a) {
  return [...new Set(a)];
}

And I wondered which one is faster. I've made sample Google Sheet to test functions. Note: ECMA 6 is not avaliable in Google Sheets, so I can't test it.

Here's the result of tests:

I expected to see that code using object { } will win because it uses hash. So I'm glad that tests showed the best results for this algorithm in Chrome and IE. Thanks to @rab for the code.

Update 2020

Google Script enabled ES6 Engine. Now I tested the last code with Sets and it appeared faster than the object method.

149

votes

You can also use underscore.js.

console.log(_.uniq([1, 2, 1, 3, 1, 4]));

<script src="http://underscorejs.org/underscore-min.js"></script>

which will return:

[1, 2, 3, 4]

84

votes

One Liner, Pure JavaScript

With ES6 syntax

list = list.filter((x, i, a) => a.indexOf(x) == i)

x --> item in array
i --> index of item
a --> array reference, (in this case "list")

With ES5 syntax

list = list.filter(function (x, i, a) { 
    return a.indexOf(x) == i; 
});

Browser Compatibility: IE9+

55

votes

I have since found a nice method that uses jQuery

arr = $.grep(arr, function(v, k){
    return $.inArray(v ,arr) === k;
});

Note: This code was pulled from Paul Irish's duck punching post - I forgot to give credit :P

54

votes

Shortest solution with ES6: [...new Set( [1, 1, 2] )];

Or if you want to modify the Array prototype (like in the original question):

Array.prototype.getUnique = function() {
    return [...new Set( [this] )];
};

EcmaScript 6 is only partially implemented in modern browsers at the moment (Aug. 2015), but Babel has become very popular for transpiling ES6 (and even ES7) back to ES5. That way you can write ES6 code today!

If you're wondering what the ... means, it's called the spread operator. From MDN: «The spread operator allows an expression to be expanded in places where multiple arguments (for function calls) or multiple elements (for array literals) are expected». Because a Set is an iterable (and can only have unique values), the spread operator will expand the Set to fill the array.

Resources for learning ES6:

Exploring ES6 by Dr. Axel Rauschmayer
Search “ES6” from JS weekly newsletters
ES6 in depth articles from the Mozilla Hacks blog

46

votes

Many of the answers here may not be useful to beginners. If de-duping an array is difficult, will they really know about the prototype chain, or even jQuery?

In modern browsers, a clean and simple solution is to store data in a Set, which is designed to be a list of unique values.

const cars = ['Volvo', 'Jeep', 'Volvo', 'Lincoln', 'Lincoln', 'Ford'];
const uniqueCars = Array.from(new Set(cars));
console.log(uniqueCars);

The Array.from is useful to convert the Set back to an Array so that you have easy access to all of the awesome methods (features) that arrays have. There are also other ways of doing the same thing. But you may not need Array.from at all, as Sets have plenty of useful features like forEach.

If you need to support old Internet Explorer, and thus cannot use Set, then a simple technique is to copy items over to a new array while checking beforehand if they are already in the new array.

// Create a list of cars, with duplicates.
var cars = ['Volvo', 'Jeep', 'Volvo', 'Lincoln', 'Lincoln', 'Ford'];
// Create a list of unique cars, to put a car in if we haven't already.
var uniqueCars = [];

// Go through each car, one at a time.
cars.forEach(function (car) {
    // The code within the following block runs only if the
    // current car does NOT exist in the uniqueCars list
    // - a.k.a. prevent duplicates
    if (uniqueCars.indexOf(car) === -1) {
        // Since we now know we haven't seen this car before,
        // copy it to the end of the uniqueCars list.
        uniqueCars.push(car);
    }
});

To make this instantly reusable, let's put it in a function.

function deduplicate(data) {
    if (data.length > 0) {
        var result = [];

        data.forEach(function (elem) {
            if (result.indexOf(elem) === -1) {
                result.push(elem);
            }
        });

        return result;
    }
}

So to get rid of the duplicates, we would now do this.

var uniqueCars = deduplicate(cars);

The deduplicate(cars) part becomes the thing we named result when the function completes.

Just pass it the name of any array you like.

40

votes

Simplest solution:

var arr = [1, 3, 4, 1, 2, 1, 3, 3, 4, 1];
console.log([...new Set(arr)]);

Or:

var arr = [1, 3, 4, 1, 2, 1, 3, 3, 4, 1];
console.log(Array.from(new Set(arr)));

34

votes

The simplest, and fastest (in Chrome) way of doing this:

Array.prototype.unique = function() {
    var a = [];
    for (var i=0, l=this.length; i<l; i++)
        if (a.indexOf(this[i]) === -1)
            a.push(this[i]);
    return a;
}

Simply goes through every item in the array, tests if that item is already in the list, and if it's not, push to the array that gets returned.

According to jsPerf, this function is the fastest of the ones I could find anywhere - feel free to add your own though.

The non-prototype version:

function uniques(arr) {
    var a = [];
    for (var i=0, l=arr.length; i<l; i++)
        if (a.indexOf(arr[i]) === -1 && arr[i] !== '')
            a.push(arr[i]);
    return a;
}

Sorting

When also needing to sort the array, the following is the fastest:

Array.prototype.sortUnique = function() {
    this.sort();
    var last_i;
    for (var i=0;i<this.length;i++)
        if ((last_i = this.lastIndexOf(this[i])) !== i)
            this.splice(i+1, last_i-i);
    return this;
}

or non-prototype:

function sortUnique(arr) {
    arr.sort();
    var last_i;
    for (var i=0;i<arr.length;i++)
        if ((last_i = arr.lastIndexOf(arr[i])) !== i)
            arr.splice(i+1, last_i-i);
    return arr;
}

This is also faster than the above method in most non-chrome browsers.

33

votes

PERFORMANCE ONLY! this code is probably 10X faster than all the codes in here *works on all browsers and also has the lowest memory impact.... and more

if you don't need to reuse the old array;btw do the necessary other operations before you convert it to unique here is probably the fastest way to do this, also very short.

var array=[1,2,3,4,5,6,7,8,9,0,1,2,1];

then you can try this

var array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 1];

function toUnique(a, b, c) { //array,placeholder,placeholder
  b = a.length;
  while (c = --b)
    while (c--) a[b] !== a[c] || a.splice(c, 1);
  return a // not needed ;)
}
console.log(toUnique(array));
//[3, 4, 5, 6, 7, 8, 9, 0, 2, 1]

I came up with this function reading this article...

http://www.shamasis.net/2009/09/fast-algorithm-to-find-unique-items-in-javascript-array/

I don't like the for loop. it has to many parameters.i like the while-- loop. while is the fastest loop in all browsers except the one we all like so much... chrome.

anyway i wrote the first function that uses while.And yep it's a little faster than the function found in the article.but not enough.unique2()

next step use modern js.Object.keys i replaced the other for loop with js1.7's Object.keys... a little faster and shorter (in chrome 2x faster) ;). Not enough!.unique3().

at this point i was thinking about what i really need in MY unique function. i don't need the old array, i want a fast function. so i used 2 while loops + splice.unique4()

Useless to say that i was impressed.

chrome: the usual 150,000 operations per second jumped to 1,800,000 operations per second.

ie: 80,000 op/s vs 3,500,000 op/s

ios: 18,000 op/s vs 170,000 op/s

safari: 80,000 op/s vs 6,000,000 op/s

Proof http://jsperf.com/wgu or better use console.time... microtime... whatever

unique5() is just to show you what happens if you want to keep the old array.

Don't use Array.prototype if yu don't know what your doing. i just did alot of copy and past. Use Object.defineProperty(Array.prototype,...,writable:false,enumerable:false}) if you want to create a native prototype.example: https://stackoverflow.com/a/20463021/2450730

Demo http://jsfiddle.net/46S7g/

NOTE: your old array is destroyed/becomestheunique after this operation.

if you can't read the code above ask, read a javascript book or here are some explainations about shorter code. https://stackoverflow.com/a/21353032/2450730

some are using indexOf ... don't ... http://jsperf.com/dgfgghfghfghghgfhgfhfghfhgfh

for empty arrays

!array.length||toUnique(array);

23

votes

["Defects", "Total", "Days", "City", "Defects"].reduce(function(prev, cur) {
  return (prev.indexOf(cur) < 0) ? prev.concat([cur]) : prev;
 }, []);

[0,1,2,0,3,2,1,5].reduce(function(prev, cur) {
  return (prev.indexOf(cur) < 0) ? prev.concat([cur]) : prev;
 }, []);

23

votes

Magic

a.filter(e=>!(t[e]=e in t))

O(n) performance (is faster than new Set); we assume your array is in a and t={}. Explanation here (+Jeppe impr.)

let t, unique= a=> ( t={}, a.filter(e=>!(t[e]=e in t)) );

// "stand-alone" version working with global t:
// a1.filter((t={},e=>!(t[e]=e in t)));

// Test data
let a1 = [5,6,0,4,9,2,3,5,0,3,4,1,5,4,9];
let a2 = [[2, 17], [2, 17], [2, 17], [1, 12], [5, 9], [1, 12], [6, 2], [1, 12]];
let a3 = ['Mike', 'Adam','Matt', 'Nancy', 'Adam', 'Jenny', 'Nancy', 'Carl'];

// Results
console.log(JSON.stringify( unique(a1) ))
console.log(JSON.stringify( unique(a2) ))
console.log(JSON.stringify( unique(a3) ))

22

votes

We can do this using ES6 sets:

var duplicatedArray = [1, 2, 3, 4, 5, 1, 1, 1, 2, 3, 4];
var uniqueArray = Array.from(new Set(duplicatedArray));

console.log(uniqueArray);

//The output will be

uniqueArray = [1,2,3,4,5];

18

votes

This prototype getUnique is not totally correct, because if i have a Array like: ["1",1,2,3,4,1,"foo"] it will return ["1","2","3","4"] and "1" is string and 1 is a integer; they are different.

Here is a correct solution:

Array.prototype.unique = function(a){
    return function(){ return this.filter(a) }
}(function(a,b,c){ return c.indexOf(a,b+1) < 0 });

using:

var foo;
foo = ["1",1,2,3,4,1,"foo"];
foo.unique();

The above will produce ["1",2,3,4,1,"foo"].

16

votes

[...new Set(duplicates)]

This is the simplest one and referenced from MDN Web Docs.

const numbers = [2,3,4,4,2,3,3,4,4,5,5,6,6,7,5,32,3,4,5]
console.log([...new Set(numbers)]) // [2, 3, 4, 5, 6, 7, 32]

16

votes

After looking into all the 90+ answers here, I saw there is room for one more:

Array.includes has a very handy second-parameter: "fromIndex", so by using it, every iteration of the filter callback method will search the array, starting from [current index] + 1 which guarantees not to include currently filtered item in the lookup and also saves time.

//                🚩              🚩 🚩
var list = [0,1,2,2,3,'a','b',4,5,2,'a']

console.log( 
  list.filter((v,i) => !list.includes(v,i+1))
)

// [0,1,3,"b",4,5,2,"a"]

Explanation:

For example, lets assume the filter function is currently iterating at index 2) and the value at that index happens to be 2. The section of the array that is then scanned for duplicates (includes method) is everything after index 2 (i+1):

           👇                    👇
[0, 1, 2,   2 ,3 ,'a', 'b', 4, 5, 2, 'a']
       👆   |---------------------------|

And since the currently filtered item's value 2 is included in the rest of the array, it will be filtered out, because of the leading exclamation mark which negates the filter rule.

15

votes

If you're using Prototype framework there is no need to do 'for' loops, you can use http://www.prototypejs.org/api/array/uniq like this:

var a = Array.uniq();

Which will produce a duplicate array with no duplicates. I came across your question searching a method to count distinct array records so after

uniq()

I used

size()

and there was my simple result. p.s. Sorry if i misstyped something

edit: if you want to escape undefined records you may want to add

compact()

before, like this:

var a = Array.compact().uniq();

15

votes

Without extending Array.prototype (it is said to be a bad practice) or using jquery/underscore, you can simply filter the array.

By keeping last occurrence:

    function arrayLastUnique(array) {
        return array.filter(function (a, b, c) {
            // keeps last occurrence
            return c.indexOf(a, b + 1) < 0;
        });
    },

or first occurrence:

    function arrayFirstUnique(array) {
        return array.filter(function (a, b, c) {
            // keeps first occurrence
            return c.indexOf(a) === b;
        });
    },

Well, it's only javascript ECMAScript 5+, which means only IE9+, but it's nice for a development in native HTML/JS (Windows Store App, Firefox OS, Sencha, Phonegap, Titanium, ...).

13

votes

You can use Set operator to get unique values from an array

const uniqueArray = [...new Set([1, 1, 1])];

console.log(uniqueArray) // [1]

12

votes

That's because 0 is a falsy value in JavaScript.

this[i] will be falsy if the value of the array is 0 or any other falsy value.

12

votes

Array.prototype.getUnique = function() {
    var o = {}, a = []
    for (var i = 0; i < this.length; i++) o[this[i]] = 1
    for (var e in o) a.push(e)
    return a
}

12

votes

This has been answered a lot, but it didn't address my particular need.

Many answers are like this:

a.filter((item, pos, self) => self.indexOf(item) === pos);

But this doesn't work for arrays of complex objects.

Say we have an array like this:

const a = [
 { age: 4, name: 'fluffy' },
 { age: 5, name: 'spot' },
 { age: 2, name: 'fluffy' },
 { age: 3, name: 'toby' },
];

If we want the objects with unique names, we should use array.prototype.findIndex instead of array.prototype.indexOf:

a.filter((item, pos, self) => self.findIndex(v => v.name === item.name) === pos);

12

votes

Now using sets you can remove duplicates and convert them back to the array.

var names = ["Mike","Matt","Nancy", "Matt","Adam","Jenny","Nancy","Carl"];

console.log([...new Set(names)])

Another solution is to use sort & filter

var names = ["Mike","Matt","Nancy", "Matt","Adam","Jenny","Nancy","Carl"];
var namesSorted = names.sort();
const result = namesSorted.filter((e, i) => namesSorted[i] != namesSorted[i+1]);
console.log(result);

11

votes

I had a slightly different problem where I needed to remove objects with duplicate id properties from an array. this worked.

let objArr = [{
  id: '123'
}, {
  id: '123'
}, {
  id: '456'
}];

objArr = objArr.reduce((acc, cur) => [
  ...acc.filter((obj) => obj.id !== cur.id), cur
], []);

console.log(objArr);

10

votes

The simplest answer is :

const array = [1, 1, 2, 2, 3, 5, 5, 2];
const uniqueArray = [...new Set(array)];
console.log(uniqueArray); // [1, 2, 3, 5]

8

votes

If you're okay with extra dependencies, or you already have one of the libraries in your codebase, you can remove duplicates from an array in place using LoDash (or Underscore).

Usage

If you don't have it in your codebase already, install it using npm:

npm install lodash

Then use it as follows:

import _ from 'lodash';
let idArray = _.uniq ([
    1,
    2,
    3,
    3,
    3
]);
console.dir(idArray);

Out:

[ 1, 2, 3 ]

7

votes

I'm not sure why Gabriel Silveira wrote the function that way but a simpler form that works for me just as well and without the minification is:

Array.prototype.unique = function() {
  return this.filter(function(value, index, array) {
    return array.indexOf(value, index + 1) < 0;
  });
};

or in CoffeeScript:

Array.prototype.unique = ->
  this.filter( (value, index, array) ->
    array.indexOf(value, index + 1) < 0
  )

7

votes

Finding unique Array values in simple method

function arrUnique(a){
  var t = [];
  for(var x = 0; x < a.length; x++){
    if(t.indexOf(a[x]) == -1)t.push(a[x]);
  }
  return t;
}
arrUnique([1,4,2,7,1,5,9,2,4,7,2]) // [1, 4, 2, 7, 5, 9]

7

votes

strange this hasn't been suggested before.. to remove duplicates by object key (id below) in an array you can do something like this:

const uniqArray = array.filter((obj, idx, arr) => (
  arr.findIndex((o) => o.id === obj.id) === idx
))

Get all unique values in a JavaScript array (remove duplicates)

More answers from duplicate question:

Similar question:

30 Answers

Use object `{ }` to prevent duplicates

Use helper array `[ ]`

Use `filter + indexOf`

Use ES6 `[...new Set(a)]`

Update 2020

One Liner, Pure JavaScript

Sorting

Magic

Explanation:

Get all unique values in a JavaScript array (remove duplicates)

More answers from duplicate question:

Similar question:

30 Answers

Use object { } to prevent duplicates

Use helper array [ ]

Use filter + indexOf

Use ES6 [...new Set(a)]

Update 2020

One Liner, Pure JavaScript

Sorting

Magic

Explanation:

Use object `{ }` to prevent duplicates

Use helper array `[ ]`

Use `filter + indexOf`

Use ES6 `[...new Set(a)]`