Additional menu

Your go-to place for programming guides

Remove Duplicates from a List
You are here: Home / Python / Remove Duplicates from a List in Six Different Ways – Python

Remove Duplicates from a List in Six Different Ways – Python

By

This article will discuss the process of getting a unique list out of a list that contains duplicate items in it. And the process of remove duplicates from a list and get a unique list as a result.

There are different ways to sort this situation. We will discuss six different ways of removing duplicate items from a list.

  • Naive Method or Iteration
  • Using set() Method
  • Preserving Order – [OrderedDict.from keys()]
  • Using list.count() Method
  • Using sort() Method
  • With Pandas Module

Naive Method or Using Iterator (with for loop)

The naive method traverses the list and appends the first occurrence of the element in a new list. It ignores the repeated or duplicate elements.

Code Example

d_list = [1, 3, 5, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))

r_list = []
for i in d_list:
	if i not in r_list:
		r_list.append(i)

print ("List after Duplicate Items Removed : " + str(r_list))

Output

Iteration to Remove Duplicates from a List

Using set() Method

The set() Method is Python’s built-in function that converts the list into a unique set. It means this method will remove all duplicates or repeated elements from the original list and returns a unique set.

Code Example

d_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))

r_list = list(set(d_list))

print ("List after Duplicate Items Removed : " + str(r_list))

Output

set() Method to Remove Duplicates from a List

Preserving Order – [OrderedDict.fromkeys() function]

It will first remove the duplicate items from a list and returns a dictionary. Then it will convert the dictionary to a unique list with no repetition of elements. You need to import the collections module as (i.e., from collections import OrderedDict).

Code Example

from collections import OrderedDict

d_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))

r_list = list(OrderedDict.fromkeys(d_list))

print ("List after Duplicate Items Removed : " + str(r_list))

Output

fromkeys() function

Using list.count() Method

The list.count() method will count the number of occurrence of the values in a list. We can use it with remove() method to remove duplicate elements in list.

Code Example

d_list = [1, 5, 3, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
 
for i in d_list:
    if d_list.count(i) > 1:
        d_list.remove(i)
 
print ("List after Duplicate Items Removed : " + str(d_list))

Output

Count

Using sort() Method

You have the sort() method to sort the set that we obtained in approach 2. It will also remove any duplicates while preserving the order but is slower than the dict.from keys() approach.

Code Example

d_list = [1, 5, 3, 3, 5, 6, 1]
print ("Original List : " + str(d_list))

r_list = list(set(d_list))
r_list.sort(key=d_list.index)
 
print ("List after Duplicate Items Removed : " + str(r_list))

Output

sort() Method

With Pandas Module

You can use the pandas.drop_duplicates() method to remove the duplicate values from the original list while also preserving the order. To use this, you need to import pandas.

Code Example

import pandas as pd

d_list = [1, 5, 3, 3, 5, 6, 1]

print("Original List : " , d_list)

r_list = pd.Series(d_list).drop_duplicates().tolist()

print("List after Duplicate items Removed: ", r_list)

Output

Pandas

Conclusion

In this article, we have discussed remove duplicate items from a list in six different ways. To understand all the six ways much better, we did their coding examples and saw the results or solutions.

You may also visit : Python List to String.