A list is a container that contains different Python objects.
In this article, we will learn how to remove duplicates from a list. There are examples of Python remove duplicates from list for better understanding.
There are different ways to remove duplicates. We will discuss six different ways to remove duplicates from list Python.
- Naive Method or Iteration
- Using set() Method
- Preserving Order – [OrderedDict.from keys()]
- Using list.count() Method
- Using sort() Method
- With Pandas Module
Table of Contents
1. Python Naive Method or Using Iterator (with for loop)
The naive method traverses the list and appends the first occurrence of the element in a new list. It ignores the repeated or duplicate elements.
Code Example
d_list = [1, 3, 5, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = []
for i in d_list:
if i not in r_list:
r_list.append(i)
print ("List after Duplicate Items Removed : " + str(r_list))
Output
2. Python set() Method
The set() Method is Python’s built-in function that converts the list into a unique set. It means this method will remove all duplicates or repeated elements from the original list and returns a unique set.
Code Example
d_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = list(set(d_list))
print ("List after Duplicate Items Removed : " + str(r_list))
Output
3. Python Preserving Order – [OrderedDict.fromkeys() function]
It will first remove the duplicate items from a list and returns a dictionary. Then it will convert the dictionary to a unique list with no repetition of elements. You need to import the collections module as (i.e., from collections import OrderedDict).
Code Example
from collections import OrderedDict
d_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = list(OrderedDict.fromkeys(d_list))
print ("List after Duplicate Items Removed : " + str(r_list))
Output
4. Python list.count() Method
The list.count() method will count the number of occurrence of the values in a list. We can use it with remove() method to remove duplicate elements in list.
Code Example
d_list = [1, 5, 3, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
for i in d_list:
if d_list.count(i) > 1:
d_list.remove(i)
print ("List after Duplicate Items Removed : " + str(d_list))
Output
5. Python sort() Method
You have the sort() method to sort the set that we obtained in approach 2. It will also remove any duplicates while preserving the order but is slower than the dict.from keys() approach.
Code Example
d_list = [1, 5, 3, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = list(set(d_list))
r_list.sort(key=d_list.index)
print ("List after Duplicate Items Removed : " + str(r_list))
Output
6. Python Pandas Module
You can use the pandas duplicated i.e. pandas.drop_duplicates() method to remove the duplicate values from the original list while also preserving the order. To use this, you need to import pandas.
Code Example
import pandas as pd
d_list = [1, 5, 3, 3, 5, 6, 1]
print("Original List : " , d_list)
r_list = pd.Series(d_list).drop_duplicates().tolist()
print("List after Duplicate items Removed: ", r_list)
Output
Conclusion
In this article, we have discussed remove duplicate items from a list in six different ways. To understand all the six ways much better, we did their coding examples and saw the results or solutions.
