Last Updated On By Anmol Lohana
A list is a container that contains different Python objects.
In this article, we will learn how to remove duplicates from a list. There are examples of Python remove duplicates from list for better understanding.
There are different ways to remove duplicates. We will discuss six different ways to remove duplicates from list Python.
Table of Contents
The naive method traverses the list and appends the first occurrence of the element in a new list. It ignores the repeated or duplicate elements.
d_list = [1, 3, 5, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = []
for i in d_list:
if i not in r_list:
r_list.append(i)
print ("List after Duplicate Items Removed : " + str(r_list))
The set() Method is Python’s built-in function that converts the list into a unique set. It means this method will remove all duplicates or repeated elements from the original list and returns a unique set.
d_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = list(set(d_list))
print ("List after Duplicate Items Removed : " + str(r_list))
It will first remove the duplicate items from a list and returns a dictionary. Then it will convert the dictionary to a unique list with no repetition of elements. You need to import the collections module as (i.e., from collections import OrderedDict).
from collections import OrderedDict
d_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = list(OrderedDict.fromkeys(d_list))
print ("List after Duplicate Items Removed : " + str(r_list))
The list.count() method will count the number of occurrence of the values in a list. We can use it with remove() method to remove duplicate elements in list.
d_list = [1, 5, 3, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
for i in d_list:
if d_list.count(i) > 1:
d_list.remove(i)
print ("List after Duplicate Items Removed : " + str(d_list))
You have the sort() method to sort the set that we obtained in approach 2. It will also remove any duplicates while preserving the order but is slower than the dict.from keys() approach.
d_list = [1, 5, 3, 3, 5, 6, 1]
print ("Original List : " + str(d_list))
r_list = list(set(d_list))
r_list.sort(key=d_list.index)
print ("List after Duplicate Items Removed : " + str(r_list))
You can use the pandas duplicated i.e. pandas.drop_duplicates() method to remove the duplicate values from the original list while also preserving the order. To use this, you need to import pandas.
import pandas as pd
d_list = [1, 5, 3, 3, 5, 6, 1]
print("Original List : " , d_list)
r_list = pd.Series(d_list).drop_duplicates().tolist()
print("List after Duplicate items Removed: ", r_list)
In this article, we have discussed remove duplicate items from a list in six different ways. To understand all the six ways much better, we did their coding examples and saw the results or solutions.