Exploratory Data Analysis in just 5 minutes using Pandas Profiling

Introduction

 If you are into Data Science or you love to analyze the data than you would understand that, how important it is to analyze the data before we start working on it. The first step before we start analyzing the data is 'Data Eyeballing'. Later we analyze the data by visualizing it using a library like Matplotlib, Pandas, etc. And after, writing many lines of codes finally we will get a few charts with which we will be able to get some insights about the data. But what if can get a detailed description of our dataset in an HTML file just by writing a few lines of codes.

To achieve this, we will use a library named 'Pandas Profiling'. Pandas profiling is an open-source python module with which we can do exploratory data analysis just by writing a few lines of code. It allows the user to generate an HTML file of all the data insights, without writing any extra line of code for that. In short, this library saves us from all the work of visualizing and understanding the data. So let's straight away dive into this module and learn how can we use it to analyze the data-

Prerequisite

Before we move further I hope that you have python3 installed in your machine. If not then follow this link to download the latest version of python-https://www.python.org/downloads/

How to make it work

The first step towards using this module is to install pandas-profiling in our machine. For that, we will type-

pip install pandas-profiling

The next step is to create a folder that contains one .py extension file and your dataset. For this example, I'm using the world dataset which contains some NaN values, which later help us to explore the features of Pandas Profiling.

World data is very handy as it comes with the installation of MySQL. You can choose any dataset but make sure that it has headers so that it will help you while reviewing your generated report.

Then, we generate the report using these commands by running them on any editor like VS code that can run python code:

import pandas as pd 

from pandas_profiling import ProfileReport

df=pd.read_csv('country.csv')

print(df)

#generate a report 

profile=ProfileReport(df)

profile.to_file(output_file='index.html')

 Here we are with an output_file named as 'index.html' which contains a detailed analysis of the entire dataset. 

In case you fumbled upon error like Pandas_profiling module not found even after installing pandas_profiling on your machine then try saving the project folder inside your python3 folder.

Well, congrats you have successfully created an EDA in less than 5 minutes.  If you want to see our dataset report click this link- https://kartikeydwi1997.github.io/EDA/.

In case you want to create a simplified short report, you can use the minimum mode of pandas profiling that was introduced in version 2.4. You can check the version installed with the command:

pandas_profiling.version._version_

 With the minimum mode, a simplified report can be generated with less information as compared to the full report which we generated earlier. The commands to be used are:

profile=ProfileReport(df, minimal=True)

profile.to_file(output_file='index.html')

Pandas Profiling Advantages

• It provides a detailed analysis of your dataset- starting from the description of each column with its basic property like mean, max, and min values, etc.
• Uses almost all types of plots to help you to get the insights of the data.
• Describes all correlations.
• Another big advantage of generating this report is that it shows a warning at the beginning. It gives all information about columns that contain NaN values, zero values, categorical and non-categorical columns with high cardinality, etc. All this helps a lot in data pre-processing before applying any machine learning model.

Pandas Profiling Disadvantages

The main disadvantage of pandas profiling is its use for large datasets. With the increase in the size of the dataset, the time to generate the report also increases hence it will require high computational power to generate reports of huge datasets.

To avoid such scenarios, we can take a sample of a large dataset (say 10000 rows) which represents the entire dataset. The command to get the random sample of a large dataset is:

profile=ProfileReport(df.sample(n=10000))

profile.to_file(output_file='index.html')

I hope this blog would have helped you to get some insights into pandas profiling. Please feel free to comment your thoughts on this. 

 

Read More

Set .discard(), .remove() & .pop()- HackerRank Solution

 Set .discard(), .remove() & .pop()- HackerRank Solution

.remove(x)

This operation removes element  from the set.
If element  does not exist, it raises a KeyError.
The .remove(x) operation returns None.

Example

>>> s = set([1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> s.remove(5)
>>> print s
set([1, 2, 3, 4, 6, 7, 8, 9])
>>> print s.remove(4)
None
>>> print s
set([1, 2, 3, 6, 7, 8, 9])
>>> s.remove(0)
KeyError: 0

---

.discard(x)

This operation also removes element  from the set.
If element  does not exist, it does not raise a KeyError.
The .discard(x) operation returns None.

Example

>>> s = set([1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> s.discard(5)
>>> print s
set([1, 2, 3, 4, 6, 7, 8, 9])
>>> print s.discard(4)
None
>>> print s
set([1, 2, 3, 6, 7, 8, 9])
>>> s.discard(0)
>>> print s
set([1, 2, 3, 6, 7, 8, 9])

---

.pop()

This operation removes and return an arbitrary element from the set.
If there are no elements to remove, it raises a KeyError.

Example

>>> s = set([1])
>>> print s.pop()
1
>>> print s
set([])
>>> print s.pop()
KeyError: pop from an empty set

---

Task

You have a non-empty set , and you have to execute  commands given in  lines.

The commands will be pop, remove and discard.

Input Format

The first line contains integer , the number of elements in the set .
The second line contains  space separated elements of set . All of the elements are non-negative integers, less than or equal to 9.
The third line contains integer , the number of commands.
The next  lines contains either pop, remove and/or discard commands followed by their associated value.

Constraints

Output Format

Print the sum of the elements of set  on a single line.

Sample Input

9
1 2 3 4 5 6 7 8 9
10
pop
remove 9
discard 9
discard 8
remove 7
pop 
discard 6
remove 5
pop 
discard 5

Sample Output

4

Explanation

After completing these  operations on the set, we get set. Hence, the sum is .

Note: Convert the elements of set s to integers while you are assigning them. To ensure the proper input of the set, we have added the first two lines of code to the editor.

Set .discard(), .remove() & .pop() Solution

#For beginner
n = int(input())
s = set(map(int, input().split()))
for i in range(int(input())):
    choice=input().split()
    if choice[0]=="pop" :
        s.pop()
    elif choice[0]=="remove" :
        s.remove(int(choice[1]))
    elif choice[0]=="discard" :
        s.discard(int(choice[1]))
print(sum(s))
#Modified version
n = int(input())
s = set(map(int, input().split())) 
for i in range(int(input())):
    eval('s.{0}({1})'.format(*input().split()+['']))
print(sum(s))

Explanation-  We have earlier used sets in many HackerRank problems using the earlier learned concepts we will declare a set s the choice is a list used to take the input. If the index 0 of the list is pop we will pop an element from s or else for remove and discard we will get the location present at index 1 of the list.

In the other code, we have used eval function which evaluates the expression if it is a legal python statement.

To read about sets follow this link-  https://www.geeksforgeeks.org/sets-in-python/

Read More

Set. add() - HackerRank Solution

 Set. add() - HackerRank Solution

 If we want to add a single element to an existing set, we can use the .add() operation.

It adds the element to the set and returns 'None'.

Example

>>> s = set('HackerRank')
>>> s.add('H')
>>> print s
set(['a', 'c', 'e', 'H', 'k', 'n', 'r', 'R'])
>>> print s.add('HackerRank')
None
>>> print s
set(['a', 'c', 'e', 'HackerRank', 'H', 'k', 'n', 'r', 'R'])

Task

Apply your knowledge of the .add() operation to help your friend Rupal.

Rupal has a huge collection of country stamps. She decided to count the total number of distinct country stamps in her collection. She asked for your help. You pick the stamps one by one from a stack of  country stamps.

Find the total number of distinct country stamps.

Input Format

The first line contains an integer , the total number of country stamps.
The next  lines contains the name of the country where the stamp is from.

Constraints

Output Format

Output the total number of distinct country stamps on a single line.

Sample Input

7
UK
China
USA
France
New Zealand
UK
France 

Sample Output

5

Explanation

UK and France repeat twice. Hence, the total number of distinct country stamps is  (five).

Set. add() Solution

N=int(())
count=set()
for i in range(N):
    country=input()
    count.add(country)
print(len(count))

Explanation-  We have earlier used sets in many HackerRank problems using the earlier learned concepts we will declare a set count. Add( ) method is joined to append elements in the set count. As we all know that set stores only unique element we can directly find the length of the set count to get the output.

To read about sets follow this link-  https://www.geeksforgeeks.org/sets-in-python/

Read More