Introduction
If you are into Data Science or you love to analyze the data than you would understand that, how important it is to analyze the data before we start working on it. The first step before we start analyzing the data is 'Data Eyeballing'. Later we analyze the data by visualizing it using a library like Matplotlib, Pandas, etc. And after, writing many lines of codes finally we will get a few charts with which we will be able to get some insights about the data. But what if can get a detailed description of our dataset in an HTML file just by writing a few lines of codes.
To achieve this, we will use a library named 'Pandas Profiling'. Pandas profiling is an open-source python module with which we can do exploratory data analysis just by writing a few lines of code. It allows the user to generate an HTML file of all the data insights, without writing any extra line of code for that. In short, this library saves us from all the work of visualizing and understanding the data. So let's straight away dive into this module and learn how can we use it to analyze the data-
Prerequisite
Before we move further I hope that you have python3 installed in your machine. If not then follow this link to download the latest version of python-https://www.python.org/downloads/
How to make it work
The first step towards using this module is to install pandas-profiling in our machine. For that, we will type-
pip install pandas-profiling
The next step is to create a folder that contains one .py extension file and your dataset. For this example, I'm using the world dataset which contains some NaN values, which later help us to explore the features of Pandas Profiling.
World data is very handy as it comes with the installation of MySQL. You can choose any dataset but make sure that it has headers so that it will help you while reviewing your generated report.
Then, we generate the report using these commands by running them on any editor like VS code that can run python code:
import pandas as pd
from pandas_profiling import ProfileReport
df=pd.read_csv('country.csv')
print(df)
#generate a report
profile=ProfileReport(df)
profile.to_file(output_file='index.html')
Here we are with an output_file named as 'index.html' which contains a detailed analysis of the entire dataset.
In case you fumbled upon error like Pandas_profiling module not found even after installing pandas_profiling on your machine then try saving the project folder inside your python3 folder.
Well, congrats you have successfully created an EDA in less than 5 minutes. If you want to see our dataset report click this link- https://kartikeydwi1997.github.io/EDA/.
In case you want to create a simplified short report, you can use the minimum mode of pandas profiling that was introduced in version 2.4. You can check the version installed with the command:
pandas_profiling.version._version_
With the minimum mode, a simplified report can be generated with less information as compared to the full report which we generated earlier. The commands to be used are:
profile=ProfileReport(df, minimal=True)
profile.to_file(output_file='index.html')
Pandas Profiling Advantages
- It provides a detailed analysis of your dataset- starting from the description of each column with its basic property like mean, max, and min values, etc.
- Uses almost all types of plots to help you to get the insights of the data.
- Describes all correlations.
- Another big advantage of generating this report is that it shows a warning at the beginning. It gives all information about columns that contain NaN values, zero values, categorical and non-categorical columns with high cardinality, etc. All this helps a lot in data pre-processing before applying any machine learning model.
Pandas Profiling Disadvantages
The main disadvantage of pandas profiling is its use for large datasets. With the increase in the size of the dataset, the time to generate the report also increases hence it will require high computational power to generate reports of huge datasets.
To avoid such scenarios, we can take a sample of a large dataset (say 10000 rows) which represents the entire dataset. The command to get the random sample of a large dataset is:
profile=ProfileReport(df.sample(n=10000))
profile.to_file(output_file='index.html')
I hope this blog would have helped you to get some insights into pandas profiling. Please feel free to comment your thoughts on this.
