## Initialize Data
- overall
```
df.shape
df.info()
df.describe()
df.dtypes
```
- duplicate and null
```
df.duplicated().sum()
df.isnull().sum()
```
- categorical data
```
df[cat_cols].value_counts(normalize=True)
df.groupby(cat_col).unique()
pd.crosstab(planes["Source"], planes["Destination"]) #by default, it counts data
pd.crosstab(planes["Source"], planes["Destination"], values=planes["Price"], aggfunc="median")
```
- Numerical data
```
sns.histplot(data=df,x='revenue',binwidth=.1)
sns.kdeplot(data=df,x='revenue',fill=True,hue='region')
```
## Validate Data
- update types
```
df['year']=df['year'].astype(int)
df.select_dtypes(include='number').columns
```
| type | pandas |
| ---------- | ------ |
| string | str |
| integer | int |
| float | float |
| dictionary | dict |
| list | list |
| boolean | bool |
- validate categorical data
```
~df['genre'].isin(['friction','non-friction'])
df['salary'].str.replace(',','').astype(int)
```
- validate numerical data
- handle skewed data and outliers [[Numerical Features Data Process]]
```
sns.boxplot(data=df,x='revenue',y='genre')
```
## Summarize Data
- aggregate data
```
df.groupby('year')['revenue'].agg(['mean','std'])
df.agg({'rating':['mean','sum'],'year':['median','std']})
df.groupby('year').agg(
mean=('rating','mean'),
std_rating=("rating", "std")
)
```
| aggregate | function |
| --------- | ----------------- |
| sum | .sum() |
| count | .count() |
| min/max | .min()/.max() |
| central | .mean()/.median() |
| variance | .var()/.std() |
- visualize categorical data
```
sns.barplot(data=df,x='genre',y='revenue',hue='year',estimator=np.std) #by default is np.mean
```
## Data Clean
- missing data
- drop missing value (5% or less)
- replace with mean, median, mode (depends on distribution)
- replace by subgroup
```
salaries_dict = salaries.groupby("Experience")["Salary_USD"].median().to_dict()
salary['Salary_USD']=salary['Salary_USD'].fillna(salary['Experience].map(salaries_dict))
```
- convert categorical data
string explore
```
df['title'].str.contains('scientist|AI|^data')#start with data
df['title]=np.select([contidions],[values],df['title'])
```
visualize frequency
```
sns.countplot(data=salary,x='year')
```
- transform numerical data
```
df['total_salary']=df['salary'].tranform('cumsum')
df['total_salary']=df['salary'].tranform(lambda x: x/2)
```
- handle outliers
```
df['salary'].quantile(0.75)
iqr = df['salary'].quantile(0.75)-df['salary'].quantile(0.25)
upper = df['salary'].quantile(0.75) + 1.5*iqr
```
## Relationship in Data
- pattern over time
```
divorce = pd.read_csv("divorce.csv", parse_dates=["divorce_date", "marriage_date"])
df['date']=pd.to_datetime(df['date'])
df['month']=df['date'].dt.month/weekday
sns.lineplot(data=df,x='month',y='revenue')
```
- correlation
```
corr = df.corr(numeric_only=True) #model linear relationship
sns.heatmap(corr,annot=True)
sns.scatterplot(data=df,x='revenue',y='height') #some relationship are not linear
sns.pairplot(data=df,
vars=["income_man", "income_woman", "marriage_duration"]) #plot scatterplot for all pairs of numerical data
```
- factor relationship
```
sns.kdeplot(data=divorce, x="marriage_duration", hue="education_man", cut=0, cumulative=True)
sns.scatterplot(data=df,x='revenue',y='height',hue='gender')
```
## Turn EDA into Action
- description analysis
```
df['price'].quantile(0.25)
labels = ["Economy", "Premium Economy", "Business Class", "First Class"]bins = [0, twenty_fifth, median, seventy_fifth, maximum]
df['category']=pd.cut(df['price'],labels=labels,bins=bins)
```