(from ChatGPT)
Data Collection and Loading:
- Collect the dataset that you want to analyze.
- Load the data into your preferred data analysis environment (e.g., Python with libraries like pandas or R).
Initial Data Inspection:
- Display the first few rows of the dataset to get a sense of its structure and the types of data it contains.
- Check for any missing values in the dataset and decide on an approach to handle them (imputation or removal).
Summary Statistics:
- Compute basic summary statistics for numeric features, such as mean, median, standard deviation, minimum, and maximum.
- For categorical features, you can calculate the frequency distribution of each category.
Data Visualization:
- Create various types of plots and visualizations to better understand the data. Use libraries like Matplotlib, Seaborn, or ggplot2 in R.
- For numeric features, consider histograms, box plots, scatter plots, and correlation matrices.
- For categorical features, create bar plots, pie charts, and count plots to visualize the distribution of categories.
Data Distribution Analysis:
- Examine the distribution of numeric features to identify any outliers or skewness in the data.
- Apply data transformations (e.g., log transformation) if needed to make the distribution more suitable for analysis.
Feature Relationships:
- Explore relationships between numeric features using scatter plots or correlation analysis.
- Investigate relationships between categorical features using contingency tables and chi-squared tests.
Feature Engineering:
- Create new features based on existing ones if it adds value to your analysis. For example, you can derive age categories from birth dates or calculate the length of text in a text field.
Grouping and Aggregation:
- Group the data based on relevant categorical features and calculate summary statistics for each group.
- This can help you understand how different groups behave within the dataset.
Hypothesis Testing (if applicable):
- If you have specific hypotheses, perform statistical tests to validate or reject them. For example, t-tests, ANOVA, or chi-squared tests.
Dimensionality Reduction (if needed):
- Use techniques like Principal Component Analysis (PCA) or t-SNE to reduce the dimensionality of the data, especially if you have many features.
Data Quality and Data Cleaning:
- Address any data quality issues identified during the analysis, such as duplicates, inconsistencies, or errors.
Final Documentation:
- Document your findings, insights, and any data preprocessing steps taken.
- Summarize the key points and observations to share with stakeholders or team members.
Iteration and Further Analysis:
- EDA is often an iterative process. Based on your initial findings, you may decide to explore specific aspects of the data in more detail or revisit earlier steps.
Visualization and Reporting:
- Create clear, informative visualizations and reports to communicate your EDA results to stakeholders or team members.
| Featuename | Action | Reason |
|---|---|---|
| Dachmaterial | delete | coralation: 0.025 and 1973 times same value |
| Kellerhöhe, Kellerzustand, Kellerbelichtung, Kellerbereich1, Kellerbereich2 | change | missing values to NA |
| Kellerbereichgroesse1, Kellerbereichgroesse2, KellerbereichgroesseGes, KellerbereichgroesseNau, KellerVollbadezimmer, KellerHalbbadezimmer | change | missing values to 0.0 |
| Kellerbereichgroesse2 | delete | |
| Mauerwerktyp | change | missing values to "Kein" |
| Mauerwerkfläche | change | missing values to 0 |
| Kaminqualitaet | change | missing values to NA |
| Funktionalitaet | change | missing values to mode |
| KuechenQualitaet | change | missing values to mode |
| Elektrik | change | missing values to mode |
| GeringequalitaetFlaeche | delete | because only 29 rows not 0 |
| OffeneVerandaflaeche/ GeschlosseneVerandaflaeche | change | aggregate columns and check correlation with price/holzdeck |
| Poolqualität und Poolfläche | reduce | mit PCA auf eine Dimension |