Compute summary statistics on a Spark DataFrame using .summary() or dbutils data summaries

Remove outliers from a Spark DataFrame based on standard deviation or IQR

• Use pyspark.sql.functions.stddev to help with this.

Create visualizations for categorical or continuous features

Compare two categorical or two continuous features using the appropriate method

Compare and contrast imputing missing values with the mean or median or mode value

Impute missing values with the mode, mean, or median value

Use one-hot encoding for categorical features

Encoding Categorical Features

Identify and explain the model types or data sets for which one-hot encoding is or is not appropriate.

• categorical (string or object) variables

  Drawbacks/Limitations

• Induces Sparsity

• Limits Split Options for Tree-Based Models

• Inefficiency with High Cardinality Variables

• Obscures Feature Importance

• Increases Computational Costs

Identify scenarios where log scale transformation is appropriate

• When our original continuous data do not follow the bell curve, we can log transform this data to make it as “normal” as possible so that the statistical analysis results from this data become more valid. In other words, the log transformation reduces or removes the skewness of our original data. The important caveat here is that the original data has to follow or approximately follow a log-normal distribution. Otherwise, the log transformation won’t work.
• A variable might undergo a log-scale transformation for several reasons, primarily to simplify the analysis, improve interpretability
  • Dealing with Skewed Distributions
  • Compressing Wide Ranges
  • Making Relationships Linear
  • Stabilizing Variance (Homoscedasticity)
    • Log transformations stabilize variance when a variable's variability increases with its magnitude (heteroscedasticity).