Statistics for CSAI

by Albert Major

• Understanding Data in R
  • Data in R is stored in different structures:
    • Vectors are the simplest form, storing a sequence of values of the same type (e.g., numbers, text). They are useful when working with simple lists of data such as a set of exam scores or a list of names.
    • Data Frames are tables containing multiple vectors (columns), where each column can have a different type of data. This makes them ideal for handling structured datasets, like a student database that includes names, grades, and attendance records.
  • Functions in R are tools that perform tasks automatically.
    • Example: mean() calculates the average of a set of numbers.
    • Functions take arguments, which are inputs that define how they work.
    • Some arguments have default values, but you can also specify them explicitly.
    • Understanding how functions work is crucial for manipulating and analyzing data efficiently.
  • Packages in R are collections of functions and data sets that expand what R can do.
    • Example: The Psyc package provides statistical functions like describe().
    • To use a package, it needs to be installed (install.packages("Psyc")) and loaded (library(Psyc)).
    • Many R packages are designed for specialized fields, such as ggplot2 for data visualization and dplyr for data manipulation.
• Exploring and Understanding a Dataset
  • Looking at the dataset:
    • The head() function shows the first few rows of data.
    • The tail() function shows the last few rows.
    • dim() tells you the number of rows (cases) and columns (variables).
    • summary() gives an overview of the dataset, showing key statistics like minimum, maximum, and median values.
    • Inspecting datasets is an essential step before performing any analysis, as it helps identify missing values and irregularities.
  • Missing values in datasets:
    • Sometimes, data is incomplete. R represents missing values as NA.
    • You can check for missing values using is.na().
    • To perform calculations while ignoring missing values, use na.rm = TRUE inside functions like mean() or median().
    • Handling missing data properly is crucial for obtaining accurate statistical results.
• Central Tendency: Finding the “Middle” of Data
  • Mean (Average): The sum of all values divided by the number of values.
    • Example: The average exam grade can be calculated with mean(data$exam, na.rm = TRUE).
    • The mean is affected by extremely high or low values (outliers).
    • Outliers can distort the mean, making it less representative of the dataset.
  • Median: The middle value when all values are arranged from lowest to highest.
    • Example: median(data$exam, na.rm = TRUE).
    • If there are extreme values (e.g., some students score very low), the median is often a better representation than the mean.
    • The median is useful when dealing with skewed data distributions.
  • Mode: The most frequently occurring value.
    • Example: The most common exam grade can be found with modeof(data$exam).
    • The mode is especially useful for categorical data, such as the most common letter grade in a grading system.
• Measuring Spread: Understanding Variability
  • Standard Deviation (SD): Measures how spread out the data is from the mean.
    • A small SD means most values are close to the average.
    • A large SD means values are more spread out.
    • Example: sd(data$exam, na.rm = TRUE).
    • Standard deviation is a key indicator of data consistency.
  • Interquartile Range (IQR): Measures the spread of the middle 50% of the data.
    • It is the difference between the 75th percentile (upper quartile) and the 25th percentile (lower quartile).
    • Helps ignore extreme values.
    • IQR is often used in boxplots to visualize data spread.
• Distribution Shape: How Data is Spread Out
  • Histograms:
    • A histogram is a bar chart that shows how often different values appear in a dataset.
    • Example: hist(data$exam) creates a histogram of exam scores.
    • Helps to see if most scores are clustered in one area or spread out.
    • Histograms are a fundamental tool in exploratory data analysis.
  • Skewness:
    • Measures if the data is balanced or leaning towards one side.
    • Positive skew: Most values are low, with a few high outliers (e.g., salary data where a few people earn millions).
    • Negative skew: Most values are high, with a few low outliers (e.g., exam scores where most students score well but some fail).
    • Example: skew(data$exam).
    • Skewness affects the interpretation of mean and median.
  • Kurtosis:
    • Measures whether data is peaked (many values close to the average) or flat (values spread out evenly).
    • Leptokurtic: A sharp peak (most values are close to the mean).
    • Platykurtic: A flat distribution (values spread more evenly).
    • Example: kurtosi(data$exam).
    • Understanding kurtosis helps in analyzing outliers and extreme values.
• Comparing Groups in a Dataset
  • Grouping Data to Compare Differences:
    • Example: Comparing exam scores between students who completed practical exercises and those who didn’t.
    • describeBy(data$exam, data$all_PE): Gives separate statistics for each group.
    • Results showed that students who completed exercises scored higher on average.
    • Group comparisons help in identifying factors that influence outcomes.