Statistics for CSAI

by Albert Major

• PE1 Solutions Review (Understanding Data in R):
  • Understanding Reliability and Validity:
    • Reliability measures consistency – if you take a measurement multiple times, will you get the same result? A highly reliable tool produces the same result under the same conditions repeatedly.
    • Validity measures accuracy – is the measurement capturing what it's supposed to measure? A measurement can be reliable but not valid if it consistently gives the wrong results.
    • Example: A broken scale might always show a person’s weight as 70 kg (reliable but not valid), while a well-functioning scale shows the correct weight every time (both reliable and valid).
    • Different types of validity exist, such as construct validity (does the test measure what it intends to?), content validity (does it cover all necessary aspects?), and criterion validity (does it correlate with real-world outcomes?).
  • Handling Data in R:
    • Checking dataset size: Use nrow(data) to see how many rows (movies) are in the dataset and ncol(data) to check the number of columns (variables).
    • Summarizing data: summary(data) provides a quick overview of key statistics for all variables, including mean, median, minimum, and maximum values.
    • Counting specific categories: Use table(column_name) to count how many times a specific category appears (e.g., how many movies belong to the drama genre).
    • Filtering data: Extract rows that meet a certain condition (e.g., movies with high ratings) using subset(data, condition). This allows users to analyze specific subsets of their dataset.
    • Changing text variables into categories: Convert a character variable into a factor using as.factor(), which makes it easier to analyze and summarize categorical data efficiently.
    • Detecting missing values: Use is.na(data) to identify missing values, which is crucial in data analysis to avoid errors.
• IMDb Movie Data Exploration:
  • Dataset Overview:
    • Contains 2,513 movies with at least 100,000 votes each, ensuring a robust dataset with well-known films.
    • Includes details like title, release year, IMDb rating, number of votes, and assigned genres.
    • Each movie may belong to multiple genres, such as both drama and thriller, which allows for richer data analysis.
  • Finding the Highest-Rated Movies:

    • The dataset contains a column called rating, which represents the IMDb score of each movie.

    • We can sort the dataset to find the highest-rated movies:

      ordered_movies <- data[order(-data$rating), ]
      head(ordered_movies, 10)
      

    • This will display the top 10 highest-rated movies.

    • Example results: The Shawshank Redemption (9.3 rating), The Godfather, and other critically acclaimed films.

    • Additional filtering can be done, such as finding only high-rated comedies or movies released after a certain year.

  • Visualizing Data with Scatter Plots:

    • A scatter plot helps us see how movie ratings change over release years:

      plot(data$year, data$rating, xlab="Release Year", ylab="IMDb Rating", main="Movie Ratings Over Time", col="blue", cex=0.5)
      

    • Observations:

      • Older movies tend to have only high ratings because bad movies from the past get forgotten.
      • Recent movies have a wider range of ratings, indicating that both great and poorly received movies receive significant attention.

    • Customizing scatter plots:

      • Change dot size: cex=0.3 makes dots smaller for better visibility.
      • Change colors: col="red" can highlight certain categories.
      • Add trend lines: abline(lm(data$rating ~ data$year), col="black") adds a regression line.
  • Understanding Rating Distribution (Histograms):

    • A histogram shows how movie ratings are distributed:

      hist(data$rating, breaks=20, col="pink", main="Distribution of IMDb Ratings", xlab="Rating", ylab="Frequency")
      

    • Observations:

      • The ratings form a bell-shaped curve (similar to a normal distribution).
      • A slightly longer tail on the lower end suggests that there are more bad movies than we would expect in a perfect normal distribution.
      • We can modify bin sizes with breaks=50 for a more detailed view.

    • Adding density lines:

      hist(data$rating, probability=TRUE, col="lightblue", main="Density of Ratings")
      lines(density(data$rating), col="red", lwd=2)
      

      This helps us see the smooth distribution of data rather than just bar counts.