Introduction to AI 2

by Albert Major

1. Introduction: What Is This Lecture About?

• Main Topic:

  How AI and cognitive science approach language, focusing on the difference between grammars (rule-based systems) and language models (probability-based systems).

• Why It Matters:

  Understanding these approaches is key for natural language processing (NLP), programming languages, and linguistics.

  

  This slide introduces the central debate between Chomsky (Nativist) and Skinner (Behaviorist). Skinner believed language is learned through stimulus-response associations and reinforcement, similar to how animals learn behavior. Chomsky rejected this, arguing that humans are born with an innate Language Acquisition Device (LAD) and can generate and understand sentences they’ve never heard before, showing language is based on internal rules (grammars), not just past input. This contrast lays the foundation for understanding language models (behaviorist-inspired) vs. grammars (nativist-inspired) in AI.

2. Two Big Theories About How We Learn Language

A. Skinner and Behaviorism

• Who: B.F. Skinner (psychologist, behaviorist)
• Theory:
  • All learning (including language) is based on forming associations between stimuli (inputs) and responses (outputs).
  • Reinforcement (reward/punishment) strengthens these associations.
  • Example: Rats learn to press a lever for food (stimulus → response → reward).
• Applied to Language:

  • Kids learn words and sentences by hearing them and being rewarded (e.g., parent praise).

  • N-gram models: Predict the next word based on previous words (e.g., in “I am going to the...”, the model predicts “store”).

  • Language is viewed as repeating what’s been rewarded in the past.

    

    This slide explains key limitations of n-gram models, which are statistical language models. First, due to data sparsity, there are too many possible n-gram combinations to assign reliable probabilities. Second, they lack awareness of lexical categories like noun or verb, so they can't enforce grammatical rules. As a result, n-gram models cannot generate all valid sentences or exclude incorrect ones. The alternative is to use a grammar, which defines a set of rules to generate all legal and no illegal sentences in a language. This connects back to Chomsky's view of language as rule-based.

    

    This slide clarifies what **“grammar”** means in the context of AI and linguistics. The **green highlighted definition**—“a set of rules governing what strings are valid or allowable in a language or text”—is the one used in computing and Chomskyan linguistics. It contrasts with the **red definition**, which is the school-taught, **prescriptive view** (e.g., "this is bad grammar")—but that’s *not* what this lecture focuses on. Instead, we care about **formal rules** that define what structures are possible, not what is stylistically “correct.” This distinction is essential for understanding grammars in NLP and computer science.

B. Chomsky and Generative Grammar

• Who: Noam Chomsky (linguist, cognitive scientist)
• Theory:
  • Children learn language in ways that cannot be explained by input-output associations alone.
  • Humans create and understand sentences they’ve never heard before—shows creativity beyond repetition.
  • Language Acquisition Device (LAD):
    • Humans are born with a built-in mechanism for learning language.
    • Language ability is innate and evolved in humans.
  • Main Argument Against Behaviorism:
    • Language is not just about repeating; it’s about following abstract rules.
    • Kids “know” what is and isn’t a valid sentence, even if they’ve never heard it.

3. Key Concepts: Grammar vs Language Model

grammar defi by mr russy and mr norvy

This slide reviews the difference between formal languages and natural languages, key to understanding grammars in AI. Formal languages (like programming) have strict syntax rules—so any sequence of symbols can be clearly judged as legal or illegal. Natural languages, however, contain ambiguity and lack a definitive set of rules to decide what is or isn’t part of the language. This helps explain why natural language parsing is much harder than parsing code—it’s fuzzier and more flexible, requiring models that can handle uncertainty and context.