by Albert Major

Artificial neural networks (ANNs)

• Traditional computers use serial processing (one task at a time)
• Brains and neural networks use parallel processing (multiple tasks simultaneously)

1. Two Main Approaches to Understanding Cognition:
   1. Knowledge-based Approach:
      • Traditional method using formal algorithms and symbols
      • Works best for well-defined problems with clear solutions
      • Uses logical rules and symbols (like in programming)
   2. Behavior-based Approach:
      • Uses networks that learn from experience
        • (e.g. “training” language models)
      • Better for handling messy, unclear problems
      • Doesn't need step-by-step planning
      • Focus is on how the network behaves overall // finding a set of connection weights between neurons that produce the appropriate behaviour
2. Artificial Neural Networks (ANNs) in Detail:

Basic Components:

• Nodes: The basic computing units

  • Nodes activate when they receive enough input

• Links: Connections between nodes

• Weights: How strong the connections are

  

  example of components

Historical Development:

How learning in the brain works:

• Hebb's Rule: "Cells that fire together, wire together"

  • When neurons activate together repeatedly, their connection gets stronger
  • This explains learning and memory formation
  • Introduces concepts of cell assemblies (small groups of self-stimulating neurons)

• Key concepts:

  • Synaptic plasticity - ability of synapses to strengthen or weaken
  • Long-term potentiation - persistent strengthening of synapses

• Hebb describes two types of cell groupings:

  • Cell assembly — small group of neurons that repeatedly stimulate themselves
  • Phase sequence — set of cell assemblies that activate each other

  A higher-order concept such as “Apple” can be the phase sequence of cell assemblies coding for qualities such as “round”, “red”, and “edible”

Types of Neural Networks:

1. Single-layer Perceptron (Rosenblatt, 1958)

   One of the first artificial neural networks

   • What it could do:
     • Recognize basic patterns like straight lines
     • Tell if a line was vertical (|) or horizontal (─)
     • Learn from its mistakes
   • Example of a single-layer, supervised, feed-forward network
   • Focused on mimicking real biological function
   • Had the ability to learn from experience
   • Earliest perceptron: artificial retina
     • Recognition of simple visual patterns (vertical and horizontal line)
     • Producing desired output for each different type of visual pattern (supervised learning)
   • Major flaw: too simple to distinguish certain more complex patterns (non-linear XOR function)

2. Multi-layer Perceptron:

   • More advanced than single-layer
   • Uses Backpropagation or the Generalized Delta Rule for learning
   • Can handle more complex tasks

3. NETTalk

   • Converts written English to spoken English
   • Input: written letters of the alphabet
   • Output: pronunciation of sounds represented by the letters (fed into a speech synthesizer)
   • Achieved 95% accuracy on training data
   • 75% accuracy on new data
   • Resilient to degradation (simulated damage by randomizing weights within range of old weights)

4. Simple Recurrent Networks (Elman, 1990)

   • Also called an Elman Network
   • Can handle sequential information
   • Good for tasks like predicting next words

Deep Learning Networks

What is “deep” about deep learning networks?

They use multiple processing layers to learn increasingly complex features, unlike shallow networks that use just one layer.

• Uses multiple layers of processing between input and output laye
• Each deeper layer learns more complex features (lines → shapes → objects)
• Requires large amounts of training data to learn effectively
• Can automatically learn important features without human guidance
• Shows significantly better performance than shallow networks on complex tasks

Evaluation of ANNs:

• Biological Plausibility: some similarity to biological neural networks, but also important differences
• Graceful degradation: gradual decrease in performance with increased damage to the network
• Pattern completion: ability to “fill in the blanks” when presented with incomplete input
• Interference: difficulty distinguishing similar patterns (which humans may also have)
• Generalization: ability to generalize to unseen patterns
• Convergent dynamics: ANNs reach stable state, real neural networks are oscillatory and chaotic
• Stability-plasticity dilemma: should be able to learn novel patterns, while maintaining old ones
• Catastrophic interference: old patterns are forgotton when learning new ones