by Albert Major

Neural Networks and Cognitive Science

Basic Principles

1. Neural Communication
   • Neurons in the brain communicate through signals
   • When a neuron receives enough excitation, it fires (all-or-nothing response)
   • This basic principle inspired artificial neural networks
   • Neural networks process information in parallel, unlike traditional computers that process serially
2. Biological vs. Artificial Networks
   • Biological networks: actual brain neurons communicating
   • Artificial networks: mathematical abstractions of biological networks
   • Components:
     • Nodes (artificial neurons)
     • Links (connections between nodes)
     • Weights (strength of connections)
   • Networks can be organized in layers:
     • Input layer (receives information)
     • Hidden layer(s) (processes information)
     • Output layer (produces final output)

Historical Development

1. Early Development
   • Started in the 1950s with simple perceptrons
   • Could recognize basic visual patterns
   • Limited by inability to solve certain problems (like XOR)
   • Led to first "AI winter"
2. Revival in 1980s
   • Led by Rumelhart and McClelland
   • Published influential "Parallel Distributed Processing" books
   • Developed backpropagation algorithm
   • Established foundation for modern AI and deep learning
3. Modern Era
   • Deep learning revolution
   • Same principles but with many more layers
   • More computational power enables larger networks
   • Applications in language models, vision, etc.

Types of Neural Networks

1. Feed-Forward Networks
   • Information flows in one direction
   • From input through hidden layers to output
   • No connections back to previous layers
   • Good for pattern recognition
2. Recurrent Neural Networks
   • Include connections that loop back
   • Can process temporal sequences
   • Example: Elman Networks
     • Include "context layer" that remembers previous states
     • Can learn grammar patterns just from exposure
     • Demonstrate how temporal information processing works
3. Deep Neural Networks
   • Multiple hidden layers
   • Each layer can detect more complex features
   • Require more computational power
   • Currently driving AI revolution

Learning in Neural Networks

1. Basic Principles
   • Networks learn by adjusting connection weights
   • Learning is gradual and iterative
   • Based on difference between output and desired result
   • Uses backpropagation to adjust weights
2. Key Features
   • Learn from experience
   • Can generalize to new situations
   • Show graceful degradation when damaged
   • Can complete partial patterns
   • Show interference effects like humans

Semantic Networks

1. Basic Concept
   • Nodes represent concepts rather than neurons
   • Connections show relationships between concepts
   • Can be hierarchical
   • Model how knowledge is organized in memory
2. Key Properties
   • Spreading activation between related concepts
   • Hierarchical organization
   • Can explain priming effects
   • Models response times in human cognition