Topics Covered in CL1, Reorganized

This is a (selective) summary of topics covered in Computational Linguistics 1 (Fall 2019), organized in a way that corresponds to the flow of CL2 (Spring 2020). The correspondence is rough, not exact; for example, applications of NLP are discussed throughout.

---

Intro

• Overview of NLP
  • Levels of analysis: morphology, POS, syntax, semantics, discourse
  • Types of ambiguity
  • Word frequency, sparsity, and Zipf's law
  • Social impact

  ---

  Language at the word level

• Distributional semantics of words
  • Co-occurrence matrices (word-word, word-document)
  • (P)PMI
  • Similarity metrics (cosine, Dice, Jaccard, etc.)
  • Smoothing
  • Semantic similarity
  • Word senses, WordNet, WSD

• Linear models and loss functions
  • Loss functions
  • Regularization
  • Optimization via gradient descent
  • Support vector machines

• Text classification
  • Feature representations
  • Logistic regression
  • Neural network basics and backpropagation

• Embeddings
  • Word2vec
  • Evaluating embeddings via similarity
  • Bias in embeddings
  • BERT (introduced with reading comprehension)

  ---

  Modeling structure

• Sequence models
  • N-gram models, perplexity, Shannon game, smoothing
  • RNNs, LSTMs
  • Sequence labeling; Viterbi, ILP, structured perceptron, RNN
  • Encoder/decoder model
  • Attention models

• Syntactic Parsing
  • Role of grammar in NLP
  • CFGs and dependency representations
  • Universal dependencies
  • Dependency parsing
  • Non-projective structures
  • Stack LSTMs

  ---

  Modeling meaning

• Semantics: Representations and Parsing
  • Logical forms and lambda calculus basics
  • CCG and rule-rule correspondence
  • CCG induction (GENLEX)
  • NL interfaces to databases

  • Under the hood in neural models
    • Bias/fairness
    • Interpretability of neural models

• Semantics: Grounding
  • Lexical meaning via grounding
  • Language and vision
  • Grounding with dialogue
  • Grounding and spatial language
  • Pragmatics (brief Rational Speech Acts)
  • Connecting language and action (cf. imitation/reinforcement learning)

  ---

  Applications

• Data collection and annotation
  • Kinds of data and annotation
  • Inter-annotator agreement (Cohen's kappa, Krippendorff's alpha)
  • Annotation process and quality control
  • Sharing of data
  • Crowdsourcing for data collection, evaluation, and experimentation

• Evaluation/Measurement of NLP systems
  • Reliability
  • Types of Validity

• Machine translation
  • Modern history of MT
  • Evaluation: WER, BLEU, METEOR
  • Neural MT

• Question answering
  • Factoid QA
  • Reading "comprehension"
  • QA in Quizbowl
  • Question generation

• Sequential decision problems
  • Imitation learning and DAGGER
  • Application to sequence labeling
  • Learning to search
  • Bandits and contextual bandits
  • Reinforcement learning