CS 540 Introduction to Artificial Intelligence: Natural Language Processing, Study notes of Artificial Intelligence

Download CS 540 Introduction to Artificial Intelligence: Natural Language Processing and more Study notes Artificial Intelligence in PDF only on Docsity!

CS 540 Introduction to Artificial Intelligence

Natural Language Processing

University of Wisconsin-Madison Fall 2022

Announcements

• Homeworks :
  • HW3 in progress.
• Class roadmap: Tuesday, Sept. 27 NLP Thursday, Sept. 29 ML Intro Tuesday, Oct. 4 ML Unsupervised I Thursday, Oct. 6 ML Unsupervised II Tuesday, Oct. 11 ML Linear Regression Machine Learning

Why is it hard? Many reasons:

• Ambiguity: “ We saw her duck”. Several meanings.
• Non-standard use of language
• Segmentation challenges
• Understanding of the world
  • “Bob and Joe are brothers”.
  • “Bob and Joe are fathers”.

Approaches to NLP A brief history

• Symbolic NLP: 50’s to 90’s
• Statistical/Probabilistic: 90’s to present
  • Neural: 2010’s to present Lots of progress! Lots more to work to do ELIZA program

Language Models

• Basic idea: use probabilistic models to assign a probability to a sentence
• Goes back to Shannon
  • Information theory: letters

Training The Model Recall the chain rule

• How do we estimate these probabilities
  • Same thing as “training”
• From data?
  • Yes, but not directly: too many sentences.
  • Can’t estimate reliably.

k=0: Uni gram Model

• Full independence assumption:
  • (Present doesn’t depend on the past)
• Example (from Dan Jurafsky’s notes) fifth, an, of, futures, the, an, incorporated, a, a, the, inflation, most, dollars, quarter, in, is, mass thrift, did, eighty, said, hard, 'm, july, bullish that, or, limited, the

k=1: Bi gram Model

• Markov Assumption:
  • (Present depends on immediate past)
• Example: texaco, rose, one, in, this, issue, is, pursuing, growth, in, a, boiler, house, said, mr., gurria, mexico, 's, motion, control, proposal, without, permission, from, five, hundred, fifty, five, yen outside, new, car, parking, lot, of, the, agreement, reached this, would, be, a, record, november

n- gram Training Issues:

• 1. Multiply tiny numbers?
  • Solution : use logs; add instead of multiply
• 2. n-grams with zero probability?
  • Solution : smoothing Dan Klein

For issue 2 , back-off methods

• Use n-gram where there is lots of information, r- gram (with r << n ) elsewhere. (trigrams / bigrams) Interpolation
• Mix different models: (tri- + bi- + unigrams) Other Solutions: Backoff & Interpolation

Vocabulary: open vs closed

• Possible to estimate size of unknown vocabulary
  • Good-Turing estimator
• Originally developed to crack the Enigma machine

Evaluating Language Models How do we know we’ve done a good job?

• Observation
• Train/test on separate data & measure metrics
• Metrics :
  • 1. Extrinsic evaluation
  • 2. Perplexity

Intrinsic Evaluation: Perplexity Perplexity is a measure of uncertainty Lower is better! Examples:

• WSJ corpus; 40 million words for training:
  • Unigram: 962, Bigram 170, Trigram 109

Further NLP Tasks Language modeling is not the only task. Two further types:

1. Auxilliary tasks: - Part-of-speech tagging, parsing, etc. 2. Direct tasks: - Question-answering, translation, summarization, classification (e.g., sentiment analysis)