Natural Language Processing (NLP) is a major research field of AI, and to most developers, it sounds like a miracle. Lately, I have developed an interest in this field since the noticeable viral news of the GPT-3 model. I decided to learn to make use of it as a tool before it somehow replaces developer jobs in the future, as predicted by many illustrious figures. However, the more I study it, the less I feel I know. There is too much background knowledge to acquire before understanding each word in the GPT-3 paper. Below is a quick summary of the works behind the scenes that will hopefully be useful to developers like me who want to make a leap to catch up with AI progress.
List of keywords
It is an inevitable, long, and exhausting journey to ensure we can understand the fairly basic terms below:
- Convolutional Neuron Network, Recurrent Neuron Network, Activation Function, Loss Function, Back Propagation, Feed Forward.
- Word Embedding, Contextual Word Embedding, Positional Encoding.
- Long – Short Term Memory (LSTM).
- Attention Mechanism.
- Encoder – Decoder Architecture.
- Language Model.
- Transformer Architecture.
- Pre-trained Model, Masked Language Modeling, Next Sentence Prediction.
- Zero-shot learning. One-shot learning, Few-shot learning.
- Knowledge Graph.
- BERT, GPT, BART, T5
What Existed Before BERT and GPT?
There was a lot of research and work in the NLP field. Working in the NLP field means solving the following common tasks:
- Tagging Part of Speech.
- Recognising Named Entities.
- Sentiment Classification.
- Question & Answering.
- Text Generation.
- Machine Translation.
- Summarization.
- Similarity Matching.
SpaCy and NLTK are two of the most famous libraries in the NLP field that provide tools, frameworks, and models for solving a few of the tasks mentioned above, but not everything. Each task usually has its own model, and there is no reusing or transferring between models until the Transformer Architecture was published. With the amazing performance and capabilities of the Transformer Architecture, researchers began to consider using this architecture to perform the aforementioned NLP tasks, aiming for a single model that can do it all. The result is the BERT and GPT models, both of which use Transformer technology. Notably, BERT powers the Google search engine, while GPT-3 powers the ChatGPT application. There are also many more applications utilizing these models that can be found around the internet.
Core Challenges in NLP
No matter what method is applied, the challenges that form the NLP field are still the same:
- A computer does not understand words; it understands numbers. Find a method to convert each word in a sentence into a vector (a group of numbers) so that given two words with similar meanings, the corresponding vectors can have a close distance to represent their similarity.
- Given a sentence with many words and of variable length, find a vector that can represent the sentence.
- Given a passage with many sentences and variable lengths, find a vector that can represent the whole passage.
- From a vector of a word, sentence, or passage, find a method to convert it back to words, sentences, or passages. This task, in turn, becomes Machine Translation or Text Summarization.
- From a vector of a word, sentence, or passage, find a method to classify it into various senses/intents. This task, in turn, becomes Sentiment Classification.
- From a vector of a word, sentence, or passage, find a method to calculate the similarity to another vector. This task, in turn, becomes Question & Answering, or Text Generation, or Text Suggestion.
It will be too long to dive into each keyword here, so subscribe, and you will receive the next posts whenever they become available.