Introduction to BERT
BERT NLP Model is one of the most influential NLP models today. In this guide, we will provide an overview of how BERT works and illustrate usage for common NLP tasks with code examples.
Released in 2018 from Google AI, BERT (Bidirectional Encoder Representations from Transformers) pioneered the bidirectional training of transformer models for NLP.
Key technical innovations include:
- Bidirectional self-attention – Looks at both left and right context in all layers
- Masked language modeling (MLM) – Randomly masks input words and predicts them
- Next sentence prediction (NSP) – Predicts relationship between two input sentences
These techniques allow BERT to pre-train a deep bidirectional Transformer model on large unlabeled text corpora. The pre-trained model can then be fine-tuned on downstream NLP tasks to achieve state-of-the-art performance with minimal task-specific engineering.
BERT NLP Model Architecture
Input Layer
(Tokenizer)
|
V
+-------------------------------------+
| BERT |
| (Bidirectional Encoder) |
+-------------------------------------+
|
V
Output Layer
(Task-specific heads)
|
V
PredictionsThe key components are:
- Input Layer: This tokenizes the input text into WordPiece tokens and generates input embeddings for each token.
- BERT Encoder: This is a multi-layer bidirectional Transformer encoder based on the original Transformer model. It uses self-attention and produces contextual token embeddings.
- Output Layer: For different tasks like classification or sequence tagging, task-specific output layers are added on top of BERT. These output layers leverage the contextual embeddings from BERT to make predictions.
- Predictions: The final output from BERT after passing through the task-specific prediction heads. For example, classification labels or tagged token sequences.
At its core, BERT uses the transformer architecture consisting of encoder blocks with self-attention and feedforward layers.
Some key elements of BERT’s architecture:
- Vocabulary with 30,000 WordPiece tokens
- 12 or 24 transformer blocks, 12 self-attention heads
- Hidden layers of size 768 and 3072
- Total parameters: 110M (BASE) or 340M (LARGE)
BERT is pretrained on two unsupervised tasks – masked language modeling and next sentence prediction. This gives the model a deep understanding of language.
Using BERT for Classification(BERT Classification model architecture)
For text classification tasks, BERT leverages pretrained contextual embeddings and adds a classification layer on top:
+---------------------+
| Input Text |
+----------+----------+
|
V
+---------+
| Tokenizer|
+---------+
|
V
+----------------BERT Encoder----------------+
| |
| +------+ +------+ +------+ +------+ +------+|
| |Block | |Block | |Block | |Block | |Block ||
| +------+ +------+ +------+ +------+ +------+|
| |
+--------------------------------------------+
|
V
+---------+
| Pooling |
+---------+
|
V
+-------------------+
|Classification Head|
+-------------------+
|
V
+---------+
| Softmax |
+---------+
|
V
Predicted
SentimentThe steps are:
- The input text is tokenized using BERT’s WordPiece tokenizer.
- The tokenized input goes through the stacked bidirectional Transformer blocks of the BERT encoder.
- An aggregation pooling layer condenses the encoder outputs into a single vector.
- A simple classification head predicts sentiment scores from the pooled vector.
- A softmax layer converts scores to normalized class probabilities.
- The highest probability class is chosen as the predicted sentiment.
Here is sample code to use pretrained BERT for sentiment analysis:
from transformers import BertTokenizer, BertForSequenceClassification
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained("bert-base-uncased")
text = "I really enjoyed this movie!"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
scores = output[0][0].detach().numpy()
scores = torch.nn.functional.softmax(torch.Tensor(scores), dim=0)
print(scores) # [0.15, 0.85] (negative, positive)This loads a pretrained BERT classifier, tokenizes the input text, feeds it through the model and prints predicted sentiment scores.
The same approach can be applied to other classification datasets. We just replace the pretrained model with one finetuned on the target dataset.
Named Entity Recognition using BERT (BERT NER model architecture)
For sequence tagging tasks like named entity recognition (NER), an additional output layer is added to BERT:
+-----------------------+
| Input |
| Sentence |
+-----------+-----------+
|
V
+---------+
| Tokenizer|
+---------+
|
V
+-----------------BERT Encoder------------------+
| |
| +------+ +------+ +------+ +------+ +------+ |
| |Block | |Block | |Block | |Block | |Block | |
| +------+ +------+ +------+ +------+ +------+ |
| |
+------------------------------------------------+
|
V
+-----------------------------+
| NER Prediction Head |
+-----------------------------+
|
V
Predicted Named
Entity TagsThe steps are:
- The input sentence is tokenized using BERT’s WordPiece tokenizer.
- The tokenized words are fed into the stacked BERT encoder blocks to generate contextual embeddings.
- The contextual embeddings are passed to a task-specific prediction head for NER.
- The prediction head outputs an NER tag prediction for each input token.
- The predicted tags label each token with entity types like Person, Location, Organization etc.
The output layer predicts a label for each input token.
Here is sample usage for NER:
from transformers import BertTokenizer, BertForTokenClassification
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = BertForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
text = "Steve Jobs is the CEO of Apple."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
scores = output[0][0].detach().numpy()
predicted_labels = np.argmax(scores, axis=2)
print(predicted_labels) # [4, 8, 10, 10, 4]This loads a pretrained BERT NER model and predicts labels like PERSON, ORG etc for each token.
BERT pushes state-of-the-art on a wide variety of NLP tasks via finetuning.
Important BERT Considerations
| Point | Explanation |
|---|---|
| Casing | BERT is case-sensitive. Use cased models for tasks involving casing. |
| Output | BERT outputs token embeddings. Add task-specific layers on top. |
| Finetuning | Finetune pretrained BERT carefully to avoid catastrophic forgetting. |
| Masking | Masking helps pretraining but actual inputs should not be masked. |
| Truncation | Long inputs may need truncating to fit BERT’s 512 token limit. |
| Regularization | Techniques like dropout help prevent overfitting during finetuning. |
These points should be kept in mind while using BERT effectively for NLP.
Conclusion
In this guide, we discussed the core BERT nlp model and saw examples of using it for text classification and sequence tagging. BERT’s self-attention transformer architecture coupled with pretrained bidirectional understanding of language makes it immensely powerful. At the same time, considerations like masking, token limits and regularization must be handled appropriately. Using BERT as a base, several other transformer models like RoBERTa and ALBERT have built upon with improved techniques. With readable implementations offered by HuggingFace and others, BERT nlp model delivers cutting-edge NLP capabilities to developers with ease.
Frequently Asked Questions
Q: What are the main benefits of BERT nlp model?
A: Bidirectional context, large scale pretraining and self-attention lead to big accuracy improvements on many NLP tasks.
Q: When should BERT nlp model be finetuned vs used as-is?
A: For generic sentence embeddings, pretrained BERT can be used as-is. For task-specific needs, finetuning on target dataset is recommended.
Q: What are good hyperparameters for finetuning BERT nlp model?
A: Low learning rates like 2e-5, 2-4 epochs, batch size of 16-32 generally work well for finetuning BERT nlp models.
Q: How much text data is required for effective finetuning?
A: Typically a few thousand quality samples are sufficient for tasks like text classification. More complex tasks may need larger datasets.
Q: What are some common issues faced using BERT nlp model?
A: Padding vs masking confusion, misuse of WordPiece tokenization, regularization to prevent catastrophic forgetting during finetuning.
