# Hugging Face Transformers v0.5.0 — Release & Upgrade Intelligence
- **Product:** Hugging Face Transformers (https://huggingface.co)
- **Version:** v0.5.0
- **Released:** Feb 11, 2019
- **Impact Score:** 5/10
- **Breaking Changes:** Not verified
- **Canonical Analysis:** https://changelogs.us/transformers/updates/v0.5.0/

## Executive Summary (TL;DR)
Hugging Face Transformers v0.5.0. Source highlight: New pretrained models:

## Improvements
- **Open AI GPT** pretrained on the *Toronto Book Corpus* ("Improving Language Understanding by Generative Pre-Training" by Alec Radford et al.).
- This is a slightly modified version of our previous PyTorch implementation to increase the performances by spliting words and position embeddings in separate embeddings matrices.
- Performance checked to be on part with the TF implementation on ROCStories: single run evaluation accuracy of 86.4% vs. authors reporting a median accuracy of 85.8% with the TensorFlow code (see details in the example section of the readme).
- Performance checked to be on part with the TF implementation on WikiText 103: evaluation perplexity of 18.213 vs. authors reporting a perplexity of 18.3 on this dataset with the TensorFlow code (see details in the example section of the readme).

## Bug Fixes
- **Transformer-XL** pretrained on *WikiText 103* ("Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" by Zihang Dai, Zhilin Yang et al.). This is a slightly modified version of Google/CMU's PyTorch implementation to match the performances of the TensorFlow version by:
- add a `never_split` option and arguments to the tokenizers (@WrRan)
- better handle errors when BERT is feed with inputs that are too long (@patrick-s-h-lewis)
- better layer normalization layer initialization and bug fix in examples scripts:  args.do_lower_case is always True(@donglixp)
- fix learning rate schedule issue in example scripts (@matej-svejda)
- readme fixes (@danyaljj, @nhatchan, @davidefiocco, @girishponkiya )

## Upgrade Command
```bash
python -m pip install 'transformers==0.5.0'
```