Hugging Face Transformers: Reformer, ElectraForSequenceClassification, ONNX conversion script

# Reformer (@patrickvonplaten) - Added a new model "Reformer": https://arxiv.org/abs/2001.04451 to the library. Original trax code: https://github.com/google/trax/tree/master/trax/models/reformer was translated to PyTorch. - Reformer uses chunked attention and reversible layers to model sequences as long as 500,000 tokens. - Reformer is currently available as a casual language model and will soon also be available as encoder only ("Bert"-like) model. - Two pretrained weights are uploaded: https://huggingface.co/models?search=google%2Freformer - https://huggingface.co/google/reformer-enwik8 is the first char lm in the library # Additional architectures - The `ElectraForSequenceClassification` was added by @liuzzi # Trainer Tweaks and fixes (@LysandreJik, @julien-c ) ## TPU (@LysandreJik): - Model saving, as well as optimizer and scheduler saving mid-training were hanging - Fixed the optimizer weight updates ## Trainer (@julien-c) - Fixed the `nn.DataParallel` support compatibility for PyTorch `v1.5.0` - Distributed evaluation: SequentialDistributedSampler + gather all results - Move model to correct device - Map optimizer to correct device after loading from checkpoint (@shaoyent) # QOL: Tokenization, Pipelines - New method for all tokenizers: `tokenizer.decode_batch`, to decode an entire batch (@sshleifer) - the NER pipeline now returns entity groups (@enzoampil) # ONNX Conversion script (@mfuntowicz) - Added a [conversion](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_graph_to_onnx.py) script to convert both PyTorch/TensorFlow models to ONNX. - Added a [notebook](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb) explaining how it works # Community notebooks We've started adding community notebooks to the repository. Three notebooks have made their way into our codebase: - [Training T5 on TPU](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) @patil-suraj - [Fine-tuning T5](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb) - [Fine-tuning DialoGPT](https://github.com/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb) @ncoop57 # Predict stage for GLUE task, easy submit to gluebenchmark.com -Adds predict stage for glue tasks, and generate result files which can be submitted to gluebenchmark.com (@stdcoutzyx) # Fixes and improvements - Support flake8 3.8 (@julien-c) - Tests are now faster thanks to using dummy smaller models (@sshleifer) - Fixed the eval loss in the trainer (@patil-suraj) - Fixed the `p_mask` in SQuAD pre-processing (@LysandreJik) - Github Actions pytorch test are no longer pinned to `torch==1.4.0` (@mfuntowicz) - Fixed the multiple-choice script with overflowing tokens (@LysandreJik) - Allow for `None` values in `GradientAccumulator` (@jarednielsen, improved by @jplu) - MBart tokenizer saving/loading id was fixed (@Mehrad0711) - TF generation: Fix issue for batch output generation of different output length.(@patrickvonplaten) - Fixed the FP-16 support in the T5 model (@patrickvonplaten) - `run_language_modeling` fix: actually use the `overwrite_cache` argument (@borisdayma) - Better, version compatible way to get the learning rate in the trainer (@rakeshchada) - Fixed the slow tests that were failing on GPU (@sshleifer, @patrickvonplaten, @LysandreJik) - ONNX conversion tokenizer fix (@RensDimmendaal) - Correct TF formatting to exclude LayerNorms from weight decay (@oliverastrand) - Removed warning of deprecation (@Colanim) - fix no grad in second pruning in run_bertology (@TobiasLee)