December 27, 2019

267 words 2 mins read



Simple transformer implementation from scratch in pytorch.

repo name pbloem/former
repo link
language Python
size (curr.) 35737 kB
stars (curr.) 287
created 2019-08-15
license MIT License


Simple transformer implementation from scratch in pytorch. See for an in-depth explanation.


The current models are designed to show the simplicity of transformer models and self-attention. As such they will not scale as far as the bigger transformers. For that you’ll need a number of tricks that complicate the code (see the blog post for details).

All models so far are a single stack of transformer blocks (that is, no encoder/decoder structures). It turns out that this simple configuration often works best.


You can clone the code and run the experiments from the root directory. E.g.

python experiments/

Hyperparameters are passed as command line arguments. The defaults should work well. The classification data is automatically downloaded, the wikipedia data is included in the repository.

You should be able to install as a package as well, with

pip install git+

but I haven’t tried this. It’s probably easier to just copy over the code you need. Let me know if you need this for anything and it doesn’t work.


Python 3.6+ is required.

The following should install all requirements pip install torch tb-nightly tqdm numpy torchtext

You may also need pip install future depending on the exact python version.

conda environment

The file environment.yml describes a complete conda environment with all dependencies. After cloning or downloading the project, you create the environment as follows:

conda env create -f environment.yml --name former
conda activate former
comments powered by Disqus