August 18, 2019

434 words 3 mins read



Arxiv’s ML papers network graph and browser

repo name ranihorev/arxiv-network-graph
repo link
language Python
size (curr.) 2378 kB
stars (curr.) 72
created 2018-12-03
license MIT License

MLG - Visual Machine Learning arxiv Graph and Textual explorer

MLG (Machine Learning Graph) is a visual representation of ML researchers and papers, and the connections between them. Each node in the graph (/citations_network) is an author or a paper, and an edge can represent a citation, a reference or a authorship.

Note: There is an old version of the graph (/network) in which edges represent co-authorship of papers, based solely on

Live demo is available at

MLG allows you to:

  1. Search for papers or authors.
  2. Navigate between related papers and authors.
  3. Click on a node to view its list of papers and double click to expand its connections.
  4. Re-organize the network after expanding.

The backend is based on arxiv-sanity but with a lot of modifications:

  • The papers data is collected from and Everything is stored on MongoDB.
  • Rebuilt the Twitter daemon - it now collects tweets from a list of prominent ML accounts, in addition for searching links on Twitter.

The project includes three parts:

  1. / - arXiv text explorer.
  2. /citations_network - The new visual network graph explorer.
  3. /network - The old arXiv visual graph explorer.


Example of the old version:

user interface


$ virtualenv env                # optional: use virtualenv
$ source env/bin/activate       # optional: use virtualenv
$ pip install -r requirements.txt

There is still some legacy code from arxiv-sanity that require some of the packages in the requirement.

Processing pipeline

  1. Install and start MongoDB
  2. Optional - Run to collect all paper from arXiv. Run to collect data from
  3. Create twitter.txt with your Twitter API credentials (values of consumer key and secret, in separate lines). You can also add accounts to the twitter_users.json file.
  4. Run to start background tasks scheduler.
  5. Run the flask server with

Old version - Generating the network graph

After fetching papers from arXiv you can build the network graph by running the notebook graph_generator.ipynb. It will overwrite the static/network_data.json.

Note: Calculating the physics of the network (nodes' position) is very slow. The current hack is to run it once (by changing the physics settings in network.js) and store the calculated positions. I tried using networkX to calculate the positions, however, the results weren’t pleasing…

Running online

If you’d like to run the flask server online (e.g. AWS) run it as python --prod.

You also want to create a secret_key.txt file and fill it with random text (see top of

comments powered by Disqus