Scripts to extract data from the COVID-19 Google Community Mobility Reports
|size (curr.)||233 kB|
mobius - Mobility Report graph extractor
PLEASE READ: As of 16/04/2020 Google have released the data in CSV format. This tool will not be maintained going forward.
For extracting every graph from any Google’s COVID-19 Community Mobility Report (182) into comma separated value (CSV) files. This code is developed at speed on the COVID-19 Community Mobility Report PDF documents published on Friday 3rd of April 2020.
10/04/2020: PDF and SVGs updated for the Friday 10th of April 2020 release of data. 16/04/2020: PDF and SVGs updated for the Wednesday 15th of April 2020 release of data.
We provide the python
requirements.txt file as well as a
poetry setup for
We recommend using a virtual environment before installing dependencies.
To install with
pip install -r requirements.txt
To install with
This project uses
Rtree which in turn depends on
On OSX this can require separate installation:
brew install spatialindex
# Check what report dates are available python ./mobius.py dt # Check if a country is available, with an option to select a date (not specifying a date will show all results) python ./mobius.py ls <DATE> # Download PDF and SVG python ./mobius.py download <COUNTRY_CODE> <DATE> # Process the PDF and SVG python ./mobius.py summary <INPUT_PDF> <OUTPUT_FOLDER> <DATES_FILE> python ./mobius.py full <INPUT_PDF> <INPUT_SVG> <OUTPUT_FOLDER> <DATES_FILE>
DATE_FILE refers to the look up file in the config directory named
dates_lookup_xxxx_xx_xx.csv where the
x mark the release date of the reports you are extracting the data from.
Full command list
Usage: mobius.py [OPTIONS] COMMAND [ARGS]... Downloader and processor for Google mobility reports Options: --help Show this message and exit. Commands: download Download pdf and svg for a given country using the country code dt List all the dates reports are available for full Produce full CSV of trend data from PDF/SVG input pdf List all the PDFs available in the buckets proc Process a given country SVG summary Produce summary CSV of regional headline figures from CSV svg List all the SVGs available in the buckets
Check what dates reports are available for using
Usage: mobius.py dt List the dates reports are available for Options: --help Show this message and exit.
Check for, and download, SVG and PDF files using
mobius.pycommand line tool to list all the available countries for PDF/SVG and then download them with the two helpful commands below.
Usage: mobius.py svg [OPTIONS][DATE] List the SVGs available in the buckets for the given date. If no date given, a list of all available SVGs is returned Options: --help Show this message and exit. <DATE> Lists all the SVGs published on the date Usage: mobius.py pdf [OPTIONS] [DATE] List the PDFs available in the buckets for the given date. If no date given, a list of all available PDFs is returned Options: --help Show this message and exit. Usage: mobius.py download [OPTIONS] COUNTRY_CODE [DATE] Download PDF and SVG for a given country using the country code and date. If no DATE argument given all available PDFs and SBGs for the given country will be downloaded Options: --help Show this message and exit.
Usage: mobius.py summary [OPTIONS] INPUT_PDF OUTPUT_FOLDER DATES_FILE Produce summary CSV of regional headline figures from CSV Options: --help Show this message and exit.
Specify the input pdf file for the individual country as the
and the output folder where you want the CSV to (e.g.
Creates a summary CSV joined to the data extracted from the SVG plots
- Run the
Usage: mobius.py full [OPTIONS] INPUT_PDF INPUT_SVG OUTPUT_FOLDER DATES_FILE Produce full CSV of trend data from PDF/SVG input Options: --help Show this message and exit.
Specify the input pdf/svg file for the individual country as the
and the output folder where you want the CSV to be saved to (e.g.
Pass in a custom the dates lookup file (e.g.
./config/dates_lookup_<date>.csv) - used to
convert coordinates to dates.
<date> is in the
YYYY_MM_DD format and all available dates for
download are available using
mobius dt command.
Creates a full CSV joined to the data extracted from the SVG plots
Command gives a short summary of any discrepancies between the summary figures and the data extracted from svg plots.
(Alternative) Run the
Usage: mobius.py proc [OPTIONS] INPUT_LOCATION OUTPUT_FOLDER [DATES_FILE] Process a given country SVG Options: -f, --folder TEXT If provided will overwrite the output folder name -s, --svgs Enables saving of SVGs that get extracted -c, --csvs Enables saving of CSVs that get extracted -p, --plots Enables creation and saving of additional PNG plots --help Show this message and exit.
Specify the input file for the individual country as the
INPUT_LOCATION and the
output folder where you want the CSV and other files to be saved to (e.g.
Optionally pass in a custom the dates lookup file (e.g.
./config/dates_lookup.csv) - used to convert coordinates to dates.
If you want simple matplotlib PNG plots to save as well as CSV files, use the
Each CSV from
proc will be saved to (
./output/<COUNTRY_CODE>/csv), starting at
1.csv. As of the COVID-19 Community Mobility Reports released on Friday 3rd April 2020, CSV files
6.csv relate to the country-level graphs in the original PDF (pages one and two). Then each set of 6 CSV files (e.g.,
12.csv) will relate to a regional area.
Each set of 6 files follows the order: 1. Retail & recreation 2. Grocery & pharmacy 3. Parks 4. Transit stations 5. Workplaces 6. Residential
G20 CSV Datasets
CSV datasets for G20 countries (except Russia and China) can be found at the Data Science Campus' Google Mobility Reports Data repository.
Creating your own SVG
To create your own SVG from a PDF, we recommend using Affinity Designer. This is because Affinity Designer flattens the SVG, which is required for
mobius.py proc. Affinity Designer also closes point features in SVG files, which other programmes do not. Use the following steps:
- Load in PDF document to Affinity Designer.
- Click Load all pages.
File > Export > SVG (for print)if using version 1.6.3 or
File > Export > SVG (digital - high quality)for version 1.8.3
- Select Area: Whole Document.
- Save the SVG file to (
To run through all available countries see
run_all.sh help for usage.
# Create summary and full CSVs for all countries returned by ./mobius.py svg ./run_all.sh
Any suggestions or issues, please use the Issues template. We welcome collaborators. To help us with this work, fork the repository and issue a Pull Request when you have added a feature, or fixed a bug. Thanks!