ggerganov/kbd-audio

Tools for capturing and analysing keyboard input paired with microphone capture


repo name	ggerganov/kbd-audio
repo link	https://github.com/ggerganov/kbd-audio
homepage	https://ggerganov.github.io/jekyll/update/2018/11/24/keytap.html
language	C++
size (curr.)	313 kB
stars (curr.)	2457
created	2018-08-27
license	MIT License

kbd-audio

This is a collection of command-line and GUI tools for capturing and analyzing audio data.

Keytap

The most interesting tool is called keytap - it can guess pressed keyboard keys only by analyzing the audio captured from the computer’s microphone.

Check this blog post for more details:

Keytap: description and some random thoughts

Video: short demo of Keytap in action

Keytap2

The keytap2 tool is another interesting tool for recovering text from audio. It does not require training data - instead it uses statistical information about the frequencies of the letters and n-grams in the English language. The tool is still in development, but you can see a short demonstration here:

Video: Keytap2 - recovering text from typing sound (7:50)

CTF: can you guess the text being typed?

Build instructions

Dependencies:

SDL2 - used to capture audio and to open GUI windows libsdl

[Ubuntu]
$ sudo apt install libsdl2-dev

[Mac OS with brew]
$ brew install sdl2

FFTW3 (optional) - some of the helper tools perform Fourier transformations fftw

Linux and Mac OS

git clone https://github.com/ggerganov/kbd-audio
cd kbd-audio
git submodule update --init
mkdir build && cd build
cmake ..
make

Windows

(todo, PRs welcome)

Tools

Short summary of the available tools. If the status of the tool is not stable, expect problems and non-optimal results.

Name	Type	Status
record	text	stable
record-full	text	stable
play	text	stable
play-full	text	stable
view-gui	gui	stable
view-full-gui	gui	stable
keytap	text	stable
keytap-gui	gui	stable
keytap2	text	development
keytap2-gui	gui	development
-	extra	-
guess_qp	text	experiment
guess_qp2	text	experiment
key_detector	text	experiment
scale	text	experiment
subreak	text	experiment
key_average_gui	gui	experiment

Tool details

record-full

Record audio to a raw binary file on disk
```
./record-full output.kbd [-cN]
```
play-full

Playback a recording captured via the record-full tool
```
./play-full input.kbd [-pN]
```
record

Record audio only while typing. Useful for collecting training data for keytap
```
./record output.kbd [-cN] [-CN]
```
play

Playback a recording created via the record tool
```
./play input.kbd [-pN]
```
keytap

Detect pressed keys via microphone audio capture in real-time. Uses training data captured via the record tool.
```
./keytap input0.kbd [input1.kbd] [input2.kbd] ... [-cN] [-CN] [-pF] [-tF]
```
keytap-gui

Detect pressed keys via microphone audio capture in real-time. Uses training data captured via the record tool. GUI version.
```
./keytap-gui input0.kbd [input1.kbd] [input2.kbd] ... [-cN] [-CN]
```
**Live demo (WebAssembly threads required) **
keytap2-gui (work in progress)

Detect pressed keys via microphone audio capture. Uses statistical information (n-gram frequencies) about the language. No training data is required. The ‘recording.kbd’ input file has to be generated via the record-full tool and contains the audio data that will be analyzed. The ‘n-gram.txt’ file has to contain n-gram probabilities for the corresponding language.
```
./keytap2-gui recording.kbd n-gram.txt
```
view-full-gui

Visualize waveforms recorded with the record-full tool. Can also playback the audio data.
```
./view-full-gui input.kbd [-pN]
```
view-gui

Visualize training data recorded with the record tool. Can also playback the audio data.
```
./view-gui input.kbd [-pN]
```