Detection Theory; ROC&AUC (Jan 17)

Today we studied the basics of detection theory (slide set 3). In the beginning of the lecture, we implemented a sinusoid detector in Python. The input was a 3-second long audio with sinusoid (beep) of 1000 Hz present in the inteval 1s...2s. The first and last second were blank. In the experiment, we deliberately added Gaussian noise on top of the clean signal, and experimented how well the sinusoid could be detected under noise.

The code is below. You need python sounddevice module, which installs in Anaconda command prompt with "pip install sounddevice". The below code uses a separate module "sin_detect", which is hidden here as it appears in next week exercises.

# Example of playing a noisy sinusoid
# Requires installation of sounddevice module.
# In anaconda: "pip install sounddevice" should do this.

import sounddevice as sd
import numpy as np
from sin_detect import detect
import matplotlib.pyplot as plt

if __name__ == "__main__":
    
    f = 1000    # Play a beep at this Hz
    Fs = 8192   # Samplerate
    sigma = 2   # Stddev of the noise
    
    n = np.arange(3 * Fs) # The sample times
    x = np.sin(2 * np.pi * f * n / Fs)
    
    # Zero out the beginning and the end
    x[:Fs] = 0
    x[2*Fs:] = 0
    
    # Add noise
    x_noisy = x + sigma * np.random.randn(*x.shape)
    
    # Play the sound
    sd.play(x_noisy, Fs)
    
    # Detect
    y = detect(x_noisy, frequency = f, Fs = Fs)
    
    # Plot results
    
    fig, ax = plt.subplots(3, 1)
    
    ax[0].plot(n, x)
    ax[0].set_title("Ground truth sinusoid")
    ax[0].grid("on")
    
    ax[1].plot(n, x_noisy)
    ax[1].set_title("Noisy sinusoid")
    ax[1].grid("on")
    
    ax[2].plot(n, y)
    ax[2].set_title("Detection result")
    ax[2].grid("on")
    
    plt.tight_layout()
    
    plt.show()

Another real life example is the TUT Age Estimator demo, which relies on face detection before the actual age estimation. Detection theory relies in Neyman-Pearson theorem, which states that the best approach is to compute the likelihood ratio and compare that to a threshold. The likelihood ratio usually simplifies to a simple computation rule (e.g., correlation between two sequences), whose output is then compared with a threshold.

The threshold is a free parameter, whose value determines the sensitivity of the detector. The smaller the threshold, the more events are detected and vice versa. Highly sensitive detector (= small threshold) probably finds all real events (e.g., locations in signal with a beep) but also produces many false alarms (locations detected as beep, but only noise).

To compare different detectors, we would like to check their performance regardless of the threshold (or for all thresholds at the same time). To this aim, we defined the Receiver Operating Characteristic (ROC) curve. For ROC, one can compute the area under the curve (AUC; AUC_ROC; AUROC). Bigger AUC values indicate that the detector performance is better. A completely random guess results in AUC = 0.5 (diagonal ROC) while perfect detection gives AUC = 1.0.