[Example] Simple Audio Detection

In this example, we will show you how to use ESP to create a simple audio classifier that can recognize sounds with distinct frequency spectrum characteristics.

The source code of this example can be found in the repository [link].

Video & Screenshots

Here is a short video of demonstrating piano key press detection.

What you will need?

A Macbook with ESP setup (see the installation guide from the README. We will use the Macbook built-in microphone for this example. Optionally, you can plug in an external microphone.

The example can also be configured to use an electret microphone hooked up to an Adruino. You should program the Arduino with this code. Also update the input stream (line 4) with SerialStream.

Background knowledge

FFT

SVM

Example code walkthrough

void setup() {
    stream.setLabelsForAllDimensions({"audio"});

    pipeline.addFeatureExtractionModule(
        FFT(kFFT_WindowSize, kFFT_HopSize,
            DIM, FFT::RECTANGULAR_WINDOW, true, false));

    pipeline.setClassifier(
        SVM(SVM::LINEAR_KERNEL, SVM::C_SVC, true, true));

    pipeline.addPostProcessingModule(ClassLabelFilter(25, 40));

    calibrator.addCalibrateProcess("Bias", "Remain silent", backgroundCollected)
              .addCalibrateProcess("Range", "Shout as much as possible", shoutCollected);

    useInputStream(stream);
    useCalibrator(calibrator);
    usePipeline(pipeline);
}

Future work and ideas