Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added New Plugin For Recording Audio #449

Merged
merged 35 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
3326510
Base module setup for PhoneAudioInput plugin
this-Aditya Jun 28, 2024
9db6791
Added superclasses to components of plugins
this-Aditya Jun 29, 2024
e72059c
Added configuration parameters and Recorder Initialization logic
this-Aditya Jun 29, 2024
9271919
Renamed variables and some minor changes
this-Aditya Jul 1, 2024
4959548
Added addtional recorder initialization logic and some calculations
this-Aditya Jul 1, 2024
4422fce
Audio recording and saving internally
this-Aditya Jul 2, 2024
0023d46
Remove permissions
this-Aditya Jul 2, 2024
3ee318a
Addition of PhoneInputAudioActivity
this-Aditya Jul 5, 2024
62da5b8
SafeHandler support added
this-Aditya Jul 5, 2024
634eab9
Wrapping RandomAccessFile in BufferedOutputStream
this-Aditya Jul 7, 2024
2c2d565
Updated UI to start and stop recording
this-Aditya Jul 7, 2024
390b062
Initial work on audio playback after saving it
this-Aditya Jul 8, 2024
b84045b
Fixed errors while recording audio with UI updates
this-Aditya Jul 9, 2024
762492b
Initial work on default microphone selection
this-Aditya Jul 11, 2024
9d936d8
More work on input device selectionand rendering logic
this-Aditya Jul 14, 2024
014e1cd
Microphones will refresh after disconnection
this-Aditya Jul 14, 2024
3de1149
Updated device selection and recording timer
this-Aditya Jul 15, 2024
b32df44
Using external directories once for testing
this-Aditya Jul 15, 2024
8237da0
Corrected header for seamless playback
this-Aditya Jul 15, 2024
8b641dd
Used functional programming and scope functions
this-Aditya Jul 16, 2024
3685526
Updated viewmodel to share state to fragment and new dialogue after r…
this-Aditya Jul 16, 2024
a54ba06
Resetting the MediaPlayer after use
this-Aditya Jul 16, 2024
614faa8
Minor changes
this-Aditya Jul 17, 2024
02a0575
UI improvements for audio playback
this-Aditya Jul 18, 2024
faca109
Updated files with Apache License
this-Aditya Jul 18, 2024
dc17ecb
Added smooth navigation between activities and fragments and ensured …
this-Aditya Jul 18, 2024
d8ce197
Deferring the viewmodel initialization on new activity instances
this-Aditya Jul 18, 2024
9ca32dc
Some UI work and updated README
this-Aditya Jul 19, 2024
7e92754
Removed extra logging calls
this-Aditya Jul 19, 2024
05fc809
Update README.md
this-Aditya Jul 19, 2024
46fbcf7
Merge branch 'dev' into audio-input
this-Aditya Jul 23, 2024
66ab343
Corrected preferred device forwarding
this-Aditya Jul 31, 2024
ecc106b
Now the audio recording can be paused and resumed
this-Aditya Aug 1, 2024
75c1602
Updated README added guide for using plugin and changed device select…
this-Aditya Aug 1, 2024
d9084a7
Minor changes
this-Aditya Aug 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions plugins/radar-android-phone-audio-input/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# RADAR PHONE AUDIO INPUT

Plugin for recording uncompressed high-quality audio, utilizing low-level classes to directly interact with hardware, with capabilities for playback and audio input device selection.


## Installation

Include this plugin in a RADAR app by adding the following configuration to `build.gradle`:
```gradle
dependencies {
implementation "org.radarbase:radar-android-phone-audio-input:$radarCommonsAndroidVersion"
}
```
Add the provider `org.radarbase.passive.phone.audio.input` to the `plugins` variable of the `RadarService` instance in your app.

## Configuration

To enable this plugin, add the provider `phone_audio_input` to `plugins` property of the configuration.

This plugin takes the following Firebase configuration parameters:


| Name | Type | Default | Description |
|------------------------------------------|------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `phone-audio-input-audio-source` | int | `1` = MediaRecorder.AudioSource.MIC | Specifies the source of audio input for recording. The default value is 1, which corresponds to MediaRecorder.AudioSource.MIC. This means the application will use the device's microphone for audio input. |
| `phone-audio-input-recorder-buffer-size` | int | `-1` | Defines the size of the buffer used by the audio recorder. The buffer is a temporary storage area for audio data before it is processed or saved. Setting this to -1 lets the system choose an optimal buffer size based on the current audio configuration. Adjusting this can impact audio latency and quality. |
| `phone-audio-input-current-audio-format` | int | `2` = AudioFormat.ENCODING_PCM_16BIT | Determines the encoding format of the recorded audio. The default value 2 corresponds to AudioFormat.ENCODING_PCM_16BIT, which means the audio will be recorded in 16-bit Pulse Code Modulation (PCM). PCM is a common uncompressed audio format that provides high-quality sound. Other formats like 8-bit PCM can also be used, but they may reduce audio quality. Note that 8-bit encoding might not work on all devices, so 16-bit should be preferred |
| `phone-audio-input-current-channel` | int | `0x10` = 16 | Specifies the number of audio channels to be used during recording. The default value 0x10 represents mono (1 channel), meaning audio will be recorded from a single channel. Using mono recording is typical for voice recording to save space and simplify processing. Stereo (2 channels) can also be used for higher quality audio that captures a sense of direction. |
| `phone-audio-input-current-sample-rate` | int | `16000` | Defines the number of audio samples captured per second. The default value is 16000 Hz (16 kHz), which is a common sample rate for voice recording, balancing audio quality and file size. Higher sample rates, such as 44100 Hz (CD quality), provide better audio fidelity but result in larger file sizes. Lower sample rates may be used for lower quality or reduced file size needs. |



This plugin produces data for the following topics: (types starts with `org.radarcns.passive.google` prefix)

| Topic | Type | Description |
|-----------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `android_phone_audio_input` | `PhoneAudioInput` | This topic captures high-quality, uncompressed audio data recorded by the user's device. The PhoneAudioInput type includes detailed information about the audio recording, such as timestamps, file metadata, and device specifications. |

## Workflow Description

This plugin provides an interactive UI for recording and managing audio data. Below is a detailed description of how the plugin works:

### Workflow

1. **Plugin Action**:
- The plugin has an action that, when clicked, opens a user interface (UI) for interacting with the plugin.

2. **Activity Launch**:
- The action opens the `PhoneAudioInputActivity`, which serves as the main interface for the plugin.

3. **Manager Connection**:
- `PhoneAudioInputActivity` is connected to the `PhoneAudioInputManager` through interfaces defined in `PhoneAudioInputState`. This connection manages the state and interactions between the UI and the underlying audio recording functionalities.

4. **Recording Capabilities**:
- The activity provides capabilities for recording audio, allowing users to start, stop, and manage audio recordings directly from the UI.

5. **Device Selection Mechanism**:
- The device selection mechanism automatically prioritizes the USB audio device if connected. If no USB device is found, the plugin selects external devices in the following order of precedence:
- `TYPE_USB_DEVICE`
- `TYPE_WIRED_HEADSET`
- `TYPE_BLUETOOTH`
- If no external device is found, the plugin defaults to using the smartphone's built-in microphone.

6. **Fragment Integration**:
- `PhoneAudioInputActivity` opens a fragment that provides additional capabilities, including:
- **Audio Playback**: Users can playback the recorded audio to review it before sending.
- **Data Sending**: Users can send the recorded audio data for storing to s3.

### Guide for Using the PhoneAudioInput Plugin
<br>
<img src="assets/images/audio_input_plugin/action_devices.jpg" width="200px" alt="action_devices"/>
<img src="assets/images/audio_input_plugin/recording_activity.jpg" width="200px" alt="recording_activity"/>
<br>
1. In the main activity, where plugins are visible along with their actions (if any), click on the specific action for the Phone Audio Input plugin.
- This will redirect you to the first activity of the Phone Audio Input plugin.
<br>
<img src="assets/images/audio_input_plugin/recording_activity_devices.jpg" width="200px" alt="recording_activity_devices"/>
<img src="assets/images/audio_input_plugin/recording_activity_paused.jpg" width="200px" alt="recording_activity_paused"/>
<br>
2. At the top of this activity, there is a dropdown menu for selecting microphones.
- The current input routing microphone is shown by default in the dropdown menu when it is collapsed.
- Click on the refresh button near dropdown menu to refresh microphones.
- Select your preferred microphone from the dropdown menu. If no preferred microphone is selected, the automatic device selection logic will take place.
<br>
<img src="assets/images/audio_input_plugin/alert_rec_ongoing.jpg" width="200px" alt="alert_rec_ongoing"/>
<img src="assets/images/audio_input_plugin/alert_rec_stopped.jpg" width="200px" alt="alert_rec_stopped"/>
<br>
3. Click Start Recording to begin recording audio.
- While recording, you can pause or resume the recording.
- Do not quit the recording activity while recording is in progress or paused, as this may cause the application to misbehave. An alert will also be shown if you attempt to quit the activity in a recording or paused state. You can safely quit the activity after stopping the recording.
<br>
<img src="assets/images/audio_input_plugin/playback_frag.jpg" width="200px" alt="playback_frag"/>
<img src="assets/images/audio_input_plugin/playback_exit_alert.jpg" width="200px" alt="playback_exit_alert"/>
<br>
4. After stopping the recording, an alert dialog will appear with three options: play the recorded audio, send it directly without playing, or discard it.
- If you click Play, you will be redirected to a new fragment for listening to the last recorded audio. Here, you can review the recording and send it after listening.


Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
33 changes: 33 additions & 0 deletions plugins/radar-android-phone-audio-input/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
apply from: "$rootDir/gradle/android.gradle"

android {
namespace "org.radarbase.passive.phone.audio.input"
}

//---------------------------------------------------------------------------//
// Configuration //
//---------------------------------------------------------------------------//

description = "Plugin for recording uncompressed high-quality audio."

//---------------------------------------------------------------------------//
// Sources and classpath configurations //
//---------------------------------------------------------------------------//
android {
buildFeatures {
viewBinding true
}
}

dependencies {
api project(":radar-commons-android")
implementation "androidx.appcompat:appcompat:$appcompat_version"
implementation "com.google.android.material:material:$material_version"
implementation "androidx.activity:activity:1.9.0"
implementation "androidx.constraintlayout:constraintlayout:$constraintlayout_version"
implementation "androidx.legacy:legacy-support-v4:$legacy_support_version"
implementation 'androidx.fragment:fragment-ktx:1.8.1'
}

apply from: "$rootDir/gradle/publishing.gradle"
apply plugin: 'org.jetbrains.kotlin.android'
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools" >

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.WAKE_LOCK" />

<uses-feature android:name="android.hardware.microphone" android:required="false"/>

<application android:allowBackup="true" >
<activity
android:name=".ui.PhoneAudioInputActivity"
android:exported="false" />

<service
android:name=".PhoneAudioInputService"
android:description="@string/phone_audio_input_description"
android:exported="false" />
</application>

</manifest>
Loading
Loading