Skip to content

Latest commit

 

History

History
132 lines (103 loc) · 7.35 KB

README.md

File metadata and controls

132 lines (103 loc) · 7.35 KB
Raw

Mesh Audio Editor

Real-time mesh viewer and editor supporting conversion of meshes to rigid body audio models using Linear Modal Analysis/Synthesis.

Features

General features:

  • Create/delete meshes and mesh instances
    • Editable mesh primitives (Rect, Circle, Cube, IcoSphere, UVSphere, Torus, Cylinder, Cone)
    • Load .obj, .off, .ply, .stl and .om mesh files (via OpenMesh)
  • Select meshes or mesh elements (vertices, edges, or faces)
  • Flat/smooth/wireframe mesh rendering
  • Bounding Volume Hierarchy (BVH) ray intersection acceleration structure
    • Used for mouse interactions - this is not a ray tracing renderer :)
  • Translate / rotate / (nonuniformly) scale meshes and instances with gizmo and numeric inputs
  • Simple camera + scene lighting model, roughly matching Blender visually
  • Edit camera with:
    • Camera rotate cube gizmo
    • Scrollwheel zoom
    • Numeric inputs
  • Edit lighting parameters
  • Render face/vertex normals as lines for debugging
  • Render bounding box / BVH wireframes for debugging
  • Edge-detection-based silhouette outline of selected mesh/instance, embedded into the scene with accurate per-pixel depth
  • Fast infinite grid with horizon fade

Audio-specific features:

  • Enable/disable audio output and change device and native format/sample rate
  • Volume / Mute
  • Load RealImpact datasets, including the object mesh and instanced cylinders for each microphone position.
  • Generate an efficient physical audio model for any mesh. (See Physical audio modeling.)

Noteworthy dev bits:

  • Terse and direct usage of Vulkan-Hpp
  • Change SPIRV shader code and trigger a runtime recompile in the UI
  • Resource reflection: Use SPIRV-Cross to automatically create descriptor set layout bindings for all shader pipelines
  • Instanced rendering of shared geometry with variable transforms

Physical audio modeling

This project supports generating an efficient physical audio model for any mesh using Linear Modal Analysis/Synthesis

The physical audio modeling components were implemented as a final project for PHYS-6260 - Computational Physics at Georgia Tech during my Master's, and this is the final report. Here is the final report paper.

And here is a 36X48 poster:

Impact audio experiments

Below are audio examples synthesized by "striking" modal audio models (by injecting a short wideband pulse at the selected vertex) for various meshes, with comparisons to impact recordings of their real-world counterparts being struck at the same position. The audio recordings and 3D-scanned meshes come from the RealImpact dataset. See the blog post for embedded audio players.

The cylinders shown in the images represent recorded microphone positions, but all recordings come from a single microphone centered near the impacted object, and the modal audio model does not implement any audio wave radiation modeling. All modal audio samples are generated by extracting estimated surface vibrations, as if recorded from a contact microphone.

Object Name Mesh Real Impact Audio Modal Impact Audio
Ceramic Koi Bowl Mesh Impact Modal
Ceramic Pitcher Mesh Impact Modal
Glass Cup Mesh Impact Modal
Iron Mortar Mesh Impact Modal
Iron Skillet Mesh Impact Modal
Plastic Scoop Mesh Impact Modal
Small Swan Ceramic Mesh Impact Modal

Build app

Install dependencies

  • Download and install the latest SDK from https://vulkan.lunarg.com/sdk/home
  • Set the VULKAN_SDK environment variable. For example, add the following to your .zshrc file: 
    export VULKAN_SDK="$HOME/VulkanSDK/{version}/macOS"

Mac

$ git clone --recursive [email protected]:khiner/MeshEditor.git
$ brew install cmake pkgconfig llvm fftw
$ brew link llvm --force

Linux

(Only tested on Ubuntu.)

$ sudo apt install llvm libc++-dev libc++abi-dev
$ ln -s llvm-config-17 llvm-config
$ export PATH="$(llvm-config --bindir):$PATH"

Install GTK (for native file dialogs):

$ sudo apt install build-essential libgtk-3-dev

Clone, clean, and build app

$ git clone --recurse-submodules [email protected]:khiner/MeshEditor.git
$ cd MeshEditor
$ CC=clang CXX=clang++ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
$ cmake --build build --config Release

Stack

  • Vulkan + ImGui + SDL3: Graphics + immediate-mode UI/UX
  • glm: Small numeric vector/matrix types + math
  • OpenMesh: Main polyhedral (half-edge) mesh data structure
  • entt: Entity Component System (ECS) for an efficient and scalable mixin-style architectural pattern
  • miniaudio: Audio stream I/O
  • fftw for computing spectrograms (visualized with ImPlot)
  • Faust: Functional audio programming language, used to render an object audio model to an audio graph
  • tetgen: Fast conversion of triangular 3D surface meshes into tetrahedral volume meshes
  • 3D FEM: VegaFEM for generating mass/stiffness matrices from tet mesh + Spectra for estimating eigenvalues/vectors
  • VulkanMemoryAllocator: Efficient Vulkan memory allocation
  • nativefiledialog-extended: Native file dialogs (TODO SDL3 now has SDL_Dialog)
  • ImPlot: Plotting
  • ImGuizmo: Mesh transform and camera rotation gizmos