An interactive 3D acoustic manifold visualization of bird vocalisations. Extracts 57-dimensional MFCC feature vectors per frame, reduces them to 3 principal components via PCA, classifies species using kNN (cosine similarity, k=5), and serves it all through a FastAPI backend with a Three.js frontend and a conversational AI assistant powered by Llama 3.1.
A full academic paper documenting the mathematical model, feature extraction pipeline, PCA reduction, and system architecture.
Built to explore the intersection of audio DSP, machine learning, and real-time graphics, the project demonstrates how complex sound data can be converted into meaningful visual structures.
The system processes raw bird audio through a Python-based ML pipeline that extracts 57-dimensional feature vectors per frame using librosa. The feature vector comprises MFCCs (1–13), delta and delta-delta MFCCs, spectral centroid, bandwidth, rolloff, zero-crossing rate, RMS energy, and mel-spectrogram bands, capturing timbre, texture, and spectral range of each 23ms frame.
Since visualizing 57 dimensions directly is impossible, the pipeline applies Principal Component Analysis (PCA) using scikit-learn to reduce the feature space into 3 principal components, mapped to Timbre (X), Texture (Y), and Spectral range (Z), while preserving the most important variation in the signal.
The reduced data is rendered as an animated 3D trajectory using Three.js, with points coloured by energy (dark → silence, amber → peak) and sized by amplitude. A kNN classifier (cosine similarity, k=5) identifies the closest matching species from the manifold.
A FastAPI backend serves the manifold data, processes uploaded audio files for real-time classification, and powers a conversational AI assistant built on Llama 3.1 8B Instant via Groq, enabling natural-language questions about each species.
Raw bird audio is loaded and segmented into 23ms frames. 57-dimensional MFCC and spectral features are extracted per frame with librosa, then normalized and scaled before PCA compresses them into 3 principal coordinates.
The 3 principal components map to Timbre (X axis), Texture (Y axis), and Spectral range (Z axis). Points are coloured by energy and sized by amplitude, turning a 57-dimensional signal into a shape you can actually see.
Three.js renders the PCA trajectory as an animated comet trail with free-orbit controls, using binary-search frame mapping to keep the audio and the animation in sync.
Visitors can drag and drop their own audio: the same pipeline extracts its features, projects them into the shared PCA space, and a cosine-similarity search (k=5) finds the closest matching species.
A conversational ornithology assistant, powered by Llama 3.1 8B Instant via Groq, answers natural-language questions about species behaviour, habitat, and characteristics, with full chat history kept per session.
Extracted 57-dimensional MFCC and spectral feature vectors from raw audio with librosa: the real DSP work underneath the visualization, not just a pre-processed dataset.
Applied PCA to compress a 57-dimensional feature space into 3 interpretable axes, then built a kNN classifier (cosine similarity, k=5) on top of that reduced space.
Built a Three.js visualization with free-orbit controls and binary-search frame-to-audio sync: an interactive scene, not a static chart.
Wired a Llama 3.1 8B Instant assistant via Groq into the same app, with per-session chat history, going beyond the visualization into a genuine conversational feature.
Try the live demo yourself: upload a bird call and watch it drop onto the manifold.
↗ Open the Live Demo