DIRSIM

Year : 2023

DirSim (Director Simulator) is a comprehensive ecosystem for musical control and visualization. It allows users to conduct a virtual orchestra using real-time hand gestures. The system integrates computer vision-based biometric tracking, dynamic protocol conversion (MIDI to OSC), and immersive visual feedback for a complete musical conducting experience.

Technologies:

System Architecture

DirSim is not a single program, but a network of three independent modules communicating in real time. First, data extraction occurs as the physical movement of the hands is captured on video and translated into spatial coordinates. This is followed by protocol translationusing a bridge server that receives signals and routes them to Reaper, a Digital Audio Workstation (DWA). Finally, the system handles the rendering of processed signals, triggering visual stimuli synchronized with the audio engine.

Controller

Developed in Python, the system utilizes Machine Learning (cvzone) for multi-hand tracking. It extracts hand landmarks to calculate the BPM through two distinct algorithms: Hand Speed Mode and Pattern Mode. It features a graphical user interface built with PyQt5 and transmits playback commands via Open Sound Control (OSC). Additionally, it includes a 'Dual Slider' system controlled by finger positioning.

Port

This module acts as the system’s middleware software. Utilizing python-rtmidi and python-osc, this module intercepts incoming MIDI hardware messages and packages them into OSC format. Its dynamic interface allows for on-the-fly reconfiguration of server ports and routing to Reaper, ensuring a data flow with zero packet loss.

Visualizer

The rendering engine, built in Processing, generates a geometric representation of a classic orchestral layout. It actively listens to the data stream and reacts by illuminating instrument families (Strings, Woodwinds, Brass, Percussion) with an intensity proportional to the velocity of the musical event, creating a responsive, immersive environment.