A virtual orchestra conductor application using the Microsoft Kinect for Windows v1. Developed as an Earlham College senior capstone research project in collaboration with the Music and Computer Science Departments.
This application tracks your hand's position and uses its vertical velocity and acceleration to detect beats and determine volume, respectively. A MIDI-like CSV file is used to store the desired music for playback. For more information, you can read the accompanying paper found in the pdf folder.
This application was developed using the following libraries. Some notable dependencies are also listed here.
Ensure that all the required libraries listed above (except XKin, see below) are installed and configured for your computer. If you are configuring libfreenect with
cmake, be sure to use the option-DBUILD_CV=ON.
- Clone this repository and
cdto the source directory. - Create a new folder called
XKinand clone the XKin library repository into that folder. Follow the instructions provided to compile it. - Return to the source directory and run
make.
The command to run the program is konductor -m [music] -f [font], where [music] is the CSV file containing the score to be played, and [font] is the SoundFont file containing the desired MIDI instruments for playback. For example, while in the source directory to play the CSV music file examples/ensemble.csv with the Fluid (R3) General MIDI SoundFont located at /usr/share/sounds/sf2/FluidR3_GM.sf2:
$ ./konductor -m examples/ensemble.csv -f /usr/share/sounds/sf2/FluidR3_GM.sf2
When you run the program for the first time, the JACK server should start up as well if it hasn't started already. If JACK is unable to use real-time scheduling, run the commands
sudo dpkg-reconfigure -p high jackdto give JACK high priority andsudo adduser [your-username] audioto add your user account to the "audio" group, then restart your computer.
Our program reads a CSV file consisting of lines of numbers in a certain format in order to play music. The first line consists of three numbers: the number of program changes to read, the number of note messages to read, and the PPQN (pulses/ticks per quarter note) value for the music.
The following lines, equal to the expected number of program changes, each consist of a channel number (0-15) and a program number (0-127 for General MIDI), telling FluidSynth which program/instrument maps to which channel.
Each of the remaining lines then represent a MIDI "note on" or "note off" message, which consists of the following five numbers in order:
- beat number
- ticks after current beat
- channel number
- key/note number
- 1 for "note on" or 0 for "note off"
All numbers must be separated by whitespace. See our example CSV files in the examples folder for reference. Ensure that each message is sorted by increasing beat number (and then preferably by increasing tick count too) as each MIDI message will be read and scheduled in the same order as in the file.
examples/ensemble.csv- Original Composition: Ode to Joy by L. V. Beethoven (1770–1827)
- Arranged for Woodwind Quintet by Edward Ly
- Distributed under Creative Commons license
23 February 2017