This is a description of the technology behind the interactive soundscapes that I’ve mainly been building as a part of the man and the machine project. It is aimed for non-nerds to explain what I do to an artist of other disciplines.
The technology is modular. This means that is works like Lego bricks so it can be picked apart and reassembled in many different ways. That allows the evolution of the project, so similar to a theatre production, I do rehearsals to learn more about how I want the lego bricks to be put together. The final result is some kind of machine that interacts with the human outside.
The machine is built out of four types of Lego bricks.
(1) Physical input bricks that collect information from the human world – like vibrations in objects, sounds and voices, and movement in space.
(2) Physical output bricks give the machine life in the human world – like instruments, sounds, notes, pictures, light effects, and much more.
(3) Digital input bricks that allow the machine to use already created digital material – like a sound file, or a manuscript, and a picture.
(4) Logical bricks that teach the machine how to connect all other bricks.
A very simple example
A camera records movement in a space (a physical input brick). The amount of movement is calculated (a logical brick) and compared to the history of movement (another logical brick). A note is picked from a pre-selected scale (a digital input brick) based development of the movement. Increasing movement equals higher and shorter notes. Decreasing movement equals lower and longer notes. The note selected note is sent to a digital instrument to create a sound (yet another digital input brick), and the sound is sent to a sound card connected to speakers (a physical output brick).
The experience outside is an interactive space that generates music based on movement.
A very complicated example
The most complicated machine that I’ve built so far in a very simplified way. Four vibration microphones mount on platforms (physical input) that detects any movement on the platform. The vibrations are translated into notes, that are looped into an echo (logical bricks). The notes are connected to noise generators (digital input) and eventually speakers (physical output). At the same time a digital manuscript is running, it plays a piece of Chopin music over and over, and randomly picks pre-recorded poetic lines that form a poem (lots of digital input and some logical bricks). A timer is triggered by the vibrations and once active the digital manuscript is blocked (another set of logical bricks). The timer is also controlling the lights, so vibrations equal hard lights, and poetic voices and Chopin equals soft lights (the lights are physical output bricks, and they are controlled by more logical bricks). There is a camera mounted in ceiling records movement, to detect small and soft movements (mostly hands and fingers), and they are connected other notes, scales, and instruments. Finally, there are three big sheets of metal that also has vibration detectors, and change their metallic sound into angelic voices. And also this is yet another load of bricks. And on it goes. Altogether over a hundred bricks.
The experience outside is a room that punishes movement with noise and hard light and rewards stillness with Chopin and poetry in soft light. While it allows small movements and vibrations to generate soft and soothing music.
The workbench
To build such a complicated machine, I have a digital workbench that allows be to easily connect the bricks together, to build new bricks, and to save my progress, and restore it later. Building the complicated example above took maybe ten rehearsals over two-three months of time. The workbench runs on a simple laptop, and requires a lot of technical skills to operate. It is built to be fast and flexible, rather then intuitive.
Physical input bricks
Here is what the machine has been interacting with so far.
(1) Cameras tracking movement, objects, and body heat.
(2) Vibration microphones tracking any vibrations in objects, like floors, furniture, and abstract items.
(3) Normal microphones tracking the sound and silence.
(4) Heartbeat monitors tracking one or many hearts.
Physical output bricks
(1) Analog sound and music generated inside the machine.
(2) MIDI notes and commands connected to instruments.
(3) DMX controlling stage lights.
(4) Video projections.
Digital and logical bricks
There are too many to list them here, because I built them when I need them, or integrate with existing software. The main technologies I use are.
(1) Python to build my own bricks. You can find some of them on GitHub.
(2) JACK Audio Connection Kit to send and receive audio and MIDI.
How stable is it?
As an old computer nerd and expert on redundancy and availability in telecommunication systems, this is a fundamental question for me. Also as a poor theatre student I also try to get away as cheap as possible.
My biggest and most complicated machine so far ran for three hours with an average of six people interacting with it. Digitally it translates to a couple of hundred MIDI messages per second, and 20 channel sound card and a cheap laptop running at 60% CPU at max.
So it’s good enough to experiment with, but probably far from my old life with teleco-grade availability (99.9999%) that equals 3 minutes downtime per year.