Augmented Listening

By Tue Haste Andersen - October 9, 2012

reBlogged from: design mind
 

Stop for a second and listen. Close your eyes, use your ears, and just listen.

Whether you are in a quiet office environment or out on a busy street, you'll be amazed by how many sounds there are around you. Most of us do not pay attention to the ambient sounds that surround us. Our brains filter them out and we don't listen. Yet the sounds we miss can be very enjoyable.

Designed Sounds

Today, what we hear in our daily lives is often designed sound- music and sound effects carefully crafted for games, devices, and products. For example, mission-critical products, such as heart rate monitors used during medical surgery or a plane’s flight deck controls, use distinctive alarming sounds that are designed to be easy to perceive and raise a sense of urgency or danger.

In interfaces for everyday tasks, sound is used to create engaging and beautiful experiences. Sounds can generate a special feeling or underline brand identity while simultaneously providing cues that a command has been received by the system. Most smart phones today come with subtle sounds that indicate the pressing of a touch screen’s virtual buttons. Since there is no way to feel if a virtual button has been pressed, the sounds reinforce the action for the user. Another example can be found in industrial design, where the latest electric cars are being designed with artificial motor sounds. The sounds alert pedestrians to the car as well as reinforce the sense of driving a powerful vehicle. These examples underline the overall trend of sound being used to create an aesthetic experience rather than serving as purely a functional aid to improve interaction.

Blurring the Border Between Listening and Composition

While systems and products are becoming more enjoyable and pleasant to listen to, they are usually not intentionally designed for sound interaction. The emergence of accessible music software on computers and mobile devices is changing this. These programs allow for easy modification of sound by the average user and blur the border between listening and sound creation. The small form and limited complexity of mobile interfaces has forced music software designers to reduce the complexity of their products, resulting in music software that is widely used by average mobile phone users.

Music apps are often top sellers. Popular applications allow people to become mobile DJs, to transform sounds, and to design ringtones.

I was interested in exploring the blur between sound creation and listening when my friend and colleague Matteo Penzo put me in contact with Matteo Milani from the U.S.O. Project sound art group. The ideas and compositions of the U.S.O. Project revolve around the use of noise and ambient sound as a foundation for sound installations and music composition. Together we wanted to create a mobile experience that would support active listening to the everyday sounds that surround us, making the listener a part of a personal sound installation. Instead of creating a tool for recording and transforming sound, we wanted to start from the sounds themselves. Our goal was to reinforce the sounds of the listener’s environment while blending them with more musical sounds. Together the sounds would form a unique experience that could be enjoyed by anybody that has an interest in sound and art.  

Early Experiments

We started with a small prototype app for iOS using simple sound algorithms to blend U.S.O. music with live recording from the iPhone microphone. The prototype was tested with real use cases that included listening to the app while taking a long walk as well as while sitting at the computer in the office. We added many parameters for the user to be able to tweak and play with the sound transformation.The parameters were mapped to on-screen sliders and buttons and to sensors like the accelerometer.

While doing the informal tests we found that the users were struggling to understand the relationship between the parameters and the sound output. Also, in most cases they would end up spending time experimenting with the parameters to discover how they work. The visual interface and controls were clearly distracting, taking attention away from the app’s original goal of reinforcing ambient sounds for the listener.  

Following these early experiments, we decided to take a drastically different approach. We limited the visual interface as much as possible and provided a set of sound themes in the app for the listener to select. This worked much better. All of a sudden the users would pick up the app and, once started, would tuck it away in a pocket while listening to the sounds. Each theme takes sounds from the microphone and blends them with sounds composed by U.S.O. Project. The sounds are blended using sound algorithms, unique to each theme. Each algorithm is carefully calibrated to replicate the work and skill that goes into producing a great listening experience.

Lis10er

The result is Lis10er (pronounced Listener), an augmented sound installation app. Sounds are blended from the listener’s surroundings, creating dynamic music that changes while maintaining its identity. Lis10er provides users with a creative way of listening to their environment and a unique experience with every listen. 

[www.lis10er.com]

Tue Haste Andersen is Senior Software Architect based in frog’s Milan studio. Tue is a Human Computer Interaction and Computer Music expert, with research ranging from DJ work practices to the use of sound and music in common interaction tasks. He is also the founder and original author of the popular open source DJ software, Mixxx.

Mirror_Mirror | Sound Installation

Concept and software design by Federico Placidi
Hardware design by Matteo Milani
Woodworker: Fabio Testa
Produced by U.S.O.Project, 2012

Where: [.BOX] Videoart project space, Via Federico Confalonieri 11, Milan
When: Thursday, September 27, 2012 | 6:30 until 21:30 PM
Free admission

The concept of multiverse was first introduced in the so-called “many-worlds interpretation” (MWI) of quantum mechanics by Hugh Everett III in his PHD thesis, "The Theory of the Universal Wavefunction". His model was thought as an alternative to the renowned theory called “Copenaghen interpretation”, developed by Niels Bohr and Werner Heisenberg.

The MWI interpretation postulates that every quantum measurement process (at Planck’s scale) creates, as a consequence, a division of the observed Universe into multiple parallel universes - as many as the possible outcomes of the measurements are.

In different formulations of this concept, all the universes - which form the Multiverse - are structurally identical, and they can coexist at different states even if they possess the same physical laws and fundamental constants.

We need to take into account that those universes are non-communicating (there cannot be any information exchange between them).

In an episode of the famous TV series “Doctor Who” - the episode was Doomsday, written by Russell T. Davies - the Doctor finds himself in the situation to make a difficult and dramatic choice: separating from the person who probably loved him the most, by “exiling” her in a parallel universe to guarantee her safety and survival.

It is worth mentioning some parts of the dialogue from the original script of the episode:

Rose comes to a halt in the middle of the beach and stands there, waiting. A short way to her left, the Doctor fades out of thin air. Rose turns to him. He's slightly translucent.

ROSE

Where are you?

THE DOCTOR

(his voice sounds distant)

Inside the TARDIS.

INT. TARDIS

The Doctor is, in reality, standing by the TARDIS console facing straight ahead.

THE DOCTOR (CONT'D)

There's one tiny little gap in the universe left, just about to close. And it takes a lot of power to send this projection, I'm in orbit around a super nova.

(laughs softly)

I'm burning up a sun just to say goodbye.

Sure enough, the TARDIS is spinning around a beautiful super nova.

EXT. BAD WOLF BAY

ROSE

(shaking her head)

You look like a ghost.

THE DOCTOR

Hold on...

He takes his sonic screwdriver out of his pocket.

INT. TARDIS

He points the sonic screwdriver at the console and somehow this strengthens his projection.

EXT. BAD WOLF BAY

The Doctor now looks as solid as if he were really there. Rose walks over to him and raises a hand to touch his face.

ROSE

Can I t--?

THE DOCTOR

(regretfully)

I'm still just an image. No touch.

ROSE

(voice trembling)

Can't you come through properly?

THE DOCTOR

The whole thing would fracture. Two universes would collapse.

The scene partially violates the no-information-transit prohibition between the two universes, but the subterfuge used in the story (the two characters cannot touch, only see each other), somehow preserves the assumption presented by the MWI, only conceding a small but necessary poetic licence.

In the same episode, there is a very touching scene, where the two characters (Rose and the Doctor), right after their isolation in two different universes, are in front of one another separated only by a simple white wall.

This border, imaginary and symbolic, which divides not only two places in the same space-time continuum, rather two whole universes, gave us an idea.

We wanted to offer that experience as an installation. We wanted to allow the audience to listen to whatever is “on the other side”, beyond that wall, and we wanted all this to happen in real time.

That’s how Mirror_Mirror was born.

The installation is organized inside a space.

What does this mean?

The space in itself, when it isn’t filled with matter (empty), is in reality permeated by low energy levels continually fluctuating.

These fluctuations in the void, have a particular significance on a quantistic level (we refer to the Planck scale, so at infinitesimally small dimensions).

In quantum mechanics, these fluctuations represent temporary shifts in the energy state of the void space, according to the Heisenberg Uncertainty Principle.

This means that the Conservation of energy principle can be violated for very brief periods of time (the lower the energy, the longer the fluctuation can persist.)

This energy can decade and take the shape of pairs of particles and antiparticles (which then annihilate each other).

In substance, the amount of average energy on a larger scale remains constant. Nothing is created, nothing is destroyed.

“There is a fact, or if you wish, a law, governing all natural phenomena that are known to date. There is no known exception to this law—it is exact so far as we know. The law is called the conservation of energy. It states that there is a certain quantity, which we call energy, that does not change in the manifold changes which nature undergoes. That is a most abstract idea, because it is a mathematical principle; it says that there is a numerical quantity which does not change when something happens. It is not a description of a mechanism, or anything concrete; it is just a strange fact that we can calculate some number and when we finish watching nature go through her tricks and calculate the number again, it is the same.” - R.Feynman

It was in our interest to draw an analogy in the sound domain, allowing the audience to directly experience this phenomenon.

As a consequence, we created an application able to “sonify” ideal energetic fluctuations (generating pressure waves, structures and emerging behaviours), starting from the lower energy level available, which is the background noise.

Thanks to the Kyma software implementation, and the use of microphones, it was possible to “measure” background noise and , through a series of negative feedback operations, enable the instrument to create “something”, using all the information available in our space/universe to create temporary energetic fluctuations (statistic variations of the density of sonic quantums’ “packets”), without violating the Conservation of energy principle - in fact, the average energy quantity, altogether, remains the same.

We’ve thus arrived to the gravitational center (here we have no more fluctuations, only numerous certainties, due to the dimensions and mass of the object) of the opera, which is represented by a wooden artifact, symbolically revisiting the white wall we came across in Doctor Who, which divides our universe from another possible universe.

Which one it is, the visitor will have to find out for himself.

By placing a stethoscope on the wooden surface. the observer will be able to “measure”, with various levels of definition, the sounds coming from another probable universe out of phase with ours.

In fact, as identical as it will seem, it strangely does not share the same temporal parameter.

From quantum mechanic and its Multi-Worlds interpretation, we know that every measurement operation will produce a further division of the universe.

So, it is possible that in the end, there will be as many universes as the present observers.

And, paying a little attention, it will be possible once again to listen to the voices of Rose and the Doctor, as if they are suspended in a temporal loop, to remind us that maybe, the current physics laws, could not be the same in every place and every time.

Federico Placidi, Matteo Milani
>>U.S.O.Project

New Album 'Unsorted Tales' Just Released!

Composed by U.S.O. Project - Unidentified Sound Object
SYN-007 © 2012 Synesthesia Recordings
Mastered @ Green Movie, Milan (Italy) by Matteo Milani & Federico Placidi

Performers:
Edoardo Carlo Natoli - Violin
Federico Placidi - Violoncello & Flute

TRACKLIST:

1) F'Shima (13.49)
2) Gretel's New Clothes (19.46)
3) Psalm 21 (22.15)

RELEASE INFO:

Artist: U.S.O. Project - Unidentified Sound Object
Title: Unsorted Tales Cat.No: SYN-007
File under: Experimental/Electronic
Format: Digital
Release date: 9.2012


F'Shima 



It's night.
There are no precise coordinates, nor recognizable signs that can lead us back to a familiar place.
And yet those sounds remind us, almost pathetically, of a summer night scene, to which we listened countless times.
But that’s not what this is.
The slow and articulated flow of pulsating energy gradually takes a structure, enabling us to be part of something very different.
The flow surrounds us, passes through us.
We can’t see it, nor touch it, but every particle interferes with our body, with our biological substrate. Far away, a storm is coming, perhaps.
It’s neither natural, nor inoffensive. It deceives us with the variety of its harmony of timbres.
The structure suddenly changes, reorganizing itself while turning us into something new.
A breach made of light. It’s oscillating.
Then, silence.
The air stagnates, suspended and motionless.
An electrical impulse.
One more. Others follow, one after another, fainter and more distant.
(Sounds), like petals made of ash.

F'Shima was designed and built around a series of recordings of electromagnetic fields generated by various electronic devices (Hard Drives, iPad, iPhone, Portable Game Consoles ...), recordings made ​​by using old analog phone-captors. The source material, sometimes fully recognizable, sometimes radically altered, has been digitally processed using the Kyma Sound Design environment.

Gretel's New Clothes 



"Nibble, nibble, gnaw,
Who is nibbling at my little house?"
The children answered: "The wind, the wind,
The heaven-born wind."

Gretel's New Clothes is, in a way, an acousmatic reinvention of the famous fairytale by the Brothers Grimm. It is in the form of a Rondò. 

There is no attempt to literally retrace the story, rather to suggest it in a subtle way, through the use of strongly characterized sound materials (footsteps in the woods, the wind, the "sounds" of the night...) which are then dialogically processed during the final Violin and Cello improvisation. 

“They walked the whole night and all the next day too from morning till evening, but they did not get out of the forest, and were very hungry, for they had nothing to eat... And as they were so weary that their legs would carry them no longer, they lay down beneath a tree and fell asleep.” 

Psalm 21



“...Your hand will find out all your enemies;
your right hand will find out those who hate you.
You will make them as a blazing oven
when you appear.
The Lord will swallow them up in his wrath,
and fire will consume them.”

Psalm21 is not a religious work. There is no implication of a celebratory or ritual nature. The dominant and recurring element during the composition of the piece, is the relationship between creation (as a constantly evolving act), creator (of which we ignore existence and features) and creature (as a self-conscious subject that reflects on his nature). These three elements are the driving force around which the work unfolds.

[Available on CD Baby Music Store]

[Digital booklet: Unsorted_Tales.pdf]