Sonification/audification of Data?
January 29, 2002 | Ross Mohan
20 Comment(s)
Might you be willing to share your thoughts on the appropriate
sonification/audification of multidimensional data, vis a vis
facilitating researchers’ ability to navigate and “mine” complex
and otherwise hard-to-comprehend data?
(The following book might be a Volume I of such an investigation.
Schafer, R. M. The Tuning of the World. New York: Knopf, 1977. )
With Gratitude and Interest,
Ross Mohan
Perhaps there are opportunities to extend sound as content: adding sound to electrocardiograms, making voice annotations to photographs, having data points when selected speak their names, and so on. We have to be alert, though, to the sources of failure of talking refrigerators (which proved highly irritating to people).
A different matter is the use of sound as a code to represent data variation–that is, sound as a glyph. It sure adds a different mode to data analysis, but a convincing demo is needed to make the case. Just now I tried downloading several sonification examples but their software didn’t seem to like my software; perhaps the sonification researchers could work on their communication skills.
Color and motion also represent opportunities for displaying multivariate information. Some impressive efforts at color and motion have not caught on.
I should see some good demos before saying anything else. Any suggestions?
Perhaps an interesting place to spur your consideration of this topic would be “video” games for the blind. The ways that they convey information that is otherwise rendered visually can be very interesting. And what a visceral impact!
An interesting article regarding helping the blind with visualizations:
http://www.upi.com/view.cfm?StoryID=20021024-034855-5479r
This article provides an interesting approach to finding programming bugs … code is translated into “music”, and one apparently catches bugs by hearing discrepancies in the sounds – from what one expects to hear.
http://www.trnmag.com/Stories/2002/080702/Programming_tool_makes_bugs_sing_080702.html
Michael Round
Victor Neiderhoffer tells an interesting story in “The Education of a Speculator”. He had some sort of computer voice system reading currency exchange rates over the public address system at his trading house. He had the programmers calculate how much each move in exchange rates lost for his company, and insert an appropriately strong obscenity or profanity into the stream of announcements.
Eventually, they took the system out, as the traders found it unnerving when the computer took the words right out of their mouths.
The Education of a Speculator
Victor Niederhoffer
ISBN: 0-471-24948-3
An interesting footnote, re: the blind and the auditory-visual information continuum:
http://www.nytimes.com/2005/12/11/magazine/11ideas_section3-14.html?ex=1135054800&en=ff2c8dda1e02cbd3&ei=5070
The composer/architect Xenakis was able to pass off sonic representations of data as some of the most challenging music (for music’s sake) in the post-war period. From Wiki:
“[Xenakis] is particularly remembered … for the use of stochastic mathematical techniques in his compositions, including probability (Maxwell-Boltzmann kinetic theory of gases in Pithoprakta, aleatory distribution of points on a plane in Diamorphoses, minimal constraints in Achorripsis, Gaussian distribution in ST/10 and Atrees, Markovian chains in Analogiques), game theory (in Duel and Strategie), group theory (Nomos Alpha), and Boolean algebra (in Herma and Eonta). In keeping with his use of probabilistic theories, many of Xenakis’ pieces are, in his own words, “a form of composition which is not the object in itself, but an idea in itself, that is to say, the beginnings of a family of compositions.” Unlike most of his contemporaries(i.e. Milton Babbitt, Schoenberg), Xenakis did not want the listener to be aware of the forms and theories used to produce his compositions.”
Probably not the best way to TEACH boolean algebra or the kinetic theory of gases, but an absorbing way to enjoy these topics’ form.
There’s a startup out of Hanover, NH (Dartmouth College) called Accentus that is selling
high-end sonification systems to financial firms: http://www.accentus.com/
“Accentus offers you leading-edge, world-class science packaged in the first commercial
auditory display product. And we created it from the ground up for Sales & Trading. The
Accentus Auditory Display software transforms market data into music-based sound to
convey market data to trading pros, complementing existing visual displays, improving
multi-tasking, reducing visual data overload, so you can realize new trading profits.”
Recent reports on National Public Radio provided “audio charts” of funds raised in the first quarter of 2007 by candidates for US President in the 2008 election.
Reporting on Democratic candidates on April 2, 2007, NPR used snippets of the song “Staying Alive” by the BeeGees, playing 0.5 second of the song per million dollars raised by each candidate.
The “audio chart” begins at 01:27 in the recording.
Reporting on Republican candidates on April 3, 2007, NPR revised their charting method. They played “one beat” of “I Will Survive” by Gloria Gaynor, per million dollars raised by each candidate. In this case, they also restarted the song for each candidate, giving a common “baseline”, whereas the previous “chart” played successive sequences of the song.
We have been experimenting with sound-enhanced graphics for a while. In 2000, we added sound to time series analysis. Each time series value is expressed as a tone from a 20 tone scale. The mapping between tones and values is done individually for each series. Thus, multiple time series may be depicted on one identical scale, but can be heard with individual scales. This is particularly helpful when the data series are of different magnitudes. In this case, the ‘lower’ series will look flat, but you can still listen to how the values vary. The difference between what is seen and what is heard is slightly puzzling. Ideally the perceived discrepancy reminds one of using a second scale.
Here is an example of sound animation for sparklines:
Some more examples are on http://www.bella-consults.com/sound-pattern
Can our Kindly Contributor also post the sparkline sonifications to our thread Sparklines:
theory and practice?
Perhaps a virtue of tying sound to the data is that it induces the viewer to track visually
through the data more closely than usual. There is also a bit of Music Animation Machine to
these images.
Blaufuss Multimedia has some very good instructional flash productions for heart sounds, probably the best this medical student has seen, short of Harvey. Try to look and listen for the synchronicity of pulses and beats in the Blaufuss tutorial’s various disease state sequences. I dare say the analysis of heart sounds sets a worthy standard for those seeking to sonify data.
Now if someone would put an eccentric roller or pusher under the rubberized surface of the computer’s mouse, we could feel the surface distension of pulses as well.
A most interesting thread. A musical score and the resulting playing of it is the ultimate relationship of sound and data. In my work I use simple sung notes to accompany an explanation of rows, rhythm, emphasis and dynamic exposition in creating work posters. With this thread, new work is apparent. Many thanks
The Blaufuss heart music is wonderful. Thump, slosh, thumpy-do.
It might be useful to
hear the sounds first at a pulse rate of 20 in order to appreciate the full structural details–
and then speed it up to 40, 60, and then to tachycardia rates.
Also good names to
the sound are very helpful, such as the dog bark followed by a click. We should be careful
about metaphors but they are helpful for recognition of complex sounds.
A link from Andy James:
link
Regarding the housing price animation submitted by Andy, I keep coming back to the idea with video or animation that time can create resolution (somewhat similar to the ideas of small multiples). For me, the animation provided greater emphasis to the points of change in housing prices than a traditional line graph does.
This led me to the observation that the tempo of the animation (specific to this style) becomes similar to the scale of a line chart. Changes in the tempo therefore change the emphasis towards specific events in the animation and almost becomes like changing the scale of the line chart midway through graph. In the housing market video the first 100 years took 1 minute 53 seconds while the last 20 years took 1 minute 8 seconds; three times longer than it should have if it kept to the same tempo as the first 100 years.
The roller-coaster graphic lacks the comparative visual persistence of a conventional time-
series plot; a
change briefly appears and goes away. The roller-coaster is dequantified; the core message
is that there have been some ups and downs, along with a recent long-run up. The RC data
display is briefly intriguing because it is different.
Viewers would learn and remember a
lot more about housing prices by devoting the RC-viewing time to a conventional time-
series. A conventional graphic would accommodate many different ways of looking at the
data. Will a Kindly Contributor post the time-series for the monthly data shown in the RC?
There’s a brief flash of the original graphic at the end of the roller-coaster video. I believe this is it:
http://www.nytimes.com/imagepages/2006/08/26/weekinreview/27leon_graph2.html
Thought this was worth sharing to reflect upon “The ear tends to be lazy, craves the familiar, and is shocked by the unexpected: the eye, on the other hand, tends to be impatient, craves the novel and is bored by repetition.”
– The Dyer’s Hand W.H.Auden.
Hello Mr. Tufte,
Responding to your request from February 2, 2002 for good sonification demos to represent data variation, please go to my heart rate variability sonification site at: http://www.michaelfalkner.de/herzmusik/heartmusicproject.html, and click on “extra systole”.
You will hear a spectral frequency analysis of heart beat intervals which sonifys activity in the autonomic nervous system regulating heart rate variability. There are three layers of sound, corresponding to sympathetic, sympathetic/parasympathetic and parasympathetic activity. Volume equals intensity. You can hear the system activity building up, “prepareing” for the extrasystol, the sudden jump in pitch. This is new because it shows that extrasystoles are not immediate aberrations in heart regulation but an intrinsic part of that regulation.
And as such, an example of sonification adding to our understanding of the world.
Best wishes,
Michael Falkner
Paracelsus Clinic
CH-9062 Lustmuehle