Back in the 1970s Stephen Malinowski came up with the Music Animation Machine, an animated graphical interface that represents a musical performance. Instead of traditional musical notation, Malinowski’s interface uses geometric shapes on a bar graph to visualize sounds, making the complexities of Tchaikovsky’s “Dance of the Sugar Plum Fairy” or Vivaldi’s “Four Seasons” easier and more intrinsic to understand for those of us who aren’t versed in the intricacies of musical notation.
Initially conceived as an animation, he used his basic programming skills—honed on an Atari 800—and made the idea into a piece of software. Björk spotted his videos on YouTube and got him on board to provide the animations for the apps in her Biophilia project.
Even though Malinowski came up with the Music Animation Machine 40 years ago, he’s still bursting with ideas, releasing apps like Harmonizer, which is a pitch trainer for infants, and is currently working on a live performance version of his animation software. We chatted with him via Skye about the Music Animation Machine and how it came to be featured in Biophilia.
The Creators Project: What made you want to create the Music Animation Machine?
Stephen Malinowski: Mainly from my inability, when I was younger, to follow complicated scores. When you’re watching an orchestral score there’s multiple instruments proceeding in parallel and you have to direct your attention around different places at different times to see where the action is—and then integrate that into one image of what the music is doing. It’s a very complex task and I was just a beginner musician back then—it was the 70s and I was in my teens—and so I couldn’t follow a very complicated score. So my idea was, let’s try and make a score that has all the information that a complicated score has, but in a simpler view.
How did you go about creating it once you had the idea?
Since I was a pianist, the first thing I tried was a piano reduction. Back then, the way you got to know a complicated piece of music is someone would arrange an orchestral score for a solo piano. Then you’d play it yourself and as much of the music as could be jammed in one player’s range would be presented just in two staves instead of one staff per instrument.
And then that was a step in the direction of putting all the instruments in the same, what you’d call, pitch space. That is, any instrument playing the same pitch would appear in the same place on the staff.
And so I did that, but what immediately happened was the problem where there were so many notes happening at once that you couldn’t see the melodic trajectories. That is, if you had two or more instruments playing in the same region, there would just be a note head for each one and a bunch of empty space and you wouldn’t be able to see where the notes go easily. So that’s what led to doing the bar graph, which is what I worked on for the first few years when I started implementing it as software.
So it wasn’t initially conceived as software?
No, the first version I did was just hand drawn on a long scroll of paper. So 20 feet long by four inches wide, and I would unroll it on the floor and I would put the music on the stereo and I would just walk by to where the music was at so the person following it would know.
What made you turn it into software?
In about 1981/2 I thought I would do it as a stop-motion animation and so I collected materials to do that—little blocks that I could move around to represent the notes. And the idea was to film it one frame at a time that way. But I had a friend who was a computer programmer and said, “This will be much easier with a computer, why don’t you buy a computer and learn how to program and do it that way?”
Above are some sketches from Malinowski’s notebook. He explains: “I was thinking about how to represent dynamics with note shapes. It did not materialize the way I had in mind”—the video below shows some of the results.
How does it differ from conventional notation?
The difference is, in a normal score you have symbolic stuff. So, for example, if you have a note that’s longer there’s a symbol which means “this is longer”, it doesn’t look any longer. So you have to learn all the symbols and their interrelations. Whereas in my score, the graphical scores, there’s no symbol, it’s all graphical so there’s nothing to learn.
How did you come to work on Biophilia? I read that Björk saw your YouTube videos then wanted you involved?
Yeah, I’m pretty sure that Björk saw the videos and liked them, then when she was about half-way through the project—I wasn’t in it from the beginning—she realized that it would be fun to have it in and so she asked her manager to contact me. And I said sure, I can do that.
With the Biophilia project already begun, how did they integrate your work into it?
There were two separate drafts, the project took a turn in the middle. The idea originally was that privy to the songs and the album—10 songs—they would give me the data I needed to make my animations: the audio recording, the midi file version of it, and then the lyrics. So they were going to give me those 10 things and I’d make 10 videos like I make for YouTube and that would be the deal, in a kind of work-for-hire kind of thing. And she sent me the first track “Crystalline” and I sent them the video and they said, “This isn’t going to work for us, these things are 250 megabytes, we wanted to put them on the iPad. They’re much too big to distribute.” And at that point it was decided that the only way it was going to work on the iOS devices was to make it be an app instead.
“When I went on a vacation trip last Thanksgiving, I took books of paintings by Klee and Kandinsky with me, and spent a few hours at my friends’ house in Eureka looking for things to steal—here is one page of those.” (Above)
“You can see two of the ‘completed’ circles have been checked on that page”—the results of those two ideas led to the above.
What happened to the videos?
The videos I ended up making anyway because they project them in the performances for the performers to look at as a sort of teleprompter. Björk doesn’t read notation very well and she liked my videos as a score. They work a lot better, because if you think about a normal score you have a page of music and there’s maybe a minute worth of music on one page and it all kind of looks the same. Whereas mine, it’s focusing on a very small area, maybe 10 seconds of music, and it’s much bigger and more visible. So even if they were equivalent in some abstract sense, as a practical matter you can see mine and see what’s going on in it and the fact that the notes light up when they play makes them much more visible across a distance. And mine have the lyrics with the music too.
There’s a lot going on in the apps, a lot of collaborations. What did you think of the end result as a new kind of way to release an album?
That it’s fabulous. It’s the first of its kind, it’s very creative. And the fact that it was done by a very small team [is incredible]. If you look at the amount of people involved in the apps, it’s around one person per app. Whereas when you think of a software title that comes out for the game market, you have twenty people working on an app and working on it for a year as opposed to one person working on it for a few weeks or a month. So that meant they were very modest, relatively simple apps compared to the industry standard of what a game is. But within that constraint—limited budget, limited time—I think it’s wonderful, very creative and amusing. Just the idea that to give someone an interactive experience with a piece of music you would make an application which would let them go into a song—that’s a wonderful idea and very much in my aesthetic as a presenter and as a music educator.
That’s one thing I haven’t mentioned in any previous articles, is that when I left school a lot of my work involved teaching music. So we’re very much on the same page—Björk and I—as far as wanting someone to have an experience of music where they can get into it and see it in a way that’s closer to how a musician would see it. As a composer would see it and be able to play with the elements that way.
It is much more like a game that way, with the interactivity and immersiveness.
One of the things about a musical score is that if there’s something in there that you’re not perceiving, because it requires more attention than you’re giving it, it gives you another way in to hear something. I’ve had people say, about pieces of music they’ve loved and known all their life, that after watching one of my videos they see something in it that they’d never heard before and now they hear it every time they listen to that piece of music. It’s part of the education, people don’t realize that when you’re listening to a piece of music you’re learning how to hear it, how to make sense of it. And when a piece of music is unfamiliar and you don’t like it, you don’t like it because you can’t hear it, you don’t understand it and it’s not music to you.