By Tim Davis

Last November, sparse matrices came together with electronic music to appear on billboards around London. What do sparse matrices have in common with the electronic music scene, you might wonder? The image on the billboards came from an algorithm I had created to translate music into visual art via a simple set of mathematical rules implemented in MATLAB. The story of how I stumbled upon this hobby of creating art with math began with my day job.  I create sparse matrix algorithms and have authored a SIAM book on the topic. The sparse matrix collection I curate is widely used by algorithm developers for benchmarking.

Matrix visualizations. Image credit: Yifan Hu, AT&T and FastCompany

To better understand these matrices, I teamed up with Yifan Hu, currently at Yahoo! Labs, who created beautiful force-directed graph visualizations of the matrices using his algorithm (available in Graphviz). The matrix visualizations have appeared several times in the popular press, such as, in a Fast Company article entitled,“Geeky Science Problems Double as Works of Art.”

Mark Robbins, who works with the London-based design agency Neighbour-UK, came across the images last June and brought up the idea of creating similar images from music for billboards promoting the London Electronic Arts Festival (LEAF 2013).

LEAF 2013 poster. Copyright 2013, Neighbour-UK

An algorithm that could create visual art from music? I love an algorithmic challenge, and I had never done algorithmic art for music before. I was intrigued.

I began with what I knew: the graph theory behind matrix visualizations, and the kinds of graphs that lead to beautiful images. The simple 2D meshes are beautiful, as are graphs with color, symmetry, and some organic irregularity to shake things up and add some action.

A simple set of well-defined rules could start with a fast Fourier transform  (FFT) and generate a 2D mesh from a few simple tones. I could then visualize the resulting graph with Graphviz. I had no idea what the rules would generate for a complex piece of music.

“Cut and Run” visualization. Image credit: Tim Davis, NotesArt Studio

Mark sent me a sample piece of music, and two days after I first put pen to paper and fingertips to keyboard, I  had my first visualizations. A week and a little refining later, I had created a visualization of “Cut and Run” (right) by Kevin Macleod, which appeared on LEAF’s website  and posters and billboards around London. You can listen to Cut and Run, royalty free.

Each image is a translation of an entire song, rather than just a single moment in the song.  There is no “true” visualization of any one piece of music.  The rules are parameterized, and there are more visualizations of any one song than there are particles in the known universe.  Part of the artistic process resides in creating thousands and sometimes tens of thousands of images, and selecting the one that best represents the song.

“Chime” visualization. Image credit: Tim Davis, NotesArt Studio

Electronic music is perfect for this type of algorithmic art because it typically contains regular repetitive bass tracks that render as blue cylindrical meshes, as in “Chime“, left.  At no point in my algorithm do I specify, “Draw a mesh here.” The rules just repeat themselves, creating a mesh. You can listen to the music here.

From the way I constructed the mathematical rules, I expected to see regular square meshes in some music.  I was at first surprised to see cylindrical meshes, until I realized that my rules did not define an edge to the mesh.  The mesh breaks only when the music changes, as in “Cut Trance” to the right. Music is here, royalty free.

“Cut Trance” visualization with close-up (inset). Image credit: Tim Davis NotesArt Studio

I was surprised to see a mesh whose components consist of cycles of 14 edges, which you can see close-up, on the panel on the right.

My rendering of “Metastaseis” (a state of stasis) below, an avant-garde orchestral piece by Iannis Xenakis, appeared in the “Art in Engineering” exhibit at the Harn Museum in October 2013.   Xenakis was a composer and architect who translated his architectural thoughts of space and structure into music while relying heavily on mathematics and algorithms.

“Metastaseis” visualization. Image credit: Tim Davis, NotesArt Studio

While the artwork is not a true inverse, I call it “Inverse in Place”.  In his preliminary sketches for Metastaseis, lines and their intersection in space form hyperbolic paraboloids, which he translated into musical notes and chords. Xenakis described his process as “vision → rules → works of art.” He described an inverse path as “rules → vision.” This piece completes Xenakis’ inverse path by using algorithms to create visual artwork from his music, which you can find on Amazon here.

“Morning Has Broken” visualization. Image credit: Tim Davis, NotesArt Studio

Some renderings come out with amazing connections to the theme of the music. The hymn, Morning Has Broken, was written in the 1930s and set to a Gaelic tune.  Much to my surprise, one of my visualizations (right) appeared with the Celtic trinity symbol.  I’ve looked at hundreds of thousands of images for dozens of songs, and have seen this structure only once.

The combination of art and music allows endless interpretations and some fun with analysis. My daughter, Emily, studies dance at Belhaven University. For one of her choreography projects, she asked me to draw “T.V. Song” by Blue Man Group.

One can see many patterns in the image, but I see a green/blue dancing man with red spiky hair, kicking his leg up in a dance while playing a funky musical instrument (below).  It reminds me of the Native American Kokopelli dancer playing a flute.

“T.V. Song” visualization. Image credit: Tim Davis, NotesArt Studio

The image is surprisingly like Blue Man Group, which is blue and plays funky instruments. Blue Man Group is a strong advocate of STEM (Science, Technology, Engineering, and Math), and STEAM (STEM plus Art) education.  “In Blue Man group, there is no separating art from science from technology from math from engineering,” as the group’s founder Matt Goldman wrote in ARCADE magazine. “It is the integration and recombination of all these disciplines that leads to our show on stages worldwide. One scene from a past Blue Man performance went as far as explaining and demonstrating the phenomenon of synesthesia — hearing colors or seeing sound.  We also give our audiences a tour through the human brain. In all cases, we try to combine science and art in a way that is informative, accurate, funny, accessible and entertaining.”

Blue Man Group. Photo by Andrea LaBarge Mills, courtesy of Blue Man Group

My algorithmic art is a perfect blend of STEAM.  It combines math plus computational science to create visual art hidden in music. Come take a look and see the sound at NotesArtStudio.com.

In his day job, Tim Davis is a professor in the Department of Computer and Information Science and Engineering at the University of Florida, where he does research in mathematical algorithms for solving matrix problems. His work appears in many widely used applications in scientific computing. For example, MATLAB uses his solvers for their matrix equations and Google uses his software to place photos in StreetView in their proper positions. Tim is a Fellow of the Society for Industrial and Applied Mathematics (SIAM) and serves as an associate editor for the SIAM Journal on Scientific Computing and for the ACM Transactions on Mathematical Software.  That’s the serious stuff he does for work. For fun? You’re looking at it!