Project Proposal — Text to Music

My term project involves developing an algorithm that generates music from textual input. The main objective of this project / interactive artwork is to examine the relationship between text and music. Text and music exhibit many similarities, as humans are capable of deriving meaning from both their visual and audible representation (In the same manner that reading based comprehension can be acquired, hearing music that is visually represented on paper in one’s mind in can also be acquired. Furthermore, both music and text are built on a metrical grid [3]. Despite their similarities, there are important differences between the two mediums. Although music and text both convey meaning, the communication of explicit constructs and cohesive arguments is a task ill-suited to the musical domain.

Research has shown that a repeated sentence fragment tends to be perceived as a melody, provided that the sentence fragment is repeated without any manipulations to the audio content [2]. Deutsch, Henthorn,and Lapidis suggest that while listening to a normal conversation pitch salience is inhibited, allowing the listener to focus on attributes of speech that convey information. Exact repetition increases pitch salience, which causes speech to be perceived as song. Although I could not find any research to back up this claim, it seems that we tend to focus on the musicality of a language that we are unfamiliar with, likely due to the fact that we can’t extrapolate much information from it.

Transprose, a system created by Davis and Mohammad, transforms text into music based on the emotions which characterize the textual input [1]. Davis and Mohammad make assumptions on the emotional effect of particular compositional choices, associating a higher pitch range with positive emotions and a lower pitch-range with negative emotions. Based on the textual input, predetermined musical constraints are used to generate a musical representation. Unfortunately, the output of the system does not reflect the characteristics of the text itself (beyond the general sentiment), and as a result, different textual inputs can sound quite similar.

In general, the aim is to create a transformation which maps text to musical chords. Then, based on the semantic analysis of the input text, performance aspects like articulation, volume, and tempo will be manipulated. Since poetry it is heavily influenced by the way in which words sound, and has an inherently musical quality, it will likely be used as the source for the music generation algorithm.

[1] Hannah Davis and Siaf M. Mohammad. “Generating Music from Literature”. In: 3rd Workshop onComputational Linguistics for Literature (CLfL 2014) (2014), pp. 1–10. arXiv: arXiv:1403.2124v1.

[2] Diana Deutsch, Trevor Henthorn, and Rachael Lapidis. “Illusory transformation from speech to song.”In: The Journal of the Acoustical Society of America 129.4 (2011), pp. 2245–2252. issn: 00014966. doi:10.1121/1.3562174.

[3] Ray Jackendoff. “Parallels and Nonpararelles between Language and Music”. In: Music Perception 26.3(2008), pp. 195–204.