Music for the Deaf

UX Research Report: resolving accessibility (A11y) issues in the music domain for smartphone devices

Design Research & Recommendations

Aditya Surendranath + Pankti Dhagia

For MSIS course in Accessible User Experience INF 385T, Fall 22

1. Introduction

1.1 Goal:

More than 1.5 billion people (nearly 20% of the global population) live with hearing loss. The goal of this project is to identify ways to enhance the music listening experience for the deaf.

The idea is to enhance the music-listening experience by making it more inclusive toward individuals who are deaf or hard of hearing. This enables them to enjoy music whenever and wherever they desire.

The aim is not to come up with a standalone music app for the deaf, but to identify features that can be integrated into existing audio-streaming platforms, based on research in related fields and by integrating existing solutions available in the market.

1.2 Scope:

Focus on mobile-first audio-streaming app designs.

For the reasons of flexibility of usage, in terms of place and time, with respect to listening to music, it is important to design for a mobile device first. Hence, we are limiting our focus to mobile-first designs for audio-streaming apps, to improve accessibility from the perspective of the deaf community and those hard of hearing.

1.3 Methodology:

Understanding how music is perceived by humans.
Understanding the science behind hearing and types of deaf (to inform technical solutions)
Identifying sensory substitutes such as haptic technology, sign languages, and audio visualizers that can be employed to enhance the experience of music within the capabilities of a smartphone.
Analysis of existing assistive and adaptive technologies in the domain.
An audit of accessibility-friendly features for the deaf community on popular music apps: Amazon Music, Spotify, Apple Music, and SoundCloud.
Getting inputs on the potential scope/challenges of UX disciplines in improving music experience.
Ideating the integration of identified features on existing platforms.

2. How humans perceive music

Human perception of music is far more complicated than tuning in to tempo- it is a combination of many intangible aspects.

Acoustic qualities like melody, harmony, and timbre also play a significant role.
Our conscious ability to apply symbolic meaning to sounds, lyrics, and the song.
It draws from memories, emotions, pleasure, and reward activities in the brain.

The aggregate of the above factors influences one’s perception of music.

3. Science behind hearing

Instruments have a distinct sound based on the range of frequencies they play, along with other elements like timbre.

For example, the kick drum has low frequencies, cymbals have high frequencies, and the piano produces a combination of both.

When music is composed, it employs varying frequency ranges of every instrument to sculpt the sound image. This process is called equalization and is achieved by a processor called the music equalizer.
(EQ Explained – Sound Basics with Stella Episode 2 - Youtube)

An able-bodied human can hear between 20-20,000 Hz.

The deaf community is made up of a diverse group of people who have a wide range of residual hearing: 125 Hz-8000 Hz

Some might be able to hear lower frequencies like drums or bass (125 Hz-1000 Hz)

Some might be able to hear higher frequencies like vocals or guitar (1000 Hz - 8000 Hz)

Figure 1: A depiction of residual hearing ranges audible to the deaf community.

And of course, everyone can FEEL the vibrations of music at live concerts !!

4. How can we make music streaming perceivable and enjoyable to the deaf?

The last factor is subjective, influenced solely by individuals feelings, choices and opinions and hence may not be useful in the synthesis of accessibility solutions.

4.1 Means to make music streaming more accessible to the deaf

Acoustic Quality: There are primarily two ways to elevate the acoustic quality of music and make melody, harmony, and timbre comprehensible for the deaf:

First is by letting the users customize amplitude (decibel levels) for the frequency ranges where they experience hearing loss. This can be achieved by giving equalizer controls to the users to enable them to manipulate it according to their needs.

If a user can only hear in the lower frequency range; they can increase amplitudes of the higher frequencies.

Similarly, users who can only hear higher frequency ranges may increase the amplitude for lower frequency ranges. This will significantly help elevate the acoustic quality for people with residual hearing range.

Figure 2: Built-in equalizer settings in the Amazon Music app for user customization

The second method to convey melody, harmony, and timbre is by providing sensory substitutes in the form of haptic feedback. One way to deliver the actual feel of the music is by translating music into forms of vibrations.

4.2 Symbolic associations to sounds and lyrics

Conveying the meaning of the music through symbolic/ visual and graphical association can help the deaf connect to music better.

Experiments have demonstrated that even by simply observing a speaker, with no auditory information, we can gather important clues about the actual sound of their voice.

A song is made up of music and lyrics. So the two ways to approach this is by translating music and the lyrics into a visual format.

Translating auditory lyrical content to time-synced text (visual) is pretty straightforward and can be done by auto-generated / curated transcription; which various music streaming platforms have already employed as part of their interface.

Another, more potent way of conveying lyrics is by real-time interpretation in American Sign Language (ASL) using animation/ motion graphics streaming.

As per research, sign languages communicate better than text to the deaf, as they are more closely associated with conversations.

Figure 3: (Left) Time-synced lyrics are featured on multiple audio-streaming apps. (Right) Live interpretation of music using American Sign Language (Photo: Randy Holmes/ABC).

Music can be translated into a visual format. Moving scores bridge the gap between sight and sound as shown below by making music understandable and comprehensible by just watching. (http://www.musanim.com/)

Figure 4: Screenshots from visualizations of Clair de lune by Debussy (watch on youtube) and Symphony No. 40 by Mozart (watch on youtube). The featured visualizer engine is discussed in further detail in Section 6.

5. Accessibility Issues in the Domain

Music platforms only offer text lyrics, not the full experience for the deaf.
Streaming platforms hinder the perception of rhythm, pitch, and audio quality.
WCAG AA requires captions for audio in synchronized media, but some platforms lack this feature.
WCAG AAA requires sign language interpretation for prerecorded audio in synchronized media, but no platforms have incorporated this feature.

6. Accessibility Initiatives in the Domain

The VibePlayer is an app that allows the user to experience accurate vibration feedback for audio files such as songs, audiobooks, etc. For example, the user can use the VibePlayer app to listen to a song, with the smart device vibrating to notes of a musical instrument within that song.

(VibePlayer - Audio/Video Player)

6.1.2 Shake - Music Vibration (iOS)

This app calls the iPhone's microphone to listen to the music around the user, converts the music from sound signals to vibration signals, and then uses the iPhone's vibration unit to deliver the music's melody to you. (Shake - Music Vibration on the App Store)

6.2 Lyrics

6.2.1 Shazam (Android & iOS)

Shazam is an application that can identify music, movies, advertising, and television shows, based on a short sample played. It can provide real-time lyrics to songs playing around.

(Shazam: Music Discovery)

6.2.2 SoundHound (Android & iOS)

Sound Hound is another popular music discovery app which, like Shazam, also provides real-time lyrics to songs playing around.

(SoundHound - Music Discovery)

6.3 Visualization

6.3.1 Music Animation Machine (MAM)

Music Animation Machine, developed by Stephen Malinowski, uses algorithm-based renderers to visualize the music being played using animated graphical scores. Examples provided in Figure 4 (with video links) are outputs of MAM.

(https://www.musanim.com/)

Figure 5: A snippet from the collection of over 100 visualization techniques/renderers displayed on the MAM website. (http://www.musanim.com/Renderers/)

6.4 Equalizers/Amplifiers

6.4.1 Equalizer and Bass Booster (Android)

This app features ten equalizer presets, a stereo-led Volume Unit (VU) meter, media volume control, and a 5-band equalizer that can control system audio. (Equalizer & Bass Booster)

6.4.2 Sound Amplifier by Google (Android)

Sound amplifier makes conversations more accessible among people who are hard of hearing. It can filter, augment and amplify sounds. It works by reducing noise to recognize audio better. The technology can have an application in music apps to reduce distraction from external noises.
(Sound Amplifier)

6.5 Frequency Generator

6.5.1 Frequency Sound Generator (Android)

This app allows users to generate sounds corresponding to different frequencies in the hearing range. The interface allows users to turn a dial to identify their audible range.

(Frequency Sound Generator)

Figure 6: A screenshot from the Frequency Sound Generator app. The interface features frequency and decibel levels.

6.6 Sign Language

6.6.1 ASL Music Visualizing Innovators

Sign language innovators are bringing music to the deaf by visualizing rhythms and rhymes through American Sign language. (Youtube: How sign language innovators are bringing music to the deaf)

6.6.2 Hand talk

Hand talk automatically translates text and audio to American sign language (ASL) and Brazilian Sign language (Libras) using artificial intelligence. (Hand Talk)

Figure 7: App screens of Hand Talk featuring a 3D sign language interpreter.

6.1 Haptics

6.1.1 VibePlayer (Android)

7. Team Perspectives

We interviewed experts from UX Research, UX Design, Content, and Dev teams to understand their scope and challenges.

7.1 UX Research

From a UX Researcher’s perspective, issues exist in the domain because deaf users are usually overlooked during the user research phase by ‘audio’ streaming platforms.

Challenges

Understanding how deaf people comprehend music
Understanding what makes people connect to music better (enjoyability)
Understanding whether users are aware of phone/app settings to aid better listening
Evaluate their familiarity with external aids (such as ASL, haptics) to enhance music experience. (To understand their openness to alternative solutions)
User testing to guage which form of alternatives/ combinations of alternatives work better to help the deaf enjoy/perceive music. Eg: Is ASL motion graphic or textual streaming easier for users ? Can music visualization be combined with ASL, or does it lead to heavy cognitive load ? .etc.
Identify key performance indicators (KPIs) for each newly-added features based on user testing.

7.1.3 Competitive Audit

An audit of accessibility-friendly features for the deaf community on popular music apps Amazon Music, Spotify, Apple Music, and SoundCloud.

Figure 8: A competitive audit of deaf-accessibility-centric features on popular music apps

Figure 9: Although Apple Music has an equalizer, it provides only limited options in the form of presets, with no option to customize the amplitudes of specific frequency ranges for use by the deaf community

Figure 10: Screenshots of animated art from Spotify. The animated art does not convey the meaning or qualities of music but supplements the music experience by conveying the theme and emotion.

Figure 11: Screenshots from SoundCloud. Although the app features a music visualizer, it does very little to convey the variations of audio quality in depth.

7.2 UX Design

From a UX Designer’s perspective, issues exist in the domain because:

The design process is structured around a primary persona, who in the case of an audio-streaming app is assumed to have hearing capability.
While assistive features such as lyrics may still get included, adaptive features are very often neglected in the process.

Challenges

Ideating the integration of sensory substitutes such as haptic technology, sign languages, and audio visualizers to enhance the music experience.
Partnering with UX Research team to determine the type of music visualization based on usability testing, from the large pool of available music visualizers (MAM, etc.).
Develop prototypes for recommended features and test them with deaf/hard of hearing users by partnering with the UX Research team.

Possible solutions by combining features

Allowing deaf users to identify their hearing range on a frequency generator can auto-calibrate the equalizer presets using AI to deliver a wholesome experience across different genres.

Figure 12: (Left) 2 existing utilities that can be combined in a single ecosystem.

  Figure 13: (Below) A suggestive flowchart for user onboarding

7.3 UX Content

From a contents perspective, issues exist in the domain because:

Since audio-streaming apps’ primary focus is audio content, deaf users may be innately excluded.
Stakeholders are skeptical about investing time and resources in the creation of alternative content as they do not see a high monetary return.

Challenges

Developing multiple content types such as ASL, visualizer, animated art, etc. for the platform.
Generating or conducting quality control for auto-generated lyrics/transcripts for all contents including music and podcasts.
Ensure that contents in newly-added adaptive features adhere to existing guidelines, including accessibility, design system, localization, and legal.
Crafting marketing landing pages and accessibility documentation to ensure that the users are made aware of the new content features and objectives
Craft success measures, for new types of content like ASL motion graphics and music visualization, in collaboration with product partners.

7.4 Front-end Development

From a front-end developer’s perspective, issues exist in the domain because:

Designing the implementation of alternative methods is time-consuming.
The execution of alternatives is extremely iterative and does not ascertain intended results.

Challenges

Collaborating with the back-end team in conceptualizing/developing algorithms for auto-generation of music visualization and ASL.
Collaborating with the UX Design team to discuss the feasibility of proposed assistive and adaptive design feature options.
Assessing feasibility from a technical perspective (frame rate of motion graphics, etc) and themes (backgrounds, solids or lines, etc) of algorithm-based ASL interpreter.
Collaborating with the UX Content team to test the algorithm-based prototypes with different types/genres/kinds of music.
Ensuring the ASL/ transcripts/visualization solution is scalable to all kinds of audio
Collaborating with back-end (algorithm/logic), UX Design (usability), and UX Research (testing) teams for conceptualizing and testing features such as calibration of equalizers with identified frequency ranges of users.

8. Recommendations

Haptic features: Leveraging the smartphone’s in-built capability for vibration can be employed to transfer rhythm and audio quality to deaf users.
The ideal solution may be envisioned as a combination of assistive and adaptive technologies. Features like haptic feedback and ASL interpretation can be considered adaptive in nature while music visualization and animated art are assistive features.
For users who are hard of hearing, it is recommended to find their audible frequency range using a frequency generator, to auto-calibrate existing equalizer presets to their audible range using AI.
An algorithm-based ASL generator to translate songs in real-time, similar to MAM.

9. Conclusion

A solution for the deaf must replace acoustics, lyrics, and context.
Audibility varies depending on individual frequency range comprehension.
Combining sensory cues enhances perception.
ASL cannot replace textual data due to limited usage in the US deaf community.
Popular music apps lack robust equalizer settings for the deaf.
Assistive technologies like time-synced lyrics, music visualization, and animated art can enhance experience for all users.
Adaptive technologies like haptics and sign language interpreters help deaf users but may not help others.