Copy of Music for the Deaf | Aditya Surendranath

Music for the Deaf

UX Analysis: resolving accessibility (A11y) issues in the music domain for smartphone devices

Research & Design Recommendations

(For Accessible User Experience INF 385T, Fall 22)

Aditya Surendranath + Pankti Dhagia

(For MSIS course in Accessible User Experience INF 385T, Fall 22)

1. Introduction

1.1 Goal:

The goal of this assignment is to identify ways to enhance the music listening experience for the deaf.

The idea is to enhance the music-listening experience by making it more inclusive toward individuals who are deaf or hard of hearing. This enables them to enjoy music whenever and wherever they desire.

The aim is not to come up with a standalone music app for the deaf, but to identify features that can be integrated into existing audio-streaming platforms, based on research in related fields and by integrating existing solutions available in the market.

1.2 Scope:

For the reasons of flexibility of usage, in terms of place and time, with respect to listening to music, it is important to design for a mobile device first. Hence, we are limiting our focus to mobile-first designs for audio-streaming apps, to improve accessibility from the perspective of the deaf community and those hard of hearing.

1.3 Methodology:

Understanding how music is perceived by humans (components/elements of music)
Understanding the science behind hearing and types of deaf (to inform the design of technical solutions)
Identifying sensory substitutes such as haptic technology, sign languages and audio visualizers (identify most viable alternatives) that can be employed to enhance the experience of music within the capabilities of a smartphone.
Analysis of existing assistive and adaptive technologies in the domain.
An audit of accessibility-friendly features for deaf community on popular music apps: Amazon Music, Spotify, Apple Music, and SoundCloud.
Getting inputs on the potential scope/challenges of UX disciplines/teams in improving music experience.
Ideating the integration of identified features on existing platforms.

2. How humans perceive music

Human perception of music is far more complicated than tuning in to tempo- it is a combination of many intangible aspects:

Acoustic qualities like melody, harmony, and timbre also play a significant role.
Our conscious ability to apply symbolic meaning to sounds, lyrics, and the song.
It draws from memories, emotions, pleasure, and reward activities in the brain.

(The Neuroscience Of Musical Perception, Bass Guitars And Drake, Bret Stetka)

The aggregate of the above factors influences one’s perception of music.

More than 1.5 billion people (nearly 20% of the global population) live with hearing loss.

This project seeks to identify ways to employ the above-identified factors to make music perceivable and enjoyable for people with hearing loss.

3. Science behind hearing

Every instrument plays a certain range of frequencies which defines its sound, and along with other elements, like timbre, distinguishes it from other instruments. For example, the kick drum has the lowest frequency, whereas symbols mostly take up high frequencies, while piano is a combination of high and low.

When music is composed, it employs varying frequency ranges of every instrument to sculpt the sound image. This process is called equalization and is achieved by a processor called the music equalizer.
(EQ Explained – Sound Basics with Stella Episode 2 - Youtube)

An able-bodied human can hear between 20-20,000 Hz.

The deaf community is made up of diverse group of people who have a wide range of residual hearing: 125 Hz-8000 Hz

Some might be able to hear lower frequencies like drums or bass (125 Hz-1000 Hz)

Some might be able to hear higher frequencies like vocals or guitar (1000 Hz - 8000 Hz)

Figure 1: A depiction of residual hearing ranges audible to the deaf community.

And of course, everyone can FEEL the vibrations of music at live concerts !!

4. How can we make music streaming perceivable and enjoyable to the deaf?

This section looks at combining the above two research (Sections 2 & 3) to make music streaming more accessible to the deaf community.

We will primarily borrow various factors that make music more perceivable:

The last factor is subjective, influenced solely by individuals feelings, choices and opinions and hence may not be useful in the synthesis of accessibility solutions.

4.1 Means to make music streaming more accessible to the deaf

Acoustic Quality: There are primarily two ways to elevate the acoustic quality of music and make melody, harmony, and timbre comprehensible for the deaf:

First is by letting the users customize amplitude (decibel levels) for the frequency ranges where they experience hearing loss. This can be achieved by giving equalizer controls to the users to enable them to manipulate it according to their needs.

For example: If a user can only hear in the lower frequency range; they can increase amplitudes of the higher frequencies. Similarly, users who can only hear higher frequency ranges may increase the amplitude for lower frequency ranges. This will significantly help elevate the acoustic quality for people with residual hearing range.

Figure 2: Built-in equalizer settings in the Amazon Music app for user customization

The second method to convey melody, harmony, and timbre is by providing sensory substitutes in the form of haptic feedback. One way to deliver the actual feel of the music is by translating music into forms of vibrations.

4.2 Symbolic associations to sounds and lyrics

Conveying the meaning of the music through symbolic/ visual and graphical association can help the deaf connect to music better. Experiments have demonstrated that even by simply observing a speaker, with no auditory information, we can gather important clues about the actual sound of their voice.

(Ken Whytock, Multi-modal perception)

A song is made up of music and lyrics. So the two ways to approach this is by translating music and the lyrics into a visual format.

Translating auditory lyrical content to time-synced text (visual) is pretty straightforward and can be done by auto-generated / curated transcription; which various music streaming platforms have already employed as part of their interface.

Another, more potent way of conveying lyrics is by real-time interpretation in American Sign Language (ASL) using animation/ motion graphics streaming. As per research, sign languages are more efficient in communicating to the deaf as compared to a textual format, as they are more closely associated with conversations. (Sign language reveals the hidden logical structure, and limitations, of spoken language)

Figure 3: (Left) Time-synced lyrics are featured on multiple audio-streaming apps. (Right) Live interpretation of music using American Sign Language (Photo: Randy Holmes/ABC).

Music can be translated into a visual format. Moving scores bridge the gap between sight and sound as shown below by making music understandable and comprehensible by just watching. (http://www.musanim.com/)

Figure 4: Screenshots from visualizations of Clair de lune by Debussy (watch on youtube) and Symphony No. 40 by Mozart (watch on youtube). The featured visualizer engine is discussed in further detail in Section 6.

5. Accessibility Issues in the Domain

Most music streaming platforms limit accessibility features to the translation of lyrics into a textual format, which doesn’t translate the experience in its full form.
Music streaming platforms impede the perception of rhythm, pitch, and audio quality for the deaf.
According to WCAG Success Criterion 1.2.2 (Level A), captions are to be provided for all prerecorded audio content in synchronized media. Although most music streaming platforms have curated lyrical texts congruent to the audio, some still lack this basic feature.
As per WCAG Success Criterion 1.2.6 (Level AAA) sign language interpretation should be provided for all prerecorded audio content in synchronized media. Although it’s a AAA criterion, none of the music streaming apps have currently incorporated this advanced feature.

6. Accessibility Initiatives in the Domain

The VibePlayer is an app that allows the user to experience accurate vibration feedback for audio files such as songs, audiobooks, etc. For example, the user can use the VibePlayer app to listen to a song, with the smart device vibrating to notes of a musical instrument within that song.

(VibePlayer - Audio/Video Player)

6.1.2 Shake - Music Vibration (iOS)

This app calls the iPhone's microphone to listen to the music around the user, converts the music from sound signals to vibration signals, and then uses the iPhone's vibration unit to deliver the music's melody to you. (Shake - Music Vibration on the App Store)

6.2 Lyrics

6.2.1 Shazam (Android & iOS)

Shazam is an application that can identify music, movies, advertising, and television shows, based on a short sample played, using the microphone on the device. It can provide real-time lyrics to songs playing around. Although, a drawback with Shazam is that once it detects a piece of music its timeline freezes and does not change in real time to align with the music if the user chooses to forward it.

(Shazam: Music Discovery)

6.2.2 SoundHound (Android & iOS)

Sound Hound is another popular music discovery app which, like Shazam, also provides real-time lyrics to songs playing around.

(SoundHound - Music Discovery)

6.3 Visualization

6.3.1 Music Animation Machine (MAM)

Music Animation Machine, developed by Stephen Malinowski, uses algorithm-based renderers to visualize the music being played using animated graphical scores. Examples provided in Figure 4 (with video links) are outputs of MAM.

(https://www.musanim.com/)

Figure 5: A snippet from the collection of over 100 visualization techniques/renderers displayed on the MAM website. (http://www.musanim.com/Renderers/)

6.4 Equalizers/Amplifiers

6.4.1 Equalizer and Bass Booster (Android)

This app features ten equalizer presets, a stereo-led Volume Unit (VU) meter, media volume control, and a 5-band equalizer that can control system audio. It is capable of drawing over/overwriting audio settings of background music apps. (Equalizer & Bass Booster)

6.4.2 Sound Amplifier by Google (Android)

Sound amplifier makes conversations more accessible among people who are hard of hearing. It can filter, augment and amplify sounds. It works by reducing noise to recognize audio better. The technology can have an application in music apps to reduce distraction from external noises while listening to music.
(Sound Amplifier)

6.5 Frequency Generator

6.5.1 Frequency Sound Generator (Android)

This app allows users to generate sounds corresponding to different frequencies in the hearing range. The interface allows users to turn a dial to identify their audible range. This technology holds the potential to be leveraged to determine the frequency range audible to the hearing-disabled person.

(Frequency Sound Generator)

Figure 6: A screenshot from the Frequency Sound Generator app. The interface features frequency and decibel levels.

6.6 Sign Language

6.6.1 ASL Music Visualizing Innovators

Sign language innovators are bringing music to the deaf by visualizing rhythms and rhymes through American Sign language. (Youtube: How sign language innovators are bringing music to the deaf)

6.6.2 Hand talk

Hand talk automatically translates text and audio to American sign language (ASL) and Brazilian Sign language (Libras) using artificial intelligence. (Hand Talk)

Figure 7: App screens of Hand Talk featuring a 3D sign language interpreter.

6.1 Haptics

6.1.1 VibePlayer (Android)

7. Team Perspectives

7.1 UX Research

From a UX Researcher’s perspective, issues exist in the domain because:

Deaf users are usually overlooked during the user research phase by ‘audio’ streaming platforms.
There is skepticism regarding the effectiveness and acceptance of alternative approaches by the deaf in their daily life.

The possible challenges and scope of the UX Research team handling this project are outlined below:

7.1.1 Subject matter research

Understanding music perception by humans to identify potential alternatives that can deliver music to the deaf through other sensory modalities.
Types of deaf (to inform the design of technical solutions)
Researching sensory attributes that can substitute for hearing loss.

7.1.2 User research

Understanding how deaf people comprehend music
Understanding what makes people connect to music better (enjoyability)
Understanding whether users are aware of phone/app settings to aid better listening
Evaluate their familiarity with external aids (such as ASL, haptics) to enhance music experience. (To understand their openness to alternative solutions)
User testing to guage which form of alternatives/ combinations of alternatives work better to help the deaf enjoy/perceive music. Eg: Is ASL motion graphic or textual streaming easier for users ? Can music visualization be combined with ASL, or does it lead to heavy cognitive load ? .etc.
Identify key performance indicators (KPIs) for each newly-added features based on user testing.

7.1.3 Competitive Audit

Assessment of strengths and weaknesses of current music apps.
Evaluation of existing assistive and adaptive technologies in the domain.
An audit of accessibility-friendly features for the deaf community on popular music apps Amazon Music, Spotify, Apple Music, and SoundCloud.
Assessment of accessibility initiatives in the domain- sound to sight, sound to feel. etc. Examples- Vibe player, Shazam, Shake, etc.

Figure 8: A competitive audit of deaf-accessibility-centric features on popular music apps

Figure 9: Although Apple Music has an equalizer, it provides only limited options in the form of presets, with no option to customize the amplitudes of specific frequency ranges for use by the deaf community

Figure 10: Screenshots of animated art from Spotify. The animated art does not convey the meaning or qualities of music but supplements the music experience by conveying the theme and emotion.

Figure 11: Screenshots from SoundCloud. Although the app features a music visualizer, it does very little to convey the variations of audio quality in depth.

7.2 UX Design

From a UX Designer’s perspective, issues exist in the domain because:

The design process is structured around a primary persona, who in the case of an audio-streaming app is assumed to have hearing capability. Not having considered accessibility in the user flow is the mistake that needs to be rectified.
While assistive features such as lyrics may still get included, adaptive features are very often neglected in the process.

From a UX Designer’s perspective, the major responsibilities in adopting accessibility-friendly features into an existing audio-streaming platform may include:

Ideating the integration of sensory substitutes such as haptic technology, sign languages, and audio visualizers to enhance the experience of music within the present capabilities of a smartphone.
Identify and distinguish between assistive features and adaptive features to determine user flow.
Translate user experience goals into intuitive interfaces that support and enhance the music streaming experience for the deaf community.
Ensure compliance of new features with existing design patterns.
Developing user flows diagrams, sitemaps, conceptual wireframes, prototypes, and mockups to guide the targetted users to the adaptive/ assistive features like haptic feedback, ASL etc.
Partnering with UX Research team to determine the type of music visualization based on usability testing, from the large pool of available music visualizers (MAM, etc.).
Develop prototypes for recommended features and test them with deaf/hard of hearing users by partnering with the UX Research team.

Assessment of the possibility of combining already existing initiatives in different domains. For instance, allowing deaf users to set their hearing range on a frequency generator can auto-calibrate the equalizer presets using AI to deliver a wholesome experience across different genres.

Figure 12: (Left) 2 existing utilities that can be combined in a single ecosystem.

  Figure 13: (Below) A suggestive flowchart for user onboarding

7.3 UX Content

From a contents perspective, issues exist in the domain because:

Since audio-streaming apps’ primary focus is audio content, deaf users may be innately excluded.
Stakeholders are skeptical about investing time and resources in the creation of alternative content as they do not see a high monetary return.

From a contents perspective, the major challenges in adopting accessibility-friendly features into an existing audio-streaming platform may include:

Developing multiple content types such as ASL, visualizer, animated art, etc. for platform integration. While ASL and visualizer are algorithmic, requiring collaboration with UXD and FED, the latter would rely on manual content creation.
Recruitment of sub-teams under the umbrella of a content team like ASL specialists, data visualizers etc.
Generating or conducting quality control for auto-generated lyrics/transcripts for all contents including music and podcasts.
Evaluating content comprehensibility based on qualitative and quantitative data presented by UX Research to ensure it is consumable and enjoyable. For instance; motiona-based
Ensure that contents in newly-added adaptive features adheres to existing guidelines, including accessibility, design system, localization, and legal.
Crafting error messages and instructions adhering to style guidelines and principles in case the device is not compatible with haptics, or any recommended features.
Crafting marketing landing pages and accessibility documentation to ensure that the users are made aware of the new content features and objectives
Craft success measures, for new types of content like ASL motion graphics and music visualization, in collaboration with product partners.
Ensure that choice of color/background expresses the harmony and tonality of the music being played by discussing with design and research teams.

7.4 Front-end Development

From a front-end developer’s perspective, issues exist in the domain because:

Designing the implementation of alternative methods is time-consuming.
The execution of alternatives is extremely iterative and does not ascertain intended results.

From a front-end developer’s perspective, the major challenges in adopting accessibility-friendly features into an existing audio-streaming platform may include:

Collaborating with the back-end team in conceptualizing/developing algorithms for auto-generation of music visualization and ASL.
Collaborating with the UX Design team to discuss the feasibility of proposed assistive and adaptive design feature options.
Assessing feasibility from a technical perspective (frame rate of motion graphics, etc) and themes (backgrounds, solids or lines, etc) of algorithm-based ASL interpreter.
Collaborating with the UX Content team to test the algorithm-based prototypes with different types/genres/kinds of music.
Ensuring the ASL/ transcripts/visualization solution is scalable to all kinds of audio
Collaborating with back-end (algorithm/logic), UX Design (usability), and UX Research (testing) teams for conceptualizing and testing features such as calibration of equalizers with identified frequency ranges of users.

8. Recommendations

Haptic features: Leveraging the smartphone’s in-built capability for vibration can be employed to transfer rhythm and audio quality to deaf users.
The ideal solution may be envisioned as a combination of assistive and adaptive technologies. Features like haptic feedback and ASL interpretation can be considered adaptive in nature while music visualization and animated art are assistive features.
For users who are hard of hearing, it is recommended to find their audible frequency range using a frequency generator, to auto-calibrate existing equalizer presets to their audible range using AI.
An algorithm-based ASL generator to translate songs real-time, similar to Music Animation Machine.

9. Conclusion

Research suggests music is perceived by humans as a combination of acoustics, lyrics, and context. A solution that can substitute music experience for the deaf should be able to replace all three.
The deafness levels may vary amongst users. Audibility depends on the frequency range each user can comprehend.
Perception can be enhanced by relying on multiple sensory modalities as a replacement for one. The more cues provided, the better the comprehension.
Although recent research suggests that sign language is better at conveying certain ideas to the listener, one needs to note that less than 500,000 (<1%) people belonging to the deaf community in the US use ASL, and therefore it cannot be a blatant replacement to textual data.
Based on the audit of popular music apps, most lack equalizer settings that are robust enough to serve the needs of the deaf community.
Assistive technologies with the application may include time-synced lyrics, music visualization, animated art, etc. that may enhance the experience for all users while helping the deaf community gain a better experience.
Adaptive technologies may include haptics and sign language interpreters, which can help deaf users but may not help others.