This presentation contains binaural audio demonstrations that are best experienced with headphones. Hello. I'm Ryan from the University of Illinois and it's a pleasure to be here at the Educational Audiology Association Virtual Summer Conference. I'm a postdoctoral researcher in electrical engineering and my research focuses on signal processing for hearing aids and other listening technologies. I'm also a person with hearing loss. I've worn hearing aids since I was a teenager and I've always been frustrated by how poorly they work in noisy environments like restaurants and parties, so, as a graduate student, I studied ways of using microphone arrays to improve their performance in noise.
I worked with a team of students to design proof-of-concept listening devices with dozens of microphones that could separately process individual sound sources from a noisy mixture. I finished my program in December of 2019, just a few weeks before the first case of COVID-19 was reported. I wasn't sure what I wanted to do next, so I signed up for a technology translation program run by the National Science Foundation. It's designed to get scientists out of the laboratory and talking to real people, both consumers and people in industry. That seemed like a great opportunity to connect with the hearing loss community and learn what's going on in the industry. Over the past year I've talked with more than 100 people, from
engineers and audiologists to community activists to kids who are newly diagnosed with hearing loss. When I started the program I expected I would hear a lot about the new trend toward over-the-counter and direct-to-consumer business models, new connectivity features, and artificial intelligence. I thought the biggest complaints from users would be about noisy places like convention centers and restaurants. Then, three months later, everything changed.
There were no more conferences, no more crowded restaurants or busy train stations. Users weren't worried about noisy crowds, they were worried about face masks and captioning, and the tech industry started pouring resources into conferencing equipment. It was fascinating watching all those changes happening. Now the world is starting to get back to normal, but I think the pandemic will have a lasting impact on listening technology. In this talk I'll weave together personal experiences as a person with hearing loss, technical concepts from engineering, and insights from the people I've met over the last year.
The pandemic changed the way we all communicate. If you think about it, the communication challenges brought on by masks and lockdowns are a lot like the ones people with hearing loss deal with every day. Masks muffle high frequency sounds just like the most common type of hearing loss, and now everyone has to rely on technology to communicate, just like we always have. And
because the pandemic affected everyone, society put a lot of resources into new audio technology. For example, classrooms and conference rooms have a lot more microphones than they did before. Going forward, we can leverage that new technology to improve accessibility. 2020 was always going to be a disruptive year for listening technology thanks to regulatory changes, new wireless standards, and competition from consumer electronics companies. The pandemic will change things even more, but I'm optimistic that they'll be changes for the better. I want to start with the most visible - and audible - change caused by the pandemic: face masks. By late spring last year, health officials were recommending that people wear
face coverings over their noses and mouths whenever they were around other people. Masks turned out to be very effective against the virus, but they also make it harder to hear. Most masks are opaque, so they block visual cues like lip movements that help us understand speech.
They also muffle sound, especially the high frequencies that are especially challenging for people with hearing loss. There had been a few studies on the acoustic effects of medical and industrial face masks, but at that stage of the pandemic the vast majority of people were wearing cloth masks, many of them homemade. A friend of mine, who works in a school that serves children with hearing loss, asked me what kind of masks their staff should use when they start teaching again. I didn't know what to tell her, so I gathered as many masks as I could find and took them down to the lab. I included some of the transparent face masks that were then just starting to catch on. I did two types of experiment. First, to get objective, repeatable measurements, I put the
masks on a head-shaped loudspeaker that was built by a student from our college of art and design. The speaker can produce the exact same sound every time, so it provides a fair comparison between different masks, but it isn't a perfect analogue for a human head or a human vocal tract. For less consistent but more realistic measurements, I also wore the masks myself. The relative performance between masks was fairly consistent between the two experiments.
To rank the masks, I set up a microphone six feet away. I also tested wearable microphones to see if those would work well with face masks. Here's a plot of frequency responses for a few of the masks. Each curve is the difference between the speech spectrum with a mask and the spectrum without a mask. With the exception of the face shield, which is just a nightmare acoustically, all the masks had similar effects. They didn't affect sound much at all
below 1000 Hertz and they muffled high frequencies above around 2000 Hertz. The amount of attenuation depends on the material the mask is made of. Here is a ranking of most of the masks we've tested to date.
The blue surgical masks are my go-to mask since they have very little effect on sound. Tight-fitting N95 masks muffle a lot more sound, which I've definitely noticed in doctors' offices. If you want a reusable mask, pay attention to the weave of the fabric. Loosely woven fabric,
like cotton jersey used in t-shirts, is pretty breathable and lets a lot of sound through. The tighter weaves used in denim and bed sheets attenuate sound more strongly. On the right side of the figure you'll see four types of flannel with different weaves and number of layers. The number of layers does affect attenuation, but not as much as the weave. Four layers of light flannel was better than two layers of heavy flannel, for example. The three transparent masks we tested performed worst. Plastic might let light through, but it's very bad for sound waves.
There have been a lot more transparent masks developed since I did these tests, so some newer ones might work a bit better. I haven't tested many new commercial designs, but there is one set I want to mention. These mask prototypes were designed by a team of high school students here in Champaign, and they combine visually transparent plastic with acoustically transparent cloth. This striped design works pretty well acoustically,
and you can still see lip motion through it. It's a really impressive design. One of the unique features of our study was an experiment on directivity. Masks are too thin to absorb much acoustic energy; they deflect it. We wanted to know how masks affected sound to the side of the talker and above and below the mouth.
To measure directivity, we put the head-shaped loudspeaker on a turntable in our lab. We measured the sound at different angles in 15 degree increments. This polar plot shows the results. The black curve is the speaker with no mask. It's not quite the same as the directivity of a human talker, but it's reasonably close. You can see from the purple and green curves that the masks have the strongest attenuation in front of the talker and a weaker effect behind and to the sides. The shield, meanwhile, actually reflects sound backward. These results suggest that we might be able to make up for the attenuation in front of a mask by capturing sound somewhere else, like the cheek or forehead or chest.
Now, the loudspeaker doesn't have a chest so for this next experiment we used the human talker. Microphones were placed on the lapel, which is commonly used for presentations and broadcasting, and on the cheek and forehead, which are sometimes used in theater. We also tried a mic just in front of the mouth, like a fitness instructor headset. Here is a plot for the plastic window mask. Each curve compares the spectrum at the same microphone with and without a mask. The headset mic has just as much attenuation as the distant mic, though of
course that mic gets more sound to begin with. The mic on the cheek was under the mask, so the reflected sound was amplified, but it was still distorted. The lapel and forehead mics weren't affected much at all. Since lapel mics are already so commonly used, they seem like a great choice. These plots compare the effects of a few masks on distant mics and on lapel mics. Each curve is the difference between the spectrum with a mask and without a mask, so the overall spectrum of a lapel mic is still different from that of a distant mic, but it isn't affected much by a mask. That means if you already have a chest-worn
microphone that works well without a mask, it should still work well with a mask. To summarize, surgical masks and loosely woven cloth masks work best for sound transmission, but they block visual cues. Masks with plastic windows are more visually transparent but they block a lot of sound. The choice of mask depends on whether the listener relies more on audible or visual cues. Personally, I understand people better with the surgical mask than the window masks, but I've heard from other people who prefer the window version. With all types of mask, amplification can help compensate for the effects of the mask, so if you have a microphone available you should use it.
Since this research came out last summer, I've had a lot of questions from people asking about options for microphones to wear with masks. In higher education we were fairly well prepared: A lot of smaller classes were moved to large auditoriums to promote social distancing, and most of those spaces have public address or voice lift systems built in. Corporate offices tend to have similar setups. In K-12 the options depend on what resources a school has available. I heard from some teachers that were ordering home karaoke systems for their classrooms because they didn't have anything better. I think the people who have the hardest time with masks are essential workers like grocery store employees who have to interact with a steady stream of customers all day. You have no control over what kind of mask customers wear or how loudly they talk, and you don't have any good options for amplification. If it were me, I might tape a microphone to the end of a
six-foot pole and stick it in people's faces, but I don't know if the manager would appreciate that. Anyway, that got me interested in assistive listening technologies like remote microphones. There's a huge variety of these devices out there, all with different strengths and weaknesses.
In the next part of the talk, I'll talk about the wireless assistive listening systems available today and how they might evolve over the next few years. When most people think of listening technology, they think of hearing aids and cochlear implants, which are designed to be worn all day and to help in a broad range of listening situations. Wireless assistive listening systems are more situational, and help users who want to hear a specific sound like a television, a phone call, a teacher, or a public announcement. Those systems have not gotten much attention from signal processing researchers, which is a shame because frankly there's a lot of room for innovation. Among the adults with hearing loss I talked to, the most popular listening tech accessory was the phone streamer. Here's mine. It connects my hearing aids to my smartphone or laptop over Bluetooth so I can listen to music or take calls with my hearing aids. I had an older version
of this in high school and I used to wear it under my school uniform so I could listen to music from my iPod in class. I felt so cool that I could do that and the normal-hearing kids couldn't! But to be honest I don't use it very often now because the sound quality is not very good and I have to charge and carry around an extra device. I usually reach for some nice consumer wireless earbuds instead.
The difference comes down to power consumption. It takes a lot of bandwidth and therefore a lot of power to transmit high-quality audio wirelessly over a long distance. That's why earbuds like these sound great but only last a few hours. Hearing devices are supposed to last a few days on a charge, so they can't use full-power Bluetooth. Instead, there's a dongle like this that talks to the phone over Bluetooth but uses a lower-quality, shorter-range proprietary protocol to talk to the hearing aids. These are low-volume accessories, so they tend to
be expensive and don't keep up with consumer products in terms of quality or features. Now, there are some newer hearing aids that can connect directly to certain models of smartphone, so users with that phone don't have to carry an extra dongle. The new Bluetooth Low Energy Audio standard, which is part of Bluetooth 5.2 and was developed in collaboration with the hearing aid
industry, should bring that direct connection feature to most new hearing aids, smartphones, and laptops over the next year or two. The standard includes a more efficient codec, so it can transmit sound with higher quality and lower bandwidth, which means it consumes less power and can be built directly into hearing devices. It will be the same standard used in the next generation of wireless earbuds, so I'm optimistic there will be good compatibility.
Another popular streaming accessory, especially since everyone's been stuck at home, is the TV streamer. It bypasses TV speakers to pipe sound directly to the listener's ears. These are useful in households where some people have hearing loss and some don't, since they mean no more fighting over the volume control. Every major hearing aid company makes a proprietary accessory that will stream to their hearing aids. Like the Bluetooth dongles, these accessories vary in quality. Personally, I use a dedicated set of analog wireless headphones. I love being able to walk around the house and do chores while I'm wearing them.
According to the industry, we can expect the new Bluetooth standard to be built into TV and stereo systems as well, so any number of hearing aids and consumer earbuds can connect to them directly. If it's implemented well, that should be really convenient. You could tune your earbuds or hearing aids to one of the TVs at the gym, for example. One downside of the TV streamer, especially the big headphones I use, is that I can't listen to a show and also have a conversation with someone else in the room. I'll say more on that later. This next gadget seems to be much less popular among adults but is widely used in K-12 classroom settings: A remote microphone. These are for one-on-one conversations with another
person in the same room. The microphone is worn by a teacher or conversation partner and it transmits sound to the hearing device. Remote microphones have a few advantages over hearing devices alone. Because the microphone is far away, there's less risk of feedback when the gain is turned up. In noisy environments like restaurants, they provide a much better signal-to-noise ratio because they're closer to the talker. They
also pick up much less reverberation, which can improve intelligibility in large, echoey rooms. Essentially, they bring the talker closer to the listener. They also work well with face masks, so they're a great tool to have on hand during the pandemic. Now, as I mentioned, remote microphones don't seem to be very popular among adults. One reason is that they're expensive proprietary accessories.
Because the existing Bluetooth audio standard has such high power consumption and high latency, every company uses their own custom wireless protocol, and some work better than others. I measured one commercial remote microphone accessory with more than 40 milliseconds of latency which, [echo effect] as you can tell from listening to me now, can be very annoying. [echo effect] That much delay also makes it hard to speak. The new Bluetooth standard is
also supposed to have lower latency, so I hope that it makes its way to remote microphones. If it does, we might see accessories that work with hearing devices from multiple companies. Third-party devices could have lower prices, higher quality, more up-to-date specs, and more innovative new features than first-party products. Remote microphones are also inconvenient to carry around and they require self-identifying as a person with a disability. It's one thing for a parent or audiologist to ask a K-12 teacher to use assistive technology in class, but for adults, not everyone is willing to ask a business partner or a date to put a gadget around their neck. That's even harder during the pandemic because you'd have to wipe it down before handing it to them.
The last category of assistive listening technology I want to talk about is broadcast systems, which are found in public venues like churches and theaters. A transmitter is connected to the venue's sound system, so the signal could be a single talker at a podium, a mix of microphones from a panel discussion or theater performance, or pre-recorded sound - whatever is being played over the loudspeaker system. The signal is received by either a dedicated headset or the listener's hearing device. In the United States, public venues are required to provide headsets for users to check out, and most use FM or IR systems. There's a lot of variability between venues in how well these systems are implemented and how popular they are.
I talked to one local venue that has a lot of older customers and uses them all the time, and another one that has a set but hasn't checked out a headset in over a decade. Personally, I love using the headsets when I see musicals in Chicago because the assistive listening system gets a special sound mix that emphasizes the vocals over the instruments, so I at least have a chance of understanding the lyrics. But for a stand-up comedy show, I usually won't bother. Headset systems have the same problem that users have to self-identify and take extra steps. You have to find out where to get it, fill out a form, leave an ID, figure out how to use it, and remember to bring it back later when everyone's rushing out.
In Europe, it's more common to find induction loops, which are special wires built into a room that transmit sound directly to the telecoils in compatible hearing devices. The advantage of hearing loops is that users don't need to take any extra steps beyond pushing a button on the device they're already wearing. The downside is that induction loops are difficult and expensive to install, especially in older buildings. I know some accessibility advocates have very strong feelings about telecoil systems, but from a technological perspective, they're kind of a pain. The newest trend in assistive listening systems is Wi-Fi. Users download an app or
visit a webpage and stream sound over the network using their own headphones or hearing devices. If you're a venue owner and you buy a new FM or IR headset today to meet regulatory requirements, chances are it will come with Wi-Fi built in too. Wi-Fi systems combine some of the advantages of headsets and induction loops: They're cheap and easy to install, and listeners can use devices they already have. But of course the user has to be comfortable using a smartphone, and if the network doesn't work well, there could be a noticeable delay. A closely related technology is online broadcasting, which became popular among churches and other live event venues during the pandemic. People would gather together in a parking lot while staying in their own cars and stream the audio over their smartphones.
According to the Bluetooth Special Interest Group, the new Bluetooth Low Energy Audio standard is expected to replace induction loops and other broadcast systems over the next decade or two. I think that will depend on how well the standard is implemented, but if it works well it could make assistive listening systems much more accessible, and we might start to see them in venues that aren't legally required to have them. Since consumer devices will use the same protocol, Bluetooth transmitters might come standard in every new sound system, so you can tune into the playlist at the coffee shop or the announcements at the airport just as easily as the broadcast at the theater.
The systems I've mentioned can be categorized as one-to-one communication, like remote microphones, and one-to-many broadcasts, like induction loops and FM headsets. But what if I'm at a dinner or a business meeting talking with multiple people at once, or if I want to listen to the TV while also talking with my partner at home? What about class discussions where students want to hear each other as well as the teacher? I see many-to-one and many-to-many systems as the next big growth area in assistive listening technology. We're starting to see a few products that can do this, either using multiple individual microphones or tabletop microphone arrays. From a technology standpoint, it's not too difficult. For example, I've tried using this pair of wireless microphones with two conversation partners. The receiver is designed to connect to a video camera for shooting two-person interviews, but you can just as easily hook it up to a pair of headphones. These mics sound great,
in part because they're big and heavy and chew through batteries, but multiple-talker wireless systems, both official ones from hearing device companies and these do-it-yourself solutions, mean carrying around a lot of extra equipment, and it's harder to provide a good listening experience when there's more than one talker. For example, these mics can route one talker to the left ear and one talker to the right ear, or they can mix them both together in both ears, neither of which really sounds like I'm talking to two people in the same room. Let's talk about what the next generation of assistive listening technology might look like and what we can learn from the new audio tech developed during the pandemic. New device technologies, wireless standards, and signal processing algorithms should lead to a lot of innovation in assistive listening systems over the next few years. It's useful to think about
what kind of systems we'd build if we no longer were constrained by size, power, or bandwidth. Listening systems have two main design criteria, which I'll call immersion and enhancement. An immersive system sounds real and makes me forget I'm using any technology at all. It should preserve the room acoustics and spatial cues, so if you're standing on my left, it sounds like you're on my left, and if you're facing away from me and mumbling into a wall, it sounds like you're facing away from me and mumbling into a wall. The gaming industry is pouring resources into algorithms that can simulate room acoustics and head-related transfer functions to provide better immersion in virtual reality audio.
Hearing devices get immersion for free because they have microphones at the ears already. The sound picked up by microphones at the ears already has the acoustic effects of the room and at least some of the acoustic effects of the head. Enhancement is about making sound easier to understand, for example by reducing noise and reverberation. Hearing aids have a hard time doing that because their microphones pick up noisy, reverberant sound. Remote microphones can do better because they're closer to the talker. Remote meetings, like the one we're having now, are another great example of enhancement.
You're hearing my voice through a high-quality microphone in a quiet room, carefully processed in software and then delivered to your headphones. When everyone has good hardware and a good connection, I find it much easier to hear in remote meetings than in-person meetings, but there's no sense of immersion: You're hearing me from inside your head. Sometimes there's a trade-off between immersion and enhancement. For example, if you're facing away from me and mumbling into the wall, I probably don't want realism; I want to enhance your voice so I can understand it. But often we can have both. For example, most remote microphones transmit the same sound to both ears, so listeners have no sense of where it's coming from. But if we have a high-quality, low-latency
wireless connection and decent processing power, then we can match the acoustic cues of the low-noise remote signal to the cues at the ears. Here's a demonstration using a hard-wired lapel microphone in our laboratory. I read a passage while walking around some speakers playing speech recordings. The sound in the video is matched to the ears of a dummy head, so if you're listening through headphones, you should be able to track my direction as I move back and forth. First you'll hear the noisy mixture through the ears, then the unprocessed remote microphone signal, and finally the enhanced signal. Listen for the spatial cues and the changes in spectral coloration. [Overlapping speech] ... with its path high above and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look but no one
ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. Throughout the centuries men have explained the rainbow in various ways. Some have accepted it as a miracle without physical explanation. The Greeks used to imagine that it was a sign from the gods to foretell war or heavy rain.
That demo only used one remote microphone, but it would work just as well with several. If we had a microphone on each talker, we could do more sophisticated types of processing than we can with hearing devices today. We could apply different gain, equalization, and compression to each talker, just like mixing artists do in recording studios. Of course, when we start interacting in person again, there's a third criterion to think about: convenience. I personally wouldn't mind carrying around a case full of microphones and clipping
them on everyone I meet, but I think I'm an exception. People are already reluctant to do that with a single remote microphone. Ideally, we wouldn't need to carry any extra devices at all. We could just walk into a room, our devices would enhance the things we want or need to hear, and the whole experience would be so immersive we would hardly notice.
It sounds like a fantasy, but a lot of the technology is already there, thanks in no small part to the pandemic. You see, now that workplaces are starting to reopen, the tech industry has decided that the future of work is hybrid. There will be some employees in the office and some working remotely, and they'll want to have meetings with each other. Now I don't know whether they're right about that, but it means the industry is investing heavily in audio hardware for hybrid meetings. Remote participants need to be able to hear everyone clearly and captioning software needs to know who said what, but in-person participants won't all be wearing their own microphones. The solution is to install microphone arrays, which can capture high quality sound at a distance, even if multiple people are talking at once, and they can track people as they move around the room. Microphone arrays were already common in smart speakers and game systems,
and now they're showing up in cell phones, laptops, and conferencing equipment to improve the quality of video calls. They've even started installing them in our local school district to help with distance learning. These systems are designed to capture sound for remote participants, but there's no technological reason why they couldn't also connect to the hearing devices worn by in-person participants. That way, people with hearing loss could benefit from this expensive new infrastructure to hear better in classrooms and meeting rooms. Especially with the new Bluetooth standard, it
would be an easy feature to add. Tech companies, if you're listening, please make it happen! These new hybrid conferencing devices are powered by microphone arrays. As it happens, microphone arrays were the topic of my dissertation, so get ready to learn all about them.
As the name suggests, a microphone array is a set of microphones that are spread apart from each other in space. Unlike single microphones, arrays can process sound spatially. Propagating sound waves will each reach different microphones at different times depending on their direction. We can process and then combine the signals captured by the different microphones so that sounds from one direction interfere constructively and get louder and sounds from other directions interfere destructively and get quieter. That geometric version of array processing is called beamforming, because we can think of capturing a beam of sound from a certain direction. To do beamforming,
we need to know how far apart the microphones are and where they are relative to the sound sources. There's another way to think of array processing which is often called source separation. In this interpretation, there's a system of equations that describes how sound propagates from each source to each microphone. If we have more microphones than sources, then we have more equations than unknowns, and we can solve for the original sources. If we're in a reverberant room where the sound bounces around, those equations get very complicated, so we need to calibrate the array somehow, and that's been a major problem for signal processing engineers for decades. In general, the performance of a microphone array depends on its size, both the number of microphones and the area they cover. Just like lenses,
larger arrays can create narrower beams, and when we have plenty of microphones spread far apart we can make the array more robust against noise and reverberation. When designing microphone array processing for listening devices, we have to be especially careful if we want them to be immersive. A basic beamformer like the kind used in a smart speaker would distort the spatial cues of everything that isn't in the direction of the beam, so it sounds like being in a tunnel. We can design beamformers that don't have the tunnel effect, but they also don't reduce noise as much. Just like with assistive listening systems, sometimes there's
a trade-off between enhancement and immersion. If we want both, we need to add more microphones. Most high-end hearing aids have two or three microphones per ear, and they're right next to each other, so they can only do a little bit of directional processing without causing perceptible distortion. It can help somewhat if most of the noise is behind the listener, but it's really no use at all in a very crowded environment. In our lab, we've been designing prototypes of larger wearable microphone arrays that have dozens of mics spread across the body. Our
most iconic prototype is the "Sombrearo" which has microphones spread around the brim of a large hat. Our engineering students have built a few functional prototypes over the years. Microphones around the torso are especially helpful because the torso is acoustically dense, and the microphones can be hidden under most types of clothing. We also brought in some design students to imagine more aesthetically pleasing wearable arrays.
With these larger arrays, we can design listening systems that separate process and recombine sounds from multiple sources, doing real-time remixing while preserving spatial cues and room acoustics. But they still have the issue of calibration. We need to learn where the sources are in the room and the acoustic paths that sound takes from each source to each microphone. Even with large wearable arrays, that's a daunting problem. Remote microphones can help. Even if they have poor bandwidth and large delay,
remote microphones still have a good signal-to-noise ratio and low reverberation, so we can use them as pilot signals to calibrate a beamformer. Here is the remote microphone demo from before, but this time, instead of listening to the processed remote microphone signal you'll hear a binaural beamformer from a 14 microphone wearable array that tracks the moving talker. [Overlapping speech] ... with its path high above and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. Throughout the centuries, men have explained the rainbow in various ways. Some have accepted it as a miracle without physical explanation.
The Greeks used to imagine that it was a sign from the gods to foretell war or heavy rain. Now, what if we went even bigger and filled an entire room with microphones? Well, we did that. This is the largest conference room in our building, with glass walls and lots of reverberation. We set up a simulated party with 10 loudspeaker "talkers" all talking at once,
four mannequin "listeners" wearing microphone arrays, and a dozen tabletop array devices designed to look like smart speakers. There were a total of 160 microphones in the room. Now, no portable listening device is going to be able to process 160 channels in real time, so we designed a hybrid system. The distributed array was used to locate the sources, learn the room acoustics, and calibrate the wearable arrays, and then the wearable devices were used for real-time processing. In this demo, you'll be listening through the ears of this mannequin in the corner as she tries to listen to the talker next to her. First
you'll hear the talker alone, then the noisy mixture, and finally the processed signal. This is another binaural beamformer, so listen for the spatial cues. ...seven years as a journalist. We must provide a long-term solution to tackle this attitude. Then, suddenly, they weren't. [Overlapping speech] ...seven years as a journalist. We must provide a long-term solution to tackle this attitude. Then, suddenly, they weren't. So at this point you might be thinking, "This guy is crazy! No one's going to cover their body in microphones, much less an entire room."
Well, that's true. This is mostly an academic exercise to show what could be done with a really extreme system. But if you think about it, we're already surrounding ourselves with microphones. On my body, there are microphones in my hearing aids, my watch, and my phone, and it's only a matter of time before augmented reality glasses finally catch on.
Looking around my living room, I count at least 30 microphones between smart speakers, game systems, computers, and other gadgets. And thanks to remote and hybrid meetings, large network-connected microphone arrays are being installed in classrooms and offices all around the world. What if listening technology could tap into all these microphones that are already all around us? What if when I walked into a meeting my hearing aids picked up sound from everyone else's hearing aids, phones, computers, and from arrays installed in the walls and ceiling? With the right signal processing, my hearing aids could pick and choose what I should hear and process it to have the right spatial cues and room acoustics. I wouldn't have to think about it. And it wouldn't just work in offices. Imagine sitting down to dinner at a restaurant and hearing
only the people at your own table while turning down everyone else. Imagine sitting in a classroom and hearing the shy quiet kid in the back just as clearly as the loud kid right next to you. Microphone array processing can make it possible. Now, this utopian future for listening technology is still a ways off.
The new Bluetooth standard can do a lot, but it can't do this. Gathering sound from every device in a room will require new standards and protocols, and buy-in from the whole tech industry. There are also obvious privacy concerns that would need to be addressed before it could be used in public spaces, and there are many signal processing challenges in making sense of data from so many different kinds of device. Our group is working to address some of those challenges, which I'll tell you about in the final part of this talk.
But first, let's take a short break from microphone arrays to hear about an unexpected connection between hearing aids and COVID-19. I've talked a lot about how COVID-19 has influenced hearing technology, but did you know that a hearing aid signal processing technique can help with COVID-19? Early in the pandemic, there were widespread fears of a ventilator shortage, so a team of engineers here at the University of Illinois came together to design a low-cost emergency ventilator that could be rapidly produced. Our research group helped design the alarm system that alerts clinicians if a patient stops breathing normally. We wanted to find an algorithm that could run on any processor, even the smallest microcontrollers. The level tracking algorithm used for dynamic range compression in hearing aids was perfect. It can follow peaks in a waveform with adjustable time constants, but it requires almost no memory and just a few computations per sample. We adapted the algorithm to track the depth and
duration of breath cycles. It sounds an alarm if breaths are too fast too slow or too shallow. The ventilator design has been licensed by more than 60 organizations and the alarm system is available as an open-source hardware and software design. Now, back to our regularly scheduled microphone array programming In most of the microphone arrays in use today, all the mics are arranged within a single device. They're connected to each other by wires so they can share all their data instantaneously, they all have a common sample clock, and they have known positions relative to each other. Remember, microphone array processing relies on precise timing differences between microphones, so it typically requires perfect synchronization and known geometry. But if we want to do really powerful spatial processing, if we want to handle noisy crowded spaces with dozens of sound sources, we need our array to span the whole room. Typically, that means we need to combine
microphones from multiple devices. These are called ad hoc arrays or distributed arrays, and they're more challenging than conventional single-device arrays. We don't always know where the microphones are relative to each other, so we can't easily translate timing differences to spatial directions. Worse, the devices might not be synchronized, so we can't tell exactly what those timing differences are, and the sample clocks of different devices might drift over time. If the devices are wireless, like most wearables,
they probably have limited bandwidth, relatively high latency, and intermittent dropouts. With ad hoc arrays, therefore, we might not be able to use all the microphones together for beamforming. Fortunately, most devices today have more than one microphone. That means our ad hoc array isn't
composed of individual microphones, but of smaller conventional microphone arrays. The sensors within each device are synchronized with each other and can be used for real-time spatial processing, and the devices can share other information with each other to improve performance, even if they can't perform large-scale beamforming in real time. For example, the devices can work together to decide which sound sources are where and what their frequency spectra are like, or to track who's talking when. Then each device can adjust its local processing using those parameters. We call this cooperative processing. The large conference room demo was a good example: The smart speakers are in fixed locations, so they can locate the talkers and relay that information to the wearable devices, which then do real-time listening enhancement.
The beamformer that tracked the moving talker is another example: The remote microphone signal was delayed by about 100 milliseconds, so it couldn't be used for real-time listening, but it could be used for tracking. Some of the most useful microphones in an ad hoc array are in wearable devices. Wearables provide low-noise reference signals for speech from their wearers and they follow them as they move, but it's hard to combine wearables into an ad hoc array because they move constantly. When we tested our wearable arrays on mannequins, they worked great, but when we tried them on live humans, the high-frequency performance plummeted.
That's because humans are always moving even when they're trying to stand perfectly still. At higher audible frequencies, a microphone on my chest will move by multiple wavelengths relative to a microphone on my ears every time I take a breath. However, if we explicitly account for that motion when we design a beamformer, we can at least partially compensate for it.
This next demo is a beamformer using a wearable microphone array on a moving subject. Now, for the algorithm to work the motion has to be fairly predictable and repetitive, so enjoy the "Mic-Array-Na". [Overlapping speech] We are not aware of any British casualties at this stage. There isn't the seriousness of other businesses. It is about control of our economy. There was a rush of water. The rainbow is a division of white light into many beautiful colors. These take the shape of a long, round arch with its path high above and its two ends apparently beyond the horizon.
Please call Stella. Ask her to bring these things with her from the store: six spoons of fresh snow peas, five thick slabs of blue cheese, and maybe a sack for her brother. One of the advantages of distributed listening systems and microphone arrays over conventional hearing devices is that they can apply different processing to different sound sources. Perhaps I want to amplify one person more than another, or apply different spectral shaping to speech versus music. That independent processing could be especially important for nonlinear
processing like dynamic range compression. Compression is used to keep sound at a comfortable level by amplifying quiet sounds and attenuating loud sounds, but it's known to cause unwanted distortion when there are multiple overlapping sounds. For example, if I'm talking quietly and there is a sudden loud noise, then the compressor will turn down the gain applied to all sounds, and my voice will also get quieter. If we compress different sounds independently before mixing them together, that distortion doesn't happen.
It remains an open question, however, whether that kind of processing would make it easier to hear. In fact, many of the open research questions in this area are better answered by hearing scientists rather than engineers. If we could perfectly separate and recombine every sound in the listener's environment, how should we do it? How many sounds can a person pay attention to at once? Does that depend on age, on hearing ability, on noise level? How do we know what sound the listener wants to hear at any given time? A lot of engineers tend to assume that we want to hear what's in front of us, but sounds from behind are also important since they alert us to things we can't see. When we have to decide between immersion and enhancement, how do we make that trade-off? As signal processing advances unlock new types of listening technology, we'll need to work with hearing scientists and with users themselves to understand how to use it.
I got into this field because I want to make hearing aids work better, but these new listening technologies could help everyone hear better whether or not they have hearing loss. Large microphone arrays and distributed sensor networks could be used for augmented reality, media production, surveillance, and much more. They could let people hear things we couldn't otherwise, giving us superhuman perception. I often like to explain listening technology by analogy to vision.
A conventional hearing aid is like a contact lens. It's intended to restore normal sensory ability, it's designed to be invisible and worn all day, and it's no larger than the sense organ that it sits on. That means it has access to the same information our senses do. Just as a contact lens will never let us see a microbe or a distant planet, a hearing aid will never let us have a quiet conversation in a crowded convention center or hear a mouse from across a busy street. Instead of a contact lens, I want to build the hearing equivalent of a telescope or a microscope: a large, situational device that we can't wear all day, but that lets us sense things we normally couldn't.
The last year has been challenging for everyone, but especially for the hearing loss community. We suddenly had to learn how to manage without seeing people's lips and how to hold video meetings with spotty support for captions. We had to deal with shortages of not just toilet paper but also karaoke systems. There were some bright spots, of course. The world got a lot quieter, at least for a little while, and we got to watch normal-hearing people struggle with the sudden loss of high frequency speech sounds and learn how to speak up.
Perhaps most importantly, the sudden shift to remote work led to a new focus on audio capture and processing technology. Suddenly everyone was talking to each other through microphones and speakers just like we always have. Thanks to the pandemic, there are a lot more microphones in the world than there were a year ago, and most of them are networked. That progress coincides with technological shifts like the new Bluetooth Low Energy standard and market trends like direct-to-consumer hearing devices that were already poised to shake up the landscape for listening technology. If the tech industry makes hearing accessibility a priority, then as we start to gather in person again, we can leverage all those new microphones and new wireless technologies to power the next generation of listening devices, both for people with hearing loss and for everyone else who wants to hear things they couldn't hear before. That way, the technology we've developed to keep us apart can help bring us back together.
I'd like to thank my team at the University of Illinois as well as the agencies and companies that have supported our research. To learn more about our work, please visit the Illinois Augmented Listening Laboratory website. If you'd like to get in touch, you can find me on Linkedin, YouTube, and Twitter @ryanmcorey. I'm always eager to talk with audiologists and listening technology users, so I would love to hear from you.
2021-08-17