Soundscape.social merges video games and concerts for an interactive virtual music experience

In the era of social distancing, in-person concerts were one of the first large-scale events to be delayed or cancelled. Many musicians took to livestreams or pre-recorded concerts to continue interactions with their fans, but University of Pittsburgh PhD students Pat Healy and Hannah Standiford have created a new way to listen with music virtually that reinserts the interactivity and spaciality of in-person concerts into the experience.

Soundscape.social is a cross between a concert space and a video game that allows anonymous attendees act as players who can interact with the concert space and with each other in real-time while enjoying classical and jazz music from musicians affiliated with Richmond, Va.-based Classical Revolution RVA. With two concerts under their belt, Healy and Standiford will present their third and final concert of the planned series Thu., May 27, at 7:30 p.m.

“While we can't all be together, one thing that was kind of missing from pretty much all of these was this participatory component on the part of an audience,” Healy says. “What you're missing is the fact that music exists in space.”

Healy and Standiford met through graduate student union organizing. While Healy’s background is in information science and game studies and Standiford’s is in ethnomusicology, both share an interest in experimental interactions with media. The project draws from anarchist philosopher and music composer John Cage’s Musicircus, a live performance that invites any number of music groups to perform simultaneously to create a unique experience for each attendee as they wander among the musicians. Healy also looked to musician and educator Christopher Small’s theory of “musicking,” in which music is not a noun but a verb and participatory activity.
The concert series, hosted on itch.io, received funding through the Immersive Fellowship Program, a program at Pitt encouraging remote engagement with the arts during COVID. The funding’s stipulations steered the project towards focusing on classical and jazz music, and Standiford reached out to members of Classical Revolution RVA with whom she was already acquainted.

The nonprofit connected her with musicians, and Healy then worked with the musicians to determine the auditory and visual aesthetic for the piece they would perform. Musicians would record their piece with separate recordings for each instrument, which Healy placed in the virtual concert space to simulate the uniqueness of hearing a song in person depending on where you’re situated relative to the instrumentalist.

“We try to shape a space around that music with some level of interactivity that sort of plays off motifs in the music,” says Healy, who created a skyscape for the second concert area to complement a strings and flute duo that the musicians wanted to give the feeling of being in the clouds.

Attendees are sorted into “rooms” of 10 people who can explore the concert space together, and the program can handle 10 rooms at a time. Upon entering, attendees are assigned one of about 35 open source emojis — possibly the poop emoji, or the pride or trans flag — and a name from a roster of almost 50 music terms such as “adagio,” “motif,” “glissando,” and “scherzo.”

“I would love to just show up in the game and be named ‘glockenspiel,’” Healy says. “That just seems really fun to me.”

Healy and Standiford hope that the anonymity of the experimental concert space also allows them to observe and test systemically engendering kinder anonymous interactions. Healy drew from the game Kind Words, a game in which people can anonymously write and request kind letters, and rejects the belief that anonymity inherently breeds online trolls or cruel interactions.

“There's a lot of things designed in the game that's specifically to make it encourage wholesome enough anonymous communication … which I feel very strongly as an anarchist is a thing that's possible, and has a lot more to do with systemic constraints than it does with like the evilness of individual people,” says Healy.

Unlike other video games such as Fortnite and Minecraft that have hosted concerts, this game was designed with music concerts specifically in mind. Healy built most of the system for the concerts over a single weekend in a game jam-like crunch, using a mixture of modeling and purchased assets. The first two required attendees to download the game beforehand, but to make the final concert more accessible, Healy is making it available in browser.
Each concert space has four environments with four different musical performances on loop. Attendees can leave messages for one another that show up in the corner of the screen in real-time while also getting tagged to the spot where they wrote the message. Later attendees can view the comments of previous attendees — such as the series of notes near a plant in the second concert lobby that say variations of “plant gang” — out of context and chronological order as part of a new, unique concert interaction.

Depending on where you stand in the concert space, various instruments will sound louder or softer as though you were standing inside an in-person concert hall. The spaces also allow for imaginative and fun concert experiences — wandering through a night market, zooming down a street on an anime-style motorcycle, meandering through a campsite — that are difficult to create in person.

The third concert is inspired by filmmaker David Lynch. While Healy is concerned about what Lynch might think of it, they also note that getting sued by Lynch “would be an honor, to be honest.” The third concert will also feature a hidden fifth performance space for more adventurous attendees. Funding for the project only covers these three concerts, but Healy hopes to continue this series in the future.

“I think to a point, a lot of these virtual experiences are here to stay after COVID,” Healy says. “I'm sure there's going to be a period where people are gonna be so sick of them and not want anything to do with them … but I think we've discovered a lot of interesting things that you can do from home now.”