CanStage_BMO

Sebastien: Voxel-based cueing of sound, video and lights

Sebastien came in to work with David to try to convert a scene from a solo play to interactive triggering. We positioned triggers for sound cues, a video cue and a lighting cue into the space to see if such a system might be usable for actual performances.

The session brought up some interesting limitations in the current software that David has produced. The software is an extension of work David did for a very specific project a few years ago and thus was not designed to address the kinds of needs Sebastien has for his performance. Based on this experiment, David has been able to adjust and reimplement parts of the software to make it more appropriate for Sebastien’s needs. This process of iterative design, in consultation with people with real-world needs is a very important part of the successful development of good usable tools.

One limitation we found is that the sound triggering in David’s system was not designed for a situation where a sound clip was triggered by activity in a zone, but then was to play through to the end. Sebastien had to keep the trigger engaged to keep the clip playing.

Despite the limitations, we were able to get a rough sketch of the scene working.

Technically, one computer was tracking the movement (as seen in the video). This computer was talking to another computer that was controlling sound cues, lighting cues and video playback. We used Open Sound Control (OSC) as the communications protocol. At the lab we are trying to provide OSC interfaces for all our tools so that they can all be set up to talk to each other.

A few days ago, Sebastien asked whether it would be possible to trigger voices instead of sounds through the system that places sound possibilities in space, so we decided to do an experiment to see how that might feel and what creative possibilities that might open up.

First we sat down and came up with a range of words and phrases that were somewhat ambiguous. and could be presented in different orders. Then Sebastien and Maev recorded these phrases, doing multiple versions of each with different expression.

Then, each utterance was converted into a sound file and loaded into the sample player.

We then created a space of triggers in space using voxel-tracking software and the Azure Kinect. The Azure Kinect produces an image that tells us how far away each part of the image is. This allows us to construct a 3 dimensional sense of whatever is in the frame. In this example, the triggers are arrayed within a 2 meter x 2 meter x 2 meter cube of space in the middle of the performance space. The most occupied of the voxels trigger sounds.

The sounds cannot be retriggered for 3 seconds to prevent complete overwhelm, but this also sets up a kind of varying loop of vocal fragments.

Here Sebastien and Maev explore this space and turn it into a sort of ‘breakdown of a marriage’ ballet. All of the voices you here are triggered, rather than spoken by the performers.

Kudos to the performers who did an amazing job on what was a very VERY rough technical sketch. Based on this experience, we can now think more about what kinds of words and phrases work best, how best to locate them in space, trigger them, and then how to perform within and around the space.

To Be Continued…

Ryan: The Challenge of Indigenous Languages for AI

As the others were unavailable, Ryan and David got together to explore a couple of issues related to Ryan’s interests. First Ryan brought up the fact that it would be challenging to replicate the kind of AI generated script experiments that we were exploring the other day in the context of Shakespeare in the space of indigenous cultural production.

The system we were using (GPT-2 from OpenAI, fine-tuned on the plays of Shakespeare) was initially trained on a very large body of English language writing. The sort of ‘learning’ that this system performs generally requires very large bodies of somewhat consistent input data in order to learn.

Ryan and David discussed the challenges this would pose to performing a similar exploration with indigenous plays.

(related background info: https://www.thecanadianencyclopedia.ca/en/article/aboriginal-people-languages)

Ryan: Automatic Spatially-based lighting cues

Then we embarked on an initial exploration of interactive lighting, using the Azure Kinect Depth sensor to allow us to place lighting cues in space. This system allows us to define locations in 3 dimensional space to be triggers or continuous controllers that control lights through standard DMX. (We had not yet refined the trigger positions in this first test so the first lighting cue area reaches a bit too far to the front, which is why the light stays on after he leaves the area lit by the light)

First we sat down to discuss the experience of the cold read of the AI generated script from last week: what worked, what didn’t, and how we might adjust the software to improve the experience. Sebastien, Maev and Rick discussed the potential to use the tool as a tool got training / rehearsal / skill development, as they all felt that the experience was both pleasurable, very challenging and productive.

Then we went further with the GPT-2 generated scripts using a modified version of the program that allow us to mix models trained on different bodies of text. The example we explored mixed one trained on all of Shakespeare’s plays with one trained on a very large corpus of popular music lyrics (spanning Broadway shows, rap, classic pop, folk songs, etc.). We started with a mix of all Shakespeare and no pop lyrics and then progressively added more of the pop lyrics model into the mix. At first, this did not seem to work, as it kept producing Shakespeare-like material with character names before speeches and stage directions. Eventually we realized that, since the model chooses the next work based on the last 512 work ‘tokens’, the history of previous material was making the mix less fluid. Playing around with some parameters, we were able to do a cold-read on a model that was shifting back and forth. Hilarity ensued!

(video excerpts coming)

After a break we talked about what it might mean to use the spatial triggering to trigger recorded spoken words and phrases instead of sounds… to fill the performance space with a field of possible text fragments and to explore it through movements or gestures. We plan to do some experiments in this direction over the next couple of weeks.

Today we went deeper with the AI generated Shakespeare. We discussed many approaches to using this material, and then decided to put the talk aside and jump in. We set up the lab so that the output of the AI was projected on the wall, so that the performers could see the text as it was being produced, and read it immediately… with the performers adopting characters on the fly as they turned up in the script. As the text is formulated anew on the spot, the performers had no idea what was coming, and often did not know where the speech was going as they were speaking it.

This ended up being much more lovely and joyful than we had expected. The text being generated is coherent in bursts but largely nonsensical, and it was fascinating from the outside to watch these wonderful performers ride the language, letting their experience carry them through. As Ryan pointed out, the AI has a dataset, which is all the words of all the plays of Shakespeare. And the performers each have their own dataset built out of their experience on stage and the roles they have played. So there are two datasets engaging here. The AI’s dataset is strictly limited to words. The performers’ datasets are much broader, incorporating their theatre experience, in language but also the embodied experience of movement on stage, but beyond that, of course, the entirety of the experiences of their lives. Seeing the often clumsy output of this limited system and dataset filtered through and interpreted by these living actors was thrilling, and hilarious!

After trying this twice (once with Ryan, Sebastien and Maev, and once after Rick Miller joined us), we talked about how to make the most of what was exciting about this adventure. During the performance, I (David) was manually advancing the generation of the script after each speech so that it did not run ahead of the performers. We talked about using the spatial triggers we had played with yesterday to allow the performer who was speaking to initiate the generation of the next sentence (i.e. by raising their hand into a sensitive zone covering the entire stage.)

Then we finished up but talking about using text in other ways, such as embedding spoken text in the space, by positioning triggers for speeches throughout the performance space (rather than triggering sounds as we had done the day before). We talked about how to make this more usable and intuitive, fantasizing about a system that would allow us to point to a location in space and speak a line that would then become embedded in the space, ready to be triggered should a performer touch that specific location in space. We talked about how writing a script for such a space is tricky, as each speech must be written to allow for many paths through the text. We talked about how it would be possible to swap ‘maps’ on the fly so that one could easily have a scene change where all the locations of the triggers and the phrases they trigger could change.