BMO LAB

Creative Lab for the Arts, Performance, Emerging Technologies and AI

Highlights

Voice Scroll: Real-time Voice to Scrolling Image for Performance

Real-time Generation of Panoramic scenes from Voice using a custom Stable Diffusion pipeline (2023)

We have extended our exploration of text to scrolling image to work with the latest version of Stable Diffusion and adapting it to our particular performance-based projects. Here is a sneak peak of a very casual interaction with the most recent version. This is the direct output responding in real-time to the spoken words.. no editing and no cherry-picking.

In July, we worked with the most recent version of this system with American poet Nick Flynn. You can see some of the preliminary results here.

Lecture Series: Performance Capture – on the margins of the computable (2022-3)

When: Oct. 25, Feb. 15, March 2 (5–7pm) 

We held a lecture series at the BMO Lab this academic year (2022-3) entitled “Performance Capture – on the margins of the computable.” These events, which ran on Tuesdays or Thursdays from 5-7pm, introduces major thinkers on themes of performance and digital culture to our community through lectures and extended, in-depth conversation (with refreshments!). This year, our speakers will focus on the tenuous compatibility and disjuncture between digital representation and the live body. 

Relational Dramaturgies: Co-Producing Spectators, Immersive Spaces and the change of the locus of dramaturgy

Imanuel Schipper – Dramaturg for Rimini Protokoll

October 25, 2022, 5-7 pm


People are walking the street, doing strange things, a theatre audience is discussing greenhouse gas reductions of countries that they do not know exist – these are just two examples of what modern theatre goers are doing. From the “Discovery of the Spectator” (Fischer-Lichte, 1991) to the “Emancipated Spectator” (Rancière, 2009) a lot has changed in the possibilities how theatre is addressing its audience. Productions in urban space and the progression of digital cultures not only in the daily life but also in all fields of the performing arts led to new paradigms in the way shows are experienced and analyzed. The use of space and dramaturgy and the question of how the spectators are included in the performativity of the theatre event in some contemporary theatre productions changed not only the way theatre is produced but obviously has had a great impact on the experience itself.

Within this change there exists also a major shift of the concept of dramaturgy: From an architecture of a textual structure to an enhancement of the work in the field of theatre to mode of being a spectator. Does the audience then lose its critical distance to the piece of work it is looking at? How could “providing an experience” not only be an unpolitical event but produce new perspectives?

This lecture aims to look at that change of concept and its change of loci as a symptom for many aesthetic phenomena and a reordering of the “aesthetic regimes” (Rancière). With the help of some examples of the most recent productions of Rimini Protokoll it will rethink the concept of dramaturgy and reconstruct a different way of how it is produced. With a special focus on the immersivity of these productions it will discuss the pro and cons of formats that asks for co-producing spectators.

Is My Robot Happy? – A History of Movement and Emotion in People and Machines

Whitney Laemmli, Carnegie Mellon University

February 15, 2023

Over the past few decades, an eclectic mix of artists, roboticists, and computer scientists have utilized a notation system called Laban Movement Analysis (LMA) to simulate human movement in their creations. LMA not only categorizes and records the actions of body parts, however, it also links specific kinds of movement to particular emotional states, relying on theories developed by the system’s originator, Rudolf Laban, in the 1920s.

This talk will describe LMA’s origins and trace its history, from its birth in the scientific and artistic ferment of Weimar Germany through its use in the British factories in World War II, to its life in the corporate boardrooms of the mid-century United States and its ultimate appearance in 21st century laboratories. Paying attention to this often-ignored history will shed light on the system’s longstanding appeal as well as the possibilities and dangers that attend its use.

Black Movement in Digital Spaces

LaJuné McMillian

March 2, 2023

In the past few years, access to motion capture data, 3D base models, and software to “make an animation of yourself” has skyrocketed. From MakeHuman to Mixamo to CMU’s motion capture database, the ability to make and finish polished projects has become easier for many. While these resources are extremely helpful to create a range of projects, they lack tools to create diverse characters and movements unexplored by systems that center assumptions of neutrality.  

The Black Movement Library (BML) started as an online database of Black motion capture data and Black character base models. However, this approach failed to address the exploitation, erasure, and dilution of Black movement and Black culture historically through appropriation, the evolution of Black face, and the commodification of our existence. BML grew into a space of convening and community building, through workshops (both movement and technology based), performances, XR experiences, conversations,  and research on how and why we move. 

BML asks how we can better hold each other both online and off, and curate spaces of care, witnessing, archiving, learning, accountability, and being. BMLasks what new (or old) ways of protection we can develop for ourselves and our information outside of copyright law, which does not have our best interest at heart, “individualizing” networks and communities of work (across generations of the Diaspora), thus erasing the true origin of the work – us.

The Resistible Rise of Arturo Ui – Feb 25-26, 2022

In partnership with Canadian Stage, the BMO Lab presented a workshop production of The Resistible Rise of Arturo Ui at Toronto’s Berkeley Street Theatre on February 25th and 26th of 2022.

Two women conquer a city and – maybe – the world!

The Resistible Rise of Arturo Ui by Bertolt Brecht was written in Finland in 1941 over a period of three weeks while the author and his family hoped and waited for their US-visas to escape the fast-approaching armies of Nazi-Germany. Intended for American audiences, it did not get a theatrical release there and was first produced in 1958 in Germany.

A comedy of horrors, a parable, a thriller full of surprising turns, a love story, a grandiose play on words. We meet bankers and gangsters, crooks big and small, greed and crime in a tight embrace. We observe countless murders without any consequences for the killers. We see how economic interests translate into criminal actions and how fake news transforms into political dogma. Remind you of anything? Please notice the “resistible” in the title!

Rehearsal photos courtesy of Dahlia Katz

Cast and Crew

Written by: Bertolt Brecht // Directed by: Johanna Schall

Cast: Ryan Cunningham, Teodora Djuric, Darcy Gerhart, Valerio Greganti, Sébastien Heins, James Hyett, Adrian Pavone, Khadijah Roberts-Abdullah, Sina Sasanifard, and Carly Street

Lighting Designer: Aidan Ware // Sound Designer and Composer: Eliot Britton // Production Designer: Snezana Pesic // Stage Manager: Seren Lannon

Technical Direction / Director of the BMO Lab: David Rokeby // Artist-in-Residence at the BMO Lab: Bronwen Sharp

Full cast and crew bios

An interview with acclaimed director Johanna Schall

A history of Bertolt Brecht and Arturo Ui

Technology in Arturo Ui

Body Pose Recognition for Sound, Lighting and Video Cues

Technology in Arturo Ui:

Real-time Motion Capture Control of Avatars

Arturo Ui – Real-time Control of Avatars in Performance

For our production of The Resistible Rise of Arturo Ui, we explored the use of live motion capture for some of the scenes. At various times, two of the performers wore rigs of 17 sensors under their costumes, with each sensor recording the rotation of one of the major joints in their body. One of the ways we used motion capture was to enable a performer to control computer generated avatars live on stage.


For the “Whitewash Song” in the production, we created a fairly realistic 3D avatar of the performer, Sébastien Heins, using a smart phone for the 3d scanning and some free, open source 3d software. We connected the output from the motion capture suit that Sébastien was wearing under his costume to the ‘skeleton’ of the avatar, so that it mirrored all of his movements. Cues were established in the qLab cue file for the performance to make the avatars appear, to drain them of colour, and to create ever increasing multiples, synchronized to the playback of Eliot Britton’s wonderful music track.

The Lab is particularly grateful to Sébastien for his patience and enthusiasm, and for the great performance in the show!

Arturo Ui – Body Pose Recognition for Sound, Lighting and Video Cues

For our production of The Resistible Rise of Arturo Ui, we explored the use of live motion capture for some of the scenes. At various times, two of the performers wore rigs of 17 sensors under their costumes, with each sensor recording the rotation of one of the major joints in their body. One of the ways we used motion capture was to enable a performer to control all the cues in a scene.

One of the things we are exploring in the lab is performer ‘agency’. We consider this notion of agency in two different ways: “What is gained if we use technology to give a performer more freedom to pace a performance as suits the moment?” and “How might performer-enacted cueing change the practical and economic challenges for small touring shows?”


For the prologue of the show, Sébastien Heins controlled every single cue through his body poses and gestures. In order to achieve this, we started by considering the information contained in any given pose of the human body. The motion capture sensors give us a fairly rich reading of the sequence of poses of a moving body. Modern AI systems are very good at pattern matching. We decided to pair the two so that the AI would learn to recognize key cueing gestures in a reliable and robust manner that accommodates the variations inevitable in live performance.

AI systems learn by being exposed to a large set of labeled examples. They start out at the beginning of the training process responding randomly, and then slowly become better and better through the training process until they are able to respond to new examples correctly even when they vary somewhat from those it was trained on.

In this particular case, we recorded motion capture data as Sébastien rehearsed, on several different days. He was encouraged to vary the gestures a bit each time so that the AI would learn a more general version of the pose. This data was then labeled. Each gesture that we wanted to trigger a cue was assigned a number and each frame of the data was labeled to indicate the cue the pose should trigger or 0 if the pose did not represent a cue pose.

A relatively simple AI was then trained on this labeled data. It successfully learned to recognize the desired gestures and to ignore all sorts of complex movement that was not associated to any trigger.