We are, as an interdisciplinary team comprising members from Performing Arts, Computer Science, Kinesiology, and Inclusive Design, developing methods for improving the representation and interpretation of human movement—in all its diversity—to enable a richer, more embodied, and more equitable presence for bodies of all sorts in engagements with machine learning systems.
THE CHALLENGE: How we are situated in the world within our individual bodies defines much of our experience and agency as humans. We navigate the world with purposeful movements and expressive gestures involving joints and muscles working in complex synergies. As we become increasingly surrounded by machine learning (ML) systems, it is essential that these systems be sensitive and responsive to the nuances of our actions. How our movements are represented within ML systems will significantly determine our experience of being in the world: how our embodied presence is accommodated and valued in our increasingly ML-mediated lives.
Many recent ML advances lean heavily on enormous pools of digitized text, image and sound. When vast amounts of data are available, a sufficiently large model given enough training develops surprisingly rich representations of salient features present in the dataset (if they prove useful in achieving its learning objective). Human motion is commonly represented with ‘kinematic’ data: sequences of geometric poses, based on joint angles and positions of limbs, absent any explicit markers of dynamics and physical embodiment. Given enough kinematic data, an ML model should learn some approximation of the interplay of muscles and joints, and of relevant laws of physics (gravity, momentum, balance). But such data is not available or accessible in such quantities. Data representing movements most relevant to our communities of performers, people of diverse racial background, and people with disabilities or injuries is much scarcer. (The data that exist tend to overrepresent white males) We must find other ways to guide ML systems to take physical rather than just geometric factors into account. Kinesiologists often gather force, torque, and muscle activation sensor readings, but usually only for specific muscles and joints. No significant dataset including such data for the whole body exists.
There are long standing debates in the humanities about whether it is possible to capture or notate the culturally significant qualities in movement at all, or whether such ‘capture’ inevitably flattens the di- mensions of movement according to the priorities of a dominant culture. Historians of technology have documented the historic drive to reduce human experience to quantifiable parameters [39,40,41].
ML research deals in the currency of quantitative evaluation: accuracies, log likelihoods, FID scores, and other defined metrics. There can be a hard-to-articulate disconnect between quantitative scores and related qualitative observations. As ML is involved in more and more human activities there is an urgent need and an extraordinary opportunity to engage experts in subjective evaluation to help shape new approaches ML practitioners can apply in these domains, ways of integrating crucial information that might appear too “soft” or “subjective” for a typical ML researcher to prioritize.
Our project will bring a critical, multidisciplinary lens to the question of what is at stake in the quantifi- cation of human movement. This is urgently needed because ML systems are increasingly operating in the human world, in real-time, providing virtually instantaneous feedback. The ways we are fed back to ourselves through our technological environment indelibly mark our sense and understanding of ourselves, ultimately profoundly shaping our culture.
INTERDISCIPLINARITY: Human movement can be considered, analysed, perceived, and experienced in myriad ways: as a source of pain or pleasure, as expressive of intention, as cultural artifact, as an in- dicator of physical health or injury, as an expression of unique abilities and disabilities, and as a reflection of intricate reciprocity between human and the environment. While these aspects can be treated in isolation, they are profoundly entangled. The high stakes of determining how we register as physical be- ings in ML systems require that we work together across disciplines to acknowledge that entanglement.
In the performing arts, human movement is considered in terms of expressiveness, aesthetics, poli- tics, and intent. The history of attempts to inscribe human motion in dance notation provides a unique historical perspective on motion capture [1,2,3,4].
Performing artists are attuned to the expressive and aesthetic implications of their movements and to connect these to their embodied experience. They can provide invaluable research feedback providing a necessary complement to the inherently quantitative approach of ML. On the other hand, the internal representations that ML systems learn are often rich in nuance and ambiguity, seeming to blur the dis- tinction between quantitative and qualitative (though often hard to interpret).
Kinesiology adopts a largely quantitative perspective, but with explicit grounding in the body, its cog- nitive and neuromuscular mechanisms, and an attention to physical forces bearing on the body.
Focused on performance and learning, injury and its reduction, physiotherapy, and rehabilitation, re- search concerns in kinesiology complement and occasionally overlap those in the performing arts.
ML systems are intended to map the entire field of possibilities of the data they have trained on, to be inclusive as it were, but risk performing poorly in response to edge cases. Inclusive Design explicitly ad- dresses the challenges faced by people who appear as edge cases to norm-focussed systems and de- signs. It rejects the deficit framing of non-normative movement and demands that non-normative bodies and movements remain legible within ML systems. To achieve this goal, people with disabilities need to be engaged as active partners, not just subjects.
We will draw these contrasting yet complementary perspectives into conversation, seeking a broad, interdisciplinary understanding of what is at stake as human movement increasingly becomes a subject of ML, as a foundation for our explorations of human motion in ML systems.
RESEARCH OBJECTIVES: Our primary research objective is inherently interdisciplinary, and as such it can be interpreted from multiple perspectives: our focus is to find ways to improve the representation of dynamic human movement in ML systems that embrace and address the full human range of movements and bodies, especially those at or beyond the edge of the normative. We seek to design ML systems that can be trained to engage with and respond to physical movement in all its expressive dynamic dimensions (distinct from functional gesture recognition); to the best of our knowledge, this is a truly novel perspective.
PRIOR WORK: There is, however, extensive and applicable related research. Computer animation researchers have developed methods for improving the quality of kinematic motion data by learning to mimic captured motion within a simulated physics environment, inferring muscle activations and other physical forces required to produce a physically plausible reconstruction of the movement, filtering out noise that is inconsistent with physics. We will explore using such techniques to infer how physics bears on the body, both to improve ML training results on limited data and for use in real- time human-computer ML interfaces. Antoine and others have applied wavelet analysis to cal- culate the spectral components of movement data, producing a dynamic movement signature, poten- tially allowing us to situate each frame of kinematic data in gestural trends across multiple time scales.
Large language models can predict motion by converting poses into distinct tokens and then gener- ating plausible sequences. Oore’s team has explored the use of such methods by pre-training a language model on a large general motion dataset and fine-tuning on a small clinical dataset.
We are inspired by ground-breaking work in Inclusive Design examining challenges ML poses to people with disabilities, spearheaded by collaborator Jutta Treviranus. ML has been thoughtfully applied to customize interfaces for people with disabilities. There is also rare, and inspiring work on modelling graphic representations of non-normative human bodies.
Perception of biological motion has been explored extensively in kinesiology, and most relevantly for us, collaborator Wang and others have examined the dynamic features of movement that serve as cues of intention and emotion, and the role of physical forces (e.g. gravity) in this context .
TEAM:
David Rokeby brings a history of working with computational approaches to human movement in the visual and performing arts, is experienced in visualizing and sonifying motion data, and has collaborated on multiple Inclusive Design projects over the past 30 years.
Sageev Oore brings to the project an interdisciplinary perspective with grounding in both founda- tional machine learning and its applications for human time-series data (from movement to speech), and an awareness of the need for humanities-based contributions to this area.
Douglas Eacho is a performance historian with a special interest in the confluence of computation and dance. He provides the project expertise in philosophical discussions of dance and media which can emphasize the stakes of the team’s findings.
As a dancer, Director of the Institute of Dance Studies, and as a scholar of black dance in Canada, Seika Boye brings to the project a broad knowledge of dance both as an embodied practice and as a subject of scholarly research.
Research into the history of dance notation, and critiques of notation systems will provide us with a performing arts perspective on capturing and representing choreography. Related research examines the lack of archives or museums of human movement, and the resistance to the possibility of notating dance, or highlights the importance of embodied cultural memory. Norman and others directly address issues around technological motion capture in the arts.
Tim Welsh, Tim Burkhart, and Xiaoye Michael Wang bring expertise in the dynamics of human movement, its health and performance implications, and the cognitive and neural mechanisms behind it. They also contribute technical background in motion capture, kinematic analysis, and signal processing.
Jutta Treviranus brings deep expertise in Inclusive Design, strong connections within the disability community, and a research focus on the challenges that ML presents to people with disabilities.
Spirit Synott brings to this project her invaluable lived experience as a dancer who uses a manual wheelchair for mobility, as an activist and as expert in inclusive design.
Balazs Kegl brings extensive expertise in ML, including physics-based pipelines and micro-data.
Leave a Reply