Computation, Improvisation, Narration

The stories we tell about, with, or for technologies matter. Can we demystify misnomers such as artificial intelligence through storytelling, role-playing, and improvisation? In this episode, we talk to dmstfctn (Francesco Tacchini and Oliver Smith) and Lawrence Lek about how they build narratives around complex computational systems.

Resources:

The Tricks of the Trade by Dario Fo
Dadda by Brood Ma
Role-play with large language models by M. Shanahan, K. McDonell & L. Reynolds 

Host & Producer: Ana-Maria Carabelea
Editing: Ana-Maria Carabelea
Music: Karl Julian Schmidinger

The Digital Deal Podcast is part of European Digital Deal, a project co-funded by Creative Europe and the Austrian Federal Ministry for Arts, Culture, the Civil Service and Sport. Views and opinions expressed in this podcast are those of the host and guests only and do not necessarily reflect those of the European Union or the European Education and Culture Executive Agency (EACEA). Neither the European Union nor the European Education and Culture Executive Agency (EACEA) can be held responsible for them.

Transcript

Ana-Maria Carabelea: Welcome to The Digital Deal Podcast, the series where we talk about how new technologies reshape our democracies and how artists and critical thinkers can help us make sense of these changes. My name is Ana Carabelea, and today I'm joined by dmstfctn and Lawrence Lek.

dmstfctn is a London-based artist duo formed by Francesco Tacchini and Oliver Smith, working with audiovisual performance, games and video installation. Their work investigates complex systems by directly involving audiences - inviting them into the 'demystification' of systems by replicating and replaying them and into their 're-mystification' by building worlds, characters, and myths atop them. Since 2018, dmstfctn have performed and exhibited internationally in venues such as Serpentine, Design Museum, Berghain, HKW, and at festivals like Unsound, Transmediale, and Impakt. They were the recipient of the 2017 transmediale Flusser artistic residency and are currently wrapping up their residency with Sineglossa as part of The European Digital Deal Project.

Lawrence Lek is an artist, filmmaker, and musician who unifies diverse practices such as architecture, gaming, video, music, and fiction — into a continuously expanding cinematic universe. Over the last decade, Lawrence has incorporated vernacular media of his generation, such as video games and computer-generated animation, into site-specific installations and digital environments, which he describes as "three-dimensional collages of found objects and situations." Often featuring interlocking narratives and the recurring figure of the wanderer, his work explores the myth of technological progress in an age of social change.

Welcome to everyone. Thank you so much for joining me. It's very exciting to have you all here.

Francesco Tacchini: Thanks a lot.

Oliver Smith: Yeah, pleased to be here.

Lawrence Lek: Thanks.

Ana-Maria Carabelea: I'm particularly excited to have you all in this constellation, mainly because of a shared aspect of your work, which is this nuanced take on complex technological systems that sits in between a tendency to view these systems as purely technical and the tendency to anthropomorphize them. I think that, because of this in-betweenness, your work marks this important moment in the continuous revision of what is defined as human intelligence and ultimately, human. Perhaps a good way to start is by asking dmstfctn, Oliver and Francesco to tell us about the project you're currently working on. Walk us through what it's about and what you think it can do for the ways in which we relate to artificial intelligence.

Francesco Tacchini: Yeah. So the work we're currently developing is called The Models, and it's effectively an endless AI-generated improvised theater installation. So it's a simulation-based installation, it has an interactive component, and it really is about the sort of quirks and wonders, and slightly lesser understood characteristics of generative AI, specifically large language models. It's an installation that builds upon a theatrical tradition from Italy called Commedia dell'Arte. Commedia dell'Arte is a fairly known popular form of theater with different masks that emerge from different cities in Italy. Each one of these masks represents a strong archetype. Our installation uses this archetype to let characteristics of generative AI emerge. These characteristics are the tendency to lie or make up things, a tendency to be overly servile, and a tendency to sometimes be antagonistic or turn against the user.

Oliver Smith: Yeah, I think it tries to allow people to see these kinds of interactions between these models in multiple ways over multiple iterations. The idea of it being kind of endless. It takes the form of short sketches. So you might see a play between two characters that lasts five minutes and you'll see another one and another one, another one. But each time the audience is able to select, in a directorial way, the masks that take part in that play - the two archetypes that we'll explore, then a prop that acts as a kind of centerpiece of their action, and then a setting as well. All these come together to provide the basis for the improvisation between the characters and basically to raise these tendencies to the forefront and allow people to look through them.

Ana-Maria Carabelea: Cool. Thank you so much. I'm going to pick up on one thing, that I think slides us into the conversation with Lawrence as well. A common thing between your works is the role of the human in all of this, which always either has a secondary role or might not even be present, like in Lawrence's work at times. I find it interesting because so much of what worries us about AI systems has to do with the impact it will have on us humans. So I want to explore a bit why this setup. Why do you think it's important to have the human sort of just as a witness, as a viewer, or perhaps not even present?

Lawrence Lek: I mean, I think for me because I was always thinking about my work, or at least the perspectives of it, not in terms of human and non-human, but more in terms of first person and third person. Part of that is because I grew up playing video games, and the shift from map-like perspectives to third-person and first-person ones is also part of a technological shift. That also reminded me of my background in architecture, where the invention of perspective in Renaissance Italy was a big invention, not just in terms of technical rendering, but also in terms of positioning the viewer or the human at the center of the world. In those views of the perspective, the human is not actually a human, not a fully realized individual, they're just, you know, a single eye. Not even two eyes, they're just the viewer, the spectator. I think this idea of the viewer, the spectator, abstracts the human biological machine into a seeing machine. Many other people have talked a lot about this abstraction or distillation of the human totality, this sacred being into this really scientific biological specimen that has all sorts of other issues in it. Fast forward 500 years, and you see this in terms of human beings or our human lives being abstracted into raw data, big data to be processed.
I think there's a whole trajectory of this question, but for me personally, it didn't really come from rethinking what it means to be human. It's more in terms of thinking of the new visual language of the first person and third person.
And, the final thing I'll say is that in cinema, for example - where the relationship is much more established between the camera as this surrogate viewer or spectator - when you do have first person, like famous tracking shots in film, it's a very deliberate move to have a subjective camera, whether that's in war documentaries or this signature cinematographic scene. For me, it's this organic idea that the viewer maybe human, maybe non-human. But this question has been going on for a long time. Or is it God looking at us from high up above? There's always that shift.

Oliver Smith: I think that's really interesting what you say about the first person and the third person because this work sits in a trajectory of our works. This is an installation, but previously we've done this set of performances of which we're two out of three in, and the first one was primarily first person. So it was that camera gaze, but that gaze was still not human. That gaze was the gaze of the artificial intelligence or the system that extracted images from a simulation. Not to layer too much of another work into this discussion, but that idea of perspective and view is interesting.
This installation, similarly to the previous performance, has this stage view. There is, of course, a camera because there needs to be a camera in order to present 3D stuff in space. That's just how the technology works. We're framing it as if the audience is there together watching something that is occurring on stage so that the human comes in through the audience, which was alluded to earlier. I guess that's not only as viewers, but that is also as interactors or directors to a certain extent, or stagehands, whatever. We frame them differently in each work.
I think that is it in terms of the work for me, the intent of it. Some of that comes from the fact that we are humans thinking about these ideas. We know that our audience is a group of humans. We don't make work for machines. You know, it may be seen by machines, it may be read, it may be turned into data, whatever. But we make work for people. So there's that space where we have those concerns, and we are aware that people will bring their own concerns. So the work doesn't necessarily need to in-world point to that. The human doesn't need to be diegetic because the human is there in encounter with the work itself.

Ana-Maria Carabelea: What I meant earlier with reconsidering what is human, I guess was a reference to how we perceive ourselves in the world or the position that we have in the world and what that means for how we relate to others in the world, be that human or non-human. I was wondering whether attributing personality traits to these non-human systems, you think opens up some kind of avenues for us to start thinking in other than humanist terms? So overcoming the humanistic tradition.

Francesco Tacchini: Perhaps something to say here is also that the personality traits that emerge out of the work that we're currently doing and previous works around AI are sometimes things we have encountered ourselves. So it’s not necessarily [a question of] wanting to actively project them on, but rather wanting to investigate them because we would see them while experimenting with these models as tools. There's something obviously technical about the fact that, in some way, these personality traits mirror human personality traits since the data used for generative AI is, of course, produced by humans - well, not only, but also. I think maybe this mirroring is what we wanted to explore with the work; and the traits that we mainly have explored, and the ones that have perhaps been most relatable for audiences, are traits such as frustration, sadness, and longing in the characters that we have been developing also, an idea of dreaming or being in denial. There's this set of fragile and vulnerable characters that have emerged through sometimes making things up or lying or being overly friendly and overly keen. So, perhaps we didn't quite project these on but rather ended up exaggerating them.

Lawrence Lek: Yeah. I think there's always this interesting debate when you get into the level of traits or archetypes, whether they're eternal and they have existed since the dawn of time and we're just fitting into these ancient prehistoric straitjackets basically, or whether we're kind of inventing things along the way. One thing I want to mention is it's really interesting that nowadays when you say comedy, people think you're talking about a very specific kind of diluted, formulaic genre of entertainment. But, obviously, the idea of comedy is - I mean, I'm not an expert - but I imagine is something a little bit more cosmic than that. Essentially, it's about finding absurd humor in the nonsensical nature of the universe and causality, right? I guess many comedies are about causality, which the protagonists in the play or film don't know about. But, hey, it all works out in the end.
Actually, something I have thought about in my own work for a long time is the kind of twin eternal genre of tragedy, which is similar. For the protagonist in a tragedy, because of some fatal gap in their knowledge, things turn out really badly, you know? This idea, of course, has both to do with knowledge and fate, whether things are deterministic or whether you really have any agency in the world. A lot of my work tends towards the tragic, not just in terms of the emotionally sad, but more in terms of the unforeseen consequences of playing with technology or this kind of thing.
If I remember correctly, as well, in a lot of Renaissance renderings of stage sets, you would have a genre that has both the characters and the stage set. So you'd have a tragic scene, and it looks like this. It would have certain marks and signifiers. Obviously, the audience would recognize at a glance whether they're watching a comedy or a tragedy, not just the costumes, but the stage set too. It's all laid out, and within that malleable formula, there is still lots of generative stuff to happen. I know that dmstfctn works a lot with improvisation, not just because of the audience interaction or engagement or the unruly AI, but this idea of improvisation within a tight structure, which is something sometimes I think about.

Ana-Maria Carabelea: That's interesting because, again, in both of your works, there seems to be an interest in error and flaw and the unpredictable or the stuff that isn't already there, that isn't prescribed. Whether that's the self-driving car in NOX that needs a self-help bot or the LLMs that confabulate. And I was wondering, what [is it] about this misbehavior or the failure of these systems?
Why is that important when we're talking about these systems?

Oliver Smith: At least for me, I think there's always something interesting - and this maybe links a little bit back to this idea of determinism as well - in computation. You know, we're talking about LLMs, we're talking about AI, these are computational systems, and they're presented in lots of different ways. One way is that they are right, that they are correct, that they are useful systems, that they're good, that they have a commercial value, that they're going to solve a lot of problems, that they're going to help us, whatever. Fine. There's also this sense that they are kind of random or chaotic, but then they're built on top of computation, which we see as perfect or inalienable. It's binary, it's ones and zeros, it's on or off, it's right or wrong, whatever. But really computation is also just another chaotic kind of form.
We've managed to get the thresholds so that most of the time the ones and the zeros fire - which is the same as neural nets -, most of the time the outcome is somewhat close to what we want, but really what we're dealing with is a kind of swirling, chaotic thing that we don't really understand, no matter how much we pretend to understand. So, things flip in and out of being presented as comprehensible and incomprehensible. I think what we're doing is just the next stage in the flip, right? If LLMs are presented as comprehensible, useful, financially viable systems, then it becomes interesting to do the ‘what if they're not’? And we find then a bunch of examples and previous works have been based on these examples of systems behaving in ways that people didn't expect.
This stems back to computing history, and in people's lived experience of using computers. I mean, this is going to show my age, I guess, but the act of trying to put images and text next to each other in Microsoft Word from school [...] being the most unknowable, chaotic computational experience you can ever have, despite being settled inside this very simplified interface that was knowable. I think there is something again, archetypal about that kind of experience of using digital things.

Ana-Maria Carabelea: Yeah. I think maybe particularly with the hype around AI these days, that's important to flip that narrative because it kind of completes the picture.

Francesco Tacchini: To circle back to what Lawrence was saying around nonsense in Commedia dell’Arte, for example, and to come back to this idea of failure. There's a really interesting relationship between these two things that I think we're trying to explore with the new work where, effectively, in Commedia dell’Arte, so much would be improvised. Of course, this improvisation is built upon both the actors' skills in representing the archetypes that their mask is imposing on them, but also in the potential for failure that improvisation brings. Literally, all of the attention of the audience would be on the actors and their ability to improvise. And this could fail quite badly if the actor wasn't skilled. Something we're trying to do with this work, which has a couple of interactive components, is to actually bring in this voice from the audience, again from the humans, which are part of the system, but not present maybe in the work.
And we're letting the audience sort of upvote and downvote the work if you want. This effectively looks like throwing flowers or coins on stage or throwing tomatoes on stage, which is kind of funny.
The masks will actually react to this and stop the performance and perhaps repeat their lines. After a certain amount of negative feedback, the scene would effectively fail and the curtains would be drawn.
We wanted to play a little bit with this idea of nonsense and improvisation because it's so interesting and particular that Commedia dell’Arte didn't actually use a script, and so much of it was improvised. Almost like set pieces which were called lazzi - scenes that would unfold according to certain props or certain keywords. These are really prompts that the actors would give to each other for a certain scene to unfold. And we kind of see this in our use of LLMs, there are certain keywords that we can use, or we can almost predict how a certain conversation may go given the right keywords. We're trying to play with this and some of the traditional set pieces from  Commedia dell’Arte that we have brought directly into the work through nonsensical objects: sometimes there'll be a glass of beer or a broken mirror, that will trigger almost set pieces taken from this tradition, which is also not properly documented and has hardly been written about.
I think a lot of the work tends to fail in this sense. We really hope that these sketches, which keep going on and on and on and are generated endlessly, fail, and we see this and it's starting to be kind of ironic in an unpredictable way.

Lawrence Lek: Yeah. I just want to say, I think, with this idea of failure, I guess failure is almost…How should I put it? It's generally something to be avoided. But, of course, in certain cultures, in a startup software development world, it's like ‘Lean into failure!’, this kind of thing, ‘You're going to learn so much’. The more you screw it up, it's fine. It's fine that you burned down the server, you won't do it again. But I think there are so many different meanings and implications of failure. And so much of theater design and game design and software design is making failure acceptable, from critical failure - it crashes or the server burns down, to hostile failure - program crashes, to friendly failure. And I was just thinking when Oliver was talking about Microsoft Word, the evolution of failure. Initially, you think you failed, the user thinks they failed because they can't work out how the program works. And then, after 25 years of friendly software development interface design later, you never fail as a user. Things just get politely underlined in green. Not even in a straight green line, like a squiggly green line. I guess, because curves are friendlier than hard angles. But this continuous transition of making failure less of a critical, technological, brutal failure - which I guess maybe in the popular imagination is like the Challenger Space shuttle disaster. That's the worst kind of tech failure you can have - rocket blows up, unfortunately -, to the most dialed down [where] the consequences of getting something wrong are very safe. And actually, when Francesco, you're saying about this inability to mess up when you're doing improv. It reminds me of not just improvised theater, where the principle is you always say yes to whatever someone else gives you. You fall over, then you lean into the falling over, and then everyone falls over and then it's funny. But it reminds me of immersive theater, where you enter into the theatrical experience in whatever abandoned Victorian building and you do a murder mystery, and as an audience member, you can't screw it up in theory because the actors are trained - it's kind of like safety training - to make sure that they are always in character, no matter how drunk the guests and participants get. And I can only imagine how unruly it can become. This continuous making of the scene or the artwork or the stage into a padded cell,  a bouncy trampoline where you can do anything and it's fine, obviously has a lot of commercial and control implications as well. But the transition from critical failure to padded cell is a lot of software development and system development in general. Obviously artworks don't have to obey that, but to some extent, we do sometimes as people who orchestrate a performance. That happens.

Ana-Maria Carabelea: I find it really interesting that we spoke about Commedia dell'Arte a lot. And generally, I think in your works you both have literary techniques and motifs. Not just Commedia dell'Arte, but Lawrence, you have the coming-of-age story or the recurrent figure of the wanderer. I find that really fascinating and was wondering what, in your opinion, makes literary techniques suitable for exploring the inner mechanisms of these systems demystifying them, or making them accessible?

Oliver Smith: I think that literary techniques are suitable for exploring a whole heap of things, right? They are malleable and flexible. There's a lot that you can do with them for the kind of work that we're doing. They offer a frame to talk about something that is otherwise potentially very, very abstract. In previous things we've done, we've used narrative structures that can be known. This also comes into the current work where you really know what's going to happen in the sketch before you do it, that means that you can then discuss things that are maybe a little more abstract, a little more technical, or a little harder to grasp in a way that gives people a hook. It's not only a pop thing in that way, but I think that is one of the things maybe that it's useful for. I don't know what the others think.

Francesco Tacchini: There's something about the fact that so much of these works that we've done around AI in the last four years have been a set of models and replicas of complex systems. The literal name of the project itself, dmstfctn reveals some of this impetus. So we're taking a complex system, perhaps replicating it, remodeling it, and then exploring it. And it feels almost necessary to then build this narrative on top of it. As you say, Oliver, the idea that you need a hook, there's this process of building mythology on top of things and giving hooks and creating meaning and carving a path through these complex models, requires us to look into literary techniques and also a lot of literary references.
In one of the earlier works we did called God Mode Episode 1, which is a performance that's now become an installation, we looked into a story by Borges called Funes the Memorious - a beautiful story of a character that falls from a horse, and from that point onwards, after this injury, can never forget anything. It becomes something that remembers everything - a perfect metaphor for an LLM, if you want, or an AI - without, however, being ever able to connect these things again. So the character [Funes] is only able to remember every dog that it has ever seen without making the connection that maybe it was the same dog seen a minute before. I think in more recent works, we have also borrowed perhaps not only literary but also meaning-making techniques. In particular, we've been looking a lot at Grammelot, this made-up language invented by Commedia dell'Arte actors, which is effectively a language full of onomatopoeia, a nonsensical language. There is absolutely no meaning in it, but there's a lot of meaning that comes from the experience of listening to it. If the actor performing the Grammelot is skilled, this language would effectively sound like a local dialect, perhaps the dialect of the location where the audience is coming from. It will sometimes say more than the traditional language of the location, or Italian versus a northern Italian Grammelot. I think there's some hand-holding that can come from reusing some of these literary techniques and mainly literary references.

Lawrence Lek: Yeah, I guess for me describing these things as literary techniques - like yes, they are, but I think sometimes it makes it sound much more sophisticated or historical than it actually is. I feel that what literary techniques are is how people talk about other things. Right. So, a coming-of-age story is how I grew up, or rather, my 50% made-up version of how I grew up. Comedy is a funny thing that happened to me. Tragedy is this really terrible thing that happened to me. Of course, over time it evolves and gets more sophisticated. For example, the Borges story it's about the limits of storytelling where it's still logical, but it's not like absurd things make sense, but it's the consequence of having a story about someone who never forgets anything or the map as big as the world, the limits of what could be real. For me, the thing that really helps is thinking about ways to tie different projects together. As artists, generally, when we get commissioned to do short works, you know, like writers get commissioned to do articles and short stories, never a novel. We make video games, but it's an episodic video game, partly out of desire, but quite often - at least in my case - very much out of necessity because of the way that we can fund, or at least I can fund the creation of this episodic stuff needs to adapt to how we get the commissions for that, which isn't a bad thing in terms of making it evolving and traveling. But I think the thing about literature that helps me ground myself is thinking about long-form narratives, things that work over the course, in some cases over a series of, let's say, novels. That's why I use cinematic universe, which is very grand, but it's just keeping the same story world over a long time. It's this long-form trajectory that I think I borrow from the most.

Ana-Maria Carabelea: That's cool. It’s interesting because I did think about telling stories for someone else or speaking for someone else in that story, which is the literary exercise that I was thinking about and that I sort of picked up on. But yeah, it's really interesting, and I didn't think about long form.
Great. We're approaching the end, and at the end, I always ask (for those who want to dive deeper into these topics) for some recommendations for some artworks, books, or whatever you can think of.

Francesco Tacchini: I'm going to go with perhaps a book that both myself and Oliver have been reading as part of the development of this project and during the research phase, called The Tricks of the Trade by Dario Fo. In Italian, this book is called Nuovo manuale minimo dell'attore, which means minimal manual for the actor. It's an amazing overview of improv theater techniques, but also a history of Commedia dell'Arte and an excavation of where the masks and their making come from.
For me, the most fun part of this was the reporting of all the lazzi that I mentioned earlier, these set pieces, which are extremely amusing. And yeah, it’s just a great read from the world of improvised theater. But I think it can teach a lot, coming back to what Lawrence was saying earlier, especially if you're working with a fairly chaotic system, like a video game engine and an installation that is interactive or performance, and having to pad that system. There's a lot that can be learned from this improv theater tradition, I think.

Ana-Maria Carabelea: Nice.

Lawrence Lek: Hard to choose something, but a friend, James Stringer, just released a video game on Steam called Dadda. I guess it's not absurd; it makes sense, but it's a music video game in which you control a music-making robot automaton, basically. So it's called Dadda by Brood Ma on Steam now.

Oliver Smith: Nice. I'm gonna set that downloading. Now that I've got all the data for the installation downloaded, I can install [video games] again. I would just add a maybe less enjoyable reference, but one that's been a part of the making of the work, which is a paper called Roleplay with Large Language Models. Francesco found it, I can't even remember when, but it's really useful. It uses roleplay as a way and it speaks to some of the questions we've been talking about today. It talks about roleplay as a way to understand the kind of empathy or the understanding that we get from LLMs, this idea that they're not beings necessarily, they're not human by any stretch of the imagination, but they are playing a role, or they're kind of programmed in a way that outputs as if they're doing that. And it uses that then as a kind of metaphor, it speaks of finding that as a new metaphor to talk about these things. I think it's a really useful paper to think about this. It speaks to the reason that some of this is theatrical, et cetera.

Ana-Maria Carabelea: It's a performance, basically.

Oliver Smith: Right, exactly. Yeah, it's a computational system performing in multiple ways as an actor, as a thing that works, and so on and so forth.

Ana-Maria Carabelea: Cool. Thank you so much. I have one more big announcement that I'm going to pass on to Oliver and Francesco because you've got a premiere coming up in Bologna. Can you tell us about that?

Francesco Tacchini: Yes. So, on Friday, 7th of February, we're launching this new work, The Models. It will be displayed for three days as part of the art week of Bologna called Art City. Actually, interestingly, this will be displayed at Technopolo, which is the location where the fourth most powerful supercomputer in the world is housed. We haven't really spoken about this today, but this supercomputer was used as part of the production of the work. [Oliver], do you want to say something only about this since you’ve just downloaded everything from it?

Oliver Smith: That's why I don't have any games left now. We've been using it to generate the scripts. I've got this set up on it: there's a folder called Writer, there's a folder called Voice Actor, there's a folder called Translator, there's a folder called Scribe. And it's basically running the improvisational process for this installation. Got about 13,000 plays so far, or baselines of things that these characters were then improvised from. And we're just kind of watching them go. It's been a really… I don't know how we got the keys to it, but I'm very thankful that we did.

Ana-Maria Carabelea: Nice. That sounds great. And very much looking forward to seeing the work. Lawrence, have you got anything exciting coming up this year?

Lawrence Lek: Yeah, I've got a few shows coming up. I'm going to Singapore and Bangkok later this month to do a couple of performances and screenings. And then I have my last short film, Empty Rider, showing at Rotterdam Film Festival. So that's exciting. And then, yeah, some shows other than that.

Ana-Maria Carabelea: Nice. Well, thank you so much for joining me today. It's been great to have you all.

Oliver Smith: Yeah. Thank you.

Lawrence Lek: Thank you.

Francesco Tacchini: Thanks.

Ana-Maria Carabelea: That's it for today. Thank you so much for tuning in, and I hope you enjoyed it. The Digital Deal Podcast is part of the European Digital Deal, a three-year project co-funded by Creative Europe. If you want to find out more about the project, check out our website, www.ars.electronica.art/eudigitaldeal.