Skip to Main Content
Skidmore College

Resurrecting a voice and creating new memories through AI

June 14, 2024
by Angela Valden

What does it mean to be in possession of your voice, and what would it mean for someone else to possess it? How do you reconstruct someone who is no longer here, and how real would that version of them be? Would it matter how real they are? 

Sarah SweeneyThese are some of the many philosophical, ethical, and technological questions that artificial intelligence (AI) has brought to the fore and that Sarah Sweeney, associate professor and chair of art, is contending with through her long-term project “Conversations with My Deepfake Dad.”

Sweeney’s father, a screenwriter who documented his work and life in tape recordings, died at the age of 44, when Sweeney was 17.  

“When I turned 44, I had all these thoughts about what it was like to be the same age he was when he died,” Sweeney says. “Was I at the same place in my life?”  

Around that time, she also began to take an interest in deepfakes — media generated using AI to replace someone’s likeness — and the implications of such capabilities.  

“As I was looking through flash drives and found old movies of my dad when he was very young, I had this thought: What would happen if you could make new memories with someone who had died?”

Candid photos of Sweeney's father

Candid photos of Sweeney's father

Sweeney contacted a company called Resemble AI and began working with the firm to create a clone of her father’s voice using its speech-to-speech software. “It takes someone’s speech — we’ve been calling them ‘the puppeteer,’ the person who’s speaking — and maps my dad’s voice onto it,” she explains.  

Over the next several months, she found six hours of her father’s recordings and worked with Resemble AI and her mother to think through the ethical and technological challenges of generating new conversations using her father’s voice. “I think a big question in this project is, ‘Who has the right to say that someone who is no longer with us can be used in a deepfake?’ I had to produce a death certificate and a letter of consent from my mother. 

As Sweeney moved forward with the project, she considered what the conversations with her father would look like. “I started thinking about, when someone dies, where might those answers come from?” she recalls. “One of the things I became really interested in is how we hold someone after they die and how we imagine they’re the only version of that person. But in fact, there are maybe 60 versions of that person in the world held by different people.” 
 
She decided she would generate six audio conversations, each based on different archival materials, interviews, and research. Sweeney writes both sides of each 20-minute-long conversation. She then speaks her side and her dad’s deepfake voice speaks his.  

“The first dad is based on interviews with my mom and my sisters. All his answers are written from what they think he would have said. So I got to meet my mom’s version of my dad and my sisters’ version of my dad. And I’m learning a lot about him through that.”  

“The second one is just him, so that one is really interesting. It’s only words that he used in his letters and his tapes.” 

“The third one is therapy sessions. They’re called empty chair therapy sessions. The fourth is a medium. The fifth is a chatbot. And the sixth is just voice memos to myself that I’ve been recording in the car.” 

The audio recordings, along with photographs, will constitute an online presence for her father.

From left, Science Machine Shop Supervisor/Technician Jeff Britton, Senior Teaching Professor of Physics Evan Halstead, Zoë Bilodeau ’23, Omar Gomez ’23, and Associate Professor of Art Sarah Sweeney work together to install the phone booth in Scribner Library.

From left, Science Machine Shop Supervisor/Technician Jeff Britton, Senior Teaching Professor of Physics Evan Halstead, Zoë Bilodeau ’23, Omar Gomez ’23, and Associate Professor of Art Sarah Sweeney work together to install the phone booth in Scribner Library.

Sweeney has taken the audio component to several classes at Skidmore, including psychology, English, media and film studies, and First-Year Experience classes.  

“I listen to this file with them and then we have a conversation.”  

Sweeney is keenly aware of the complex range of issues that her project raises, and she is one of a number of Skidmore faculty members — from Associate Professor of Psychology Jess Sullivan, who taught the course Artificial Self in spring 2023, to Professor of English Mason Stokes, who taught the course Writing with AI in fall 2023 — who are grappling with the complexity of AI through their research and teaching. Skidmore’s Center for Leadership, Teaching, and Learning and the College’s Learning Experience Design and Digital Scholarship Support (LEDS) teams even organized a semester-long AI Think Tank devoted to exploring some of those issues this past fall.  

Sweeney’s project continues to unfold, and rapidly evolving technologies continue to raise ethical, philosophical, legal, and other questions. Sweeney hopes her project will be thought-provoking in a multitude of ways, to many different audiences.

“If you look at the news, the worry right now is that our world is changing such that we can’t figure out what’s real and not real. This piece is a little bit about saying, ‘Is there something interesting in something that’s both real and not real?’ And I think that’s what I really want us to consider about these technologies that work with our memories. They’re kind of incredible but also really scary. And I think that’s what this piece is really about — is this incredible possibility of this strange conversation I’m having with my dad.

“One of the things people often ask me is, ‘Does it matter that it’s real or not real?’ And I don’t think it does. At some points it really doesn’t matter. It feels like my dad.” 

Presented through a collaborative Skidmore effort

The “Conversations with My Deepfake Dad” audio recordings that Associate Professor and Chair of Art Sarah Sweeney creates are being shared online as well as through physical exhibitions.  

The first conversation was presented through a phone booth, which debuted in October 2022 at CultureHub in New York City. A parallel piece — a rotary phone — was sent to Los Angeles, and the phone booth installation was brought to Scribner Library for the spring and summer of 2023.

A number of collaborators at Skidmore have helped to bring the installation to life. The Art Department’s shop supervisor and building safety coordinator, Paul Davis, constructed the phone booth.

Andrea Casey, art technology support coordinator and program coordinator, created the prints for inside. Senior Teaching Professor Evan Halstead and Science Machine Shop Supervisor/Technician Jeff Britton from the Physics Department, along with three of Halstead’s independent study students — Omar Gomez ’23, Zoë Bilodeau ’23, and David O’Connell ’23 — assembled the electrical components that allow the conversation to start and stop with the opening of the phone booth’s doors.

“What I’ve learned is that by partnering with people who have this incredible expertise — students, faculty, staff — the piece is so much richer, and I could never have made this thing by myself,” Sweeney says.  

To listen to the conversations, visit www.mydeepfakedad.com

 

Related News


Forensic+Anthropology%3A+Bones%2C+Bodies%2C+and+Trauma
The course Forensic Anthropology: Bones, Bodies, and Trauma is an interactive introduction to crime scene investigation with a racial justice component.
Jun 28 2024

The+Skidmore+community+celebrating+Pride
The Skidmore community celebrated all things Pride during LGBTQIA+ Pride Month with mocktails, light snacks, music, Pride swag, and great company.
Jun 24 2024

Jennifer+Cristiano+%E2%80%9918+stands+on+the+deck+of+a+boat+in+her+uniform.+A+range+of+mountains+can+be+seen+from+across+the+water.
The Skidmore Scholars in Science and Mathematics (S3M) program was established in 2007 as a way to provide financial support to students pursuing STEM majors; since then, some 130 students have benefitted. Hear the stories of five S3M alumni who have gone on to do amazing things.
Jun 18 2024