AMIA 2024 | AI Program Stream

In the program for 2024, the Conference Committee recognized an organic stream of sessions addressing the opportunities and challenges of AI. This informal stream of programming will kick off with a keynote on Wednesday morning, December 4th.

8:45 AM – 10:45 AM
AMIA 2024 Welcome
Keynote Conversation

2:00 PM – 3:00 PM
Best Practices for Use of Generative AI in Archival Documentaries
Rachel Antell, Archival Producers Alliance
Stephanie Jenkins, Archival Producers Alliance
Jennifer Petrucelli, Archival Producers Alliance

Generative AI (GenAI) is flooding our world with a dizzying amount of synthetic media, and there is little guidance on how to responsibly navigate this new reality. The Archival Producers Alliance will present the Best Practices Guidelines we have developed for use of GenAI in documentary films. In this panel, we will explore the potential risks that GenAI presents to the Archive, to preservation, to the historical record, and to documentary film–as well as ways of mitigating these risks when using it. We will also discuss the methodology that went into developing the guidelines and how we see the Archival and Documentary spaces evolving in light of this powerful new technology.

4:30 PM – 5:30 PM
Whisper AI Transcription, Human Implementation
Nina Rao, Emory University
Simon O’Riordan, Emory University
Owen King, GBH Archives
Emily Lynema, Indiana University

Since the 2022 debut of Whisper, OpenAI’s automatic speech recognition software, an ecosystem of complementary tools and modifications has evolved, now offering organizations stewarding AV materials unprecedented opportunities to leverage this tool to increase the accessibility and discoverability of their digital collections. In this session, panelists from Emory, GBH, and Indiana will discuss their experiences and research implementing Whisper into their media management and preservation workflows, discussing Whisper’s performance across varied AV collections as well as the human impacts of working with Whisper.  These insights from three organizations with varied scopes of collections and distinct but connected avenues of research may be helpful to a variety of organizations engaged in similar work or interested in starting up such a project. Attendees will gain insight into the strengths and weaknesses of leveraging Whisper, the challenges and opportunities presented by the technology, and practical guidance on implementing or expanding AV accessibility-related projects.

9:30 AM – 10:30 AM
Human-centered AI-assisted Video Cataloging
Raananah Sarid-Segal, WGBH
Owen King, WGBH
Miranda Villesvik, WGBH
Caroline Mango, WGBH

This panel will present a human-centered approach to AI-assisted cataloging.  Panelists from GBH Archives, working on the American Archive of Public Broadcasting, will describe the development, implementation, testing, and use of AI-based media analysis tools within workflows for item-level cataloging digital videos.  Panelists include cataloging, digital ingest, and metadata operations staff who have been involved in the creation and use of AI-based cataloging tools. We will discuss the CLAMS (Computational Linguistics Applications for Multimedia Services) suite of open source AI tools, post-processing CLAMS output for use in cataloging, questions we sought to answer regarding cataloging ease and efficiency, and the results of our experiments with tool integration. We will explain how our approach and roadmap differs from initiatives seeking to make cataloging fully automatic.

11:00 AM – 12:00 PM
Roundtable: GenAI and Archives: Understanding the Implications
Rachel Antell, Archival Producers Alliance
Stephanie Jenkins, Archival Producers Alliance
Jennifer Petrucelli, Archival Producers Alliance

Having published a set of best practices guidelines for the use of generative AI in documentaries, the APA is looking to join with archives in a discussion of the risks and benefits of engaging with genAI. This roundtable will be a space to share experiences, insights & questions; and to potentially lay the groundwork to create guidelines for archives in navigating this new landscape.

11:00 AM – 11:30 AM
Navigating AI Integration in Audiovisual Archives: Practice & Policy
Johan Oomen, Netherlands Institute for Sound & Vision

This presentation explores research on integrating AI in various organisational contexts, with a focus on audiovisual archiving. AI’s potential is evident in search & exploration, preservation, artistic expression, and big-data analysis. Policies like the AI Act and Ethical Guidelines for Trustworthy AI shape strategies for responsible AI use. However, integrating AI in audiovisual domains poses challenges, such as choosing between off-the-shelf and bespoke solutions, aligning AI with legacy systems, considering public values in procurement, ensuring scalability and long-term viability, and fostering AI literacy. The audiovisual domain’s specificity necessitates developing good practices. The AI4Media Network of Excellence engaged media practitioners to capture their experiences. This presentation shares these insights, offering practical guidance on sustainable, responsible AI integration into workflows and formulating policies for AI technology selection and use.

11:30 AM – 12:00 PM
Exploring Experimental Machine Learning in Film Restoration
Fabio Bedoya, Filmworkz

In this session, we will explore the cutting-edge applications of machine learning in film restoration, addressing not only color recovery and frame replacement but also the intricate processes of colorization, source/gauge matching, and nitrate decay recovery. Through a series of detailed case studies, I will demonstrate how AI tools are revolutionizing the field, making advanced restoration techniques more accessible.  Drawing from a rich background in digital intermediates and a history of international collaboration, I will guide attendees through the ethical and practical considerations of integrating AI into film preservation workflows. The session is designed for those with an intermediate understanding of the field, but beginners will also find the discussions enlightening and informative.  Participants will leave with a deeper appreciation of the capabilities of machine learning in film restoration. We will delve into how these tools can be leveraged to overcome traditional challenges, ensuring the longevity and integrity of our cinematic heritage.

12:00 PM – 2:00 PM
Poster: Responsible AI in the Archive: Finding Equity in Partnerships
Jimi Jones, University of Illinois at Urbana-Champaign

AI companies are eager to amass collections of content to train both large language and niche models. In some cases, companies turn to archives and pay handsomely for what, for them, is valuable and unique raw material. Only the largest and most well-resourced archives have been able to take advantage of this opportunity. We are exploring models that better highlight the value of archival content, give institutions more opportunity for partnerships, and try to give more balance in negotiations. We outline how rights issues can be resolved with more technical knowledge and better understanding of intended use of archival material. Importantly, we propose how, working collectively, archives can foster a new revenue stream for digitization, preservation, and improved access to their collections.

 2:00 PM – 3:00 PM
Transforming Audiovisual Archives with AI: Innovations, Challenges, and Ethical Considerations
Zack Ellis, TheirStory
Sandra Aguilar, USC Shoah Foundation
Doug Boyd, University of Kentucky Louie B. Nunn Center for Oral History

Join us for an insightful discussion on the transformative impact of AI on audiovisual archiving. Traditional methods of transcription, indexing, and cataloguing have been laborious and costly, yet essential for enhancing the usability of collections. Over the past decade, AI has promised a revolution in these processes, though its effectiveness has been debated. Recent advancements, including OpenAI’s Whisper and ChatGPT, have significantly enhanced AI transcription accuracy and mainstreamed AI technologies. Concurrent developments in AI Named Entity Recognition (NER) further facilitate the automatic extraction of key entities. Organizations like the USC Shoah Foundation, University of Kentucky’s Louie B. Nunn Center for Oral History, and the TheirStory oral history platform have embraced these innovations, integrating them with tools. Join Sandra Aguilar (USC), Doug Boyd (University of Kentucky), and Zack Ellis (TheirStory) as they share their experiences, explore the benefits and challenges, and discuss the ethical considerations shaping the future of audiovisual archiving.

3:15 PM – 3:45 PM
Large AI Models for Video Content Summarization
Kyeongmin Rim, Brandeis University
Kelley Lynch, Brandeis University

This presentation will introduce video content summarization (VCS) and its real-world applications. We’ll explore how artificial intelligence (AI), particularly large language models (LLMs) and large vision-language models (LVLMs), can be used to understand and summarize video content, and then we will show how pipelined approaches with proper pre- and post-processing elements can improve the results. Additionally, we’ll cover the effectiveness of AI in video summarization and the challenges that remain. Our goal is to provide the audience with a broad understanding of VCS and how AI is transforming this field. We’ll also showcase a practical implementation of VCS pipelines in the CLAMS project, an open-source AI-assisted metadata extraction platform developed via collaboration between Brandeis University and AAPB (GBH MLA).