richardbrath

TLDR? Use this link for a visualization of the 800 movies in the National Film Registry. Any movie with the movie emoji 🎥will link directly to that movie (or clip) on Internet Archive or Library of Congress. Find classic movies, see similar films, click to watch or read about them. You’ll get a visualization that looks like this, you may need to use browser-zoom and/or scrolling if your screen is small:

Longer answer:

I’ve been fiddling with the dataset of movies in the National Film Registry (from the Library of Congress) for a while, including using LLMs throughout the entire process. The first step was simply to scrape the data and process it into a form that I could use (using an LLM to write a script to scrape and process the data). In addition to the movie title and year, each movie includes a description: e.g.

“Seventh Heaven” (also referred to as “7th Heaven”), directed by Frank Borzage and based on the play by Austin Strong, tells the story of Chico (Charles Farrell), the Parisian sewer worker-turned-street cleaner, and his wife Diane (Janet Gaynor), who are separated during World War I, yet whose love manages to keep them connected. “Seventh Heaven” was initially released as a silent film but proved so popular with audiences that it was re-released with a synchronized soundtrack later that same year. The popularity of the film resulted in it becoming one of the most commercially successful silent films as well as one of the first films to be nominated for a Best Picture Academy Award. Janet Gaynor, Frank Borzage, and Benjamin Glazer won Oscars for their work on the film, specifically awards for Best Actress, Best Directing (Dramatic Picture), and Best Writing (Adaptation), respectively. “Seventh Heaven” also marked the first time often-paired stars Janet Gaynor and Charles Farrell worked together.

Themes by decades: a dead-end

It’s an interesting dataset with all this qualitative data (paragraph per movie). But how to depict all those movies? The immediate response is to slot all those movies into standard movie categories: horror, drama, action, comedy, scifi, adult, children, etc., … but that just uses standard conventions. I wanted something a bit more nuanced, perhaps using the qualitative data.

To do a quick viz to get a sense of the data, I plotted it as a timeseries based on release date, with length of commentary on the y-axis, little stars based on title length, and color by year (using an LLM to write the code):

It’s pretty standard viz and not very interesting. But the axis has tick marks per decade: perhaps a decade-based analysis? An LLM could process the paragraphs into themes per decade—that might provide a zeitgeist of various decades as reflected in film: decades of angst, decades of hope, etc. After prompting an LLM to create themes with descriptions, then align those with the visualization, plus add some interaction linking movies to themes and vice versa, and some color coding, here’s the result:

The visualization nicely aligns the themes (table on right side) with the movie dots (scatterplot on left side), with a shared axis of decades (thanks LLM). But the themes are under-whelming: e.g. Silent Comedy Era, City Life & Progress, Blaxploitation & Black Cinema. There is an interactive linkage between the two: click on theme (right), highlights and labels corresponding movies (left); then point at a movie for the paragraph. In the snapshot above, the theme “SciFi” in 1980’s has been clicked, highlighting movies such as E.T., Bladerunner, Back to the Future, and Star Trek has a tooltip. If the goal of this design is to understand a theme in it’s context, there is no way to see the contextual movies in a decade without tedious mouse-over across each movie. And the screen is already full of text, so it becomes difficult to fit more contextual text in. Design-wise, this is a fail.

Team LLM: designer, data processor, viz maker, code explainer

I decided to go all in with an LLM-centric approach, using several simultaneous LLMs:
1) Design assistant LLM, to ideate on design ideas,
2) Data processor LLM, to fetch and enrich data,
3) Visualization code generator LLM, to build and refine the overall visualization,
4) Code explainer LLM, to help me understand and enhance bits of code.

The design assistant LLM was instructed to suggest visualizations for movie data. Those suggestions needed to use the descriptions per movie, and furthermore was not allowed to suggest visualizations that used standard movie categorization. It suggested many possibilities: scatterplot (narrative structure vs observational to expressive, with time animation), timeline with bands (similar to what I’d just done), ternary plot with time slider, heatmap timeline of emotional tone over decades, semantic embedding e.g. t-sne or UMAP, parallel coordinates, semantic embedding with decade contours.

Even though I explicitly asked it to use text-centric representations, the LLM pushed back, following it’s training material (i.e. current visualization texts recommend against too much text). So I pushed back, and decided that movie titles would be good to incorporate: many people will recognize a movie by it’s title, and the title will automatically trigger memories. By exposing the titles, the visualization should be more engaging.

One common thread (bias) across all suggestions from the LLM is to extract quantitative values from the text. Semantic embedding seemed interesting: a semantic embedding takes text, such as paragraphs, and converts those into a many dimensional vector. Then, those vectors can be processed so that things that are near each other in the high-dimensional space will be near each other in a plot; and things far away from each other in the high-dimensional space are far away in the plot. The LLM data processor wrote a script to process paragraph descriptions into document vectors, and the visualization generator LLM created a UMAP visualization of the result, including movie titles as marks instead of dots, and a graticule to indicate how the UMAP projection compressed and stretched space in computing the projection (i.e. a stretchy grid behind the dots). Here’s the underwhelming result:

Essentially, there appears to be one cluster, and a bunch of outliers. Projection visualizations are finicky to various parameters, so I had the explainer LLM outline the parameters and fiddled much. But at best I only achieved two distinct clusters. This might be due to a lot of common words about how movies are discussed within a single paragraph, and might need much better text processing to differentiate between the descriptions.

Looking back over the other visualization suggestions, I noted that the designer LLM had suggested a wide variety of metrics across the different visualization suggestions. Instead of clustering based on words, other movie attributes could be used. Working together the designer LLM we came up with 20 attributes, such as:

• Narrative_Abstraction: ranging from a simple linear narrative, an episodic narrative (e.g. flashbacks), or abstract narrative (e.g. stream of consciousness).
• Fictionality: ranging from a direct recording of the realworld, to staged recording intended to be like the realworld, to a recording entirely unlike the realworld.
• Comic emphasis: ranging from none, to light, to slapstick.
• Etc.

Then, we need to get values for each of those, for all the movies. We could derive those values from the paragraph of text. Or, since the LLM is trained on the entire Internet, ask the LLM to use all of its training knowledge to generate meaningful values for each attribute for each movie. Using the latter approach, the data processor LLM (Gemini) generated 20 attributes x 800 movies in half an hour (i.e. 16,000 values). A quick spot check on these suggested they seemed appropriate, but a thorough analysis was not done. Then, these new vectors are used with the same UMAP visualization. A slightly better layout of the movie results (with a number of tooltips displayed):

Now there are at least five distinct clusters. There are actually more clusters but there’s a fundamental design problem making it difficult to see them. The clustering layout code is pulling similar items close together. But the text collision code is pushing items too close together apart so the text doesn’t overlap. As a result adjacent clusters end up blending together and similar clusters aren’t distinct. Fiddling with parameters is insufficient.

Something different is needed.

High dimensional projections squish high-dimensional data down to two dimensions. The distance between points on the 2D plane is not uniform. In one part of the plot, two adjacent points may be extremely close in their original vector space, in another part of the plot, two adjacent points may be far apart. Locally, points can be compared, but observations related to distance between points in one part of the vis are not relevant in another region of plot. The UMAP layout is too focused on the placement of individual points for our needs.

Thus, why not throw away the projection of individual points? Instead, cluster the individual movies directly; depict each cluster in a way that is distinct, compact, and with each title readable; and, keep adjacent clusters near each other. (I did something similar in Chapter 10 of Visualizing with Text, but displayed only top keywords in each cluster, not hundreds of movie titles and thousands of words).

The design LLM suggests a data science clustering program to prepare the data (i.e. leveraging Python libraries for clustering), and a separate visualization program (using D3 and HTML for interactive visualization). Hence the data processing LLM creates a clustering program; and the visualization LLM creates a boxes with text plus graph-layout viz. Here’s the result after a few iterations:

Iterations are required to tweak many things, e.g. don’t allow clusters that have 80% of the dataset, fill in data for movies that are missing vectors, add descriptions for each cluster (i.e. identify which attributes are most different for each cluster compared to the rest of the data), enhance data with valid urls from other sources when there is no movie or essay available to click through in the original dataset, adjust color scales, multiple columns per cluster to make them more square, adjust collision bias, add a color legend, add a cluster legend, darken colors, add halos to make text more legible, and so on.

One interesting iteration was to work with the designer LLM to create better human-readable labels for the attributes. The initial names of the attributes included RealityIndexicality, WorldDistance, and Scale. Which is fine for behind-the-scenes data science, but not for a user interface. For example, the preceding attributes were renamed to Fictionality, World_Fantasy, and Scope_of_Action. (And then reprocess all the data, and update the visualization).

The result is the visualization image at the beginning of this post (and this interactive version).

So…. what does this visualization show? It’s a plain-looking visualization: a bunch of boxes jostling around so they stay close but don’t overlap. It’s much less exciting visually than some of those earlier visualizations. But it shows some interesting similarities within and across groups, over time, with all the movie titles clearly readable, with actionable flow via tooltips and click-throughs. Here’s a closeup:

The two complete boxes in the center are highly similar. They both contain alot of animation (Shrek is part of the National Film Registry? Really?) and fantastic worlds (King Kong, Flash Gordon). The cluster higher world_fantasy implies the world modeled is more fanastic than the inventive story, and slightly the other way around for cluster higher fictionality (i.e. more emphasis on fictional story or more emphasis on the fictional world).

Below left is a cluster differentiated with future_orientation (2001, Alien, Blade Runner, Matrix). Below right is cluster with lower future_orientation (Robin Hood, Black Pirate, A League of their Own; i.e. movies looking backwards in time). Within clusters we can find some interesting films that might not be part of the same cluster if we’d used strict genres such as horror, scifi, etc., For example, Michael Jackson’s Thriller (a music video), and Apocalypse Now (a non-standard war film leaning towards psychological thriller) are in the same cluster as Alien… that makes sense. And E.T. isn’t in future_orientation, rather E.T. is together with animations such as the Little Mermaid and King Kong, which might also make sense: all fantasies where someone from a different world comes into our world and friction happens.

The boxes are also tinged with the average color of the films contained. In the image above the future_orientation box is more orange—indicating it is composed of more recent films. The two full boxes above (high fictionality) have many purple (old) films and yellow (recent) films; whereas the future_orientation box has many orange (1970’s,80’s) films. This implies high fiction/high world_fantasy skipped a couple decades where the focus was on similar but future_oriented films.

Team LLM reflection

In many of the previous visualizations that I’ve created via LLM in this blog, I’ve had a clear objective of the target visualization upfront. The challenge, for those visualizations, is crafting the appropriate prompt for the LLM, and to do so in a manner without a lot of back-and-forth with the LLM (it is desirable to avoid back-and-forth interaction with the LLM to avoid introducing new bugs).

In this case, the target visualization was not defined. It was a more nebulous: “here’s 800 movies, lots of context, how can we arrange them (without using standard movie categories), to see similarities across movies”. Which calls for an iterative design process allowing for investigation of a variety of different approaches with successive refinement on those approaches to understand what a given design is affording or hindering. Team LLM uses many LLM personas iterating together in a small scale visualization design and development task.

I work with teams of amazing people who are on a well deserved year-end holiday. Team LLM is a poor stand-in compared to those teams. But Team LLM is not bad: you can still iterate with Team LLM, and move back and forth between ideating, sourcing data, modelling data, visualizing data, repeat. Team LLM doesn’t push back the way humans do, so it’s easier to end up in dead-ends. Improving Team LLM’s code incrementally can be frustrating: some directions and tweaks you have give up on, even though human-team would be able to better ascertain the objective of a particular feature and solve for the objective. The multi-team member approach to Team LLM might not be a good idea: one member can break something needed by the next member. Eventually, there will be more ways to weave together human teams and iterative elements from Team LLM.