Video scraping is only one of many new methods attainable when the newest giant language fashions (LLMs), corresponding to Google’s Gemini and GPT-4o, are literally “multimodal” fashions, permitting audio, video, picture, and textual content enter. These fashions translate any multimedia enter into tokens (chunks of knowledge), which they use to make predictions about which tokens ought to come subsequent in a sequence.
A time period like “token prediction mannequin” (TPM) is likely to be extra correct than “LLM” today for AI fashions with multimodal inputs and outputs, however a generalized various time period hasn’t actually taken off but. However it doesn’t matter what you name it, having an AI mannequin that may take video inputs has attention-grabbing implications, each good and probably dangerous.
Breaking down enter boundaries
Willison is much from the primary particular person to feed video into AI fashions to attain attention-grabbing outcomes (extra on that under, and this is a 2015 paper that makes use of the “video scraping” time period), however as quickly as Gemini launched its video enter functionality, he started to experiment with it in earnest.
In February, Willison demonstrated one other early software of AI video scraping on his weblog, the place he took a seven-second video of the books on his bookshelves, then obtained Gemini 1.5 Professional to extract the entire ebook titles it noticed within the video and put them in a structured, or organized, listing.
Changing unstructured knowledge into structured knowledge is vital to Willison, as a result of he is additionally a knowledge journalist. Willison has created instruments for knowledge journalists previously, such because the Datasette challenge, which lets anybody publish knowledge as an interactive web site.
To each knowledge journalist’s frustration, some sources of knowledge show proof against scraping (capturing knowledge for evaluation) attributable to how the info is formatted, saved, or offered. In these instances, Willison delights within the potential for AI video scraping as a result of it bypasses these conventional boundaries to knowledge extraction.
