Get the latest tech news
Unlocking the power of time-series data with multimodal models
November 25, 2024 Mathias Bellaiche, Data Scientist, and Marc Wilson, Software Engineer, Google Research We compare the performance of multimodal models on the understanding of time-series data when presented visually as plots compared to numerical values. We find significant performance improvements when presented with plots on tasks like fall detection.
This type of data is made up of streams of values that change over time, and can represent topics as varied as a patient’s ECG signal in the ICU or a storm system moving across the Earth. We did this to probe performance generally on tasks of increasing difficulty and reasoning sophistication, ranging from naming the pattern being plotted to multiple choice questions a university calculus student might encounter. We would like to acknowledge Mayank Daswani for leading this work and all the collaborators across Google including Desislav Ivanov, Mikhail Papkov, Eva Schnider, Jing Tang, Kay Lamerigts, Gabriela Botea, Michael Sanchez, Yojan Patel, Shruthi Prabhakara, Shravya Shetty and Umesh Telang for contributing to the research and to Sebastien Baur, Yun Liu and Diego Ardila for their valuable input.
Or read this on Hacker News
