Searching Graphics and Text in Technical Documents: A Brief Overview and Plan

          @misc{ scivideos_PIRSA:25040072,
            doi = {10.48660/25040072},
            url = {https://pirsa.org/25040072},
            author = {Zanibbi, Richard},
            keywords = {},
            language = {en},
            title = {Searching Graphics and Text in Technical Documents: A Brief Overview and Plan},
            publisher = {Perimeter Institute for Theoretical Physics},
            year = {2025},
            month = {apr},
            note = {PIRSA:25040072 see, \url{https://scivideos.org/pirsa/25040072}}
          }
          

Abstract

What would effective and usable tools for searching text and graphics in research papers look like? In this talk we sketch a partial answer to this question, with reference to recent work in the Document and Pattern Recognition Lab at RIT. Two multimodal paper search prototypes, one for math (MathDeck) and one for chemistry (ReactionMiner search) will be used for illustration. A simple framework based on 'jars' of available information sources can organize and relate the actions performed by people and automated systems when retrieving, analyzing, and synthesizing sources. We will organize our answer sketch around this framework, and share open questions and research opportunities related to enhancing multi-modal search tools for expert and non-expert users. Note: ReactionMiner was developed in collaboration with NCSA and the Han lab at the University of Illinois, Urbana-Champaign. MathDeck demo: [https://people.rit.edu/ma5339/mathdeck_landing](https://people.rit.edu/ma5339/mathdeck_landing) ReactionMiner search demo: [https://reactionminer.platform.moleculemaker.org/home](https://reactionminer.platform.moleculemaker.org/home) Biography: Richard Zanibbi is a Professor of Computer Science at the Rochester Institute of Technology (RIT, USA) where he directs the Document and Pattern Recognition Lab (dprl@RIT). His research focuses upon the recognition and retrieval of graphical notations, particularly for mathematics and chemistry. He is also a member of the Molecule Maker Lab Institute (MMLI), one of the first NSF AI Centers. He received his PhD from Queen's University (Canada), and was an NSERC Postdoctoral Fellow at the Centre for Pattern Recognition and Machine Learning (CENPARMI) at Concordia University before joining RIT.