Toby Jia-Jun Li

Publication Activity (10 Years)

Years Active: 2014-2024
Publications (10 Years): 82

Top Topics

Natural Language

Intelligent Assistant

Qualitative Analysis

Top Venues

CHI Extended Abstracts

Publications

Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu
AutoDroid: LLM-powered Task Automation in Android. MobiCom (2024)
Jie Gao, Simret Araya Gebreegziabher, Kenny Tsu Wei Choo, Toby Jia-Jun Li, Simon Tangi Perrault, Thomas W. Malone
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration. CoRR (2024)
Jie Gao, Simret Araya Gebreegziabher, Kenny Tsu Wei Choo, Toby Jia-Jun Li, Simon Tangi Perrault, Thomas W. Malone
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration. CHI Extended Abstracts (2024)
Yuwen Lu, Meng Chen, Qi Zhao, Victor V. Cox, Yang Yang, Meng Jiang, Jay Brockman, Tamara Kay, Toby Jia-Jun Li
Crepe: A Mobile Screen Data Collector Using Graph Query. CoRR (2024)
Shivani Kapania, Ruiyi Wang, Toby Jia-Jun Li, Tianshi Li, Hong Shen
"I'm categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices. CoRR (2024)
Yifan Zhang, Jiliang Li, Zachary Karas, Aakash Bansal, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, Yu Huang
EyeTrans: Merging Human and Machine Attention for Neural Code Summarization. Proc. ACM Softw. Eng. 1 (FSE) (2024)
Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, Simon Tangi Perrault
CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models. CHI (2024)
Yifan Zhang, Jiliang Li, Zachary Karas, Aakash Bansal, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, Yu Huang
EyeTrans: Merging Human and Machine Attention for Neural Code Summarization. CoRR (2024)
Aakash Bansal, Robert Wallace, Zachary Karas, Ningzhi Tang, Yu Huang, Toby Jia-Jun Li, Collin McMillan
Programmer Visual Attention During Context-Aware Code Summarization. CoRR (2024)
Zheng Ning, Brianna L. Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, Toby Jia-Jun Li
SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers. CHI (2024)
Zheng Ning, Zheng Zhang, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, Toby Jia-Jun Li
MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos. Creativity & Cognition (2024)
Ningzhi Tang, Meng Chen, Zheng Ning, Aakash Bansal, Yu Huang, Collin McMillan, Toby Jia-Jun Li
A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actions. CoRR (2024)
Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, Toby Jia-Jun Li
AI Assistance for UX: A Literature Review Through Human-Centered AI. CoRR (2024)
Yuwen Lu, Ziang Tong, Qinyi Zhao, Yewon Oh, Bryan Wang, Toby Jia-Jun Li
Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows. CoRR (2024)
Zheng Ning, Brianna Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, Toby Jia-Jun Li
SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers. CoRR (2024)
Ruyuan Wan, Simret Araya Gebreegziabher, Toby Jia-Jun Li, Karla Badillo-Urquiola
CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents. Creativity & Cognition (2024)
Chaoran Chen, Weijun Li, Wenxin Song, Yanfang Ye, Yaxing Yao, Toby Jia-Jun Li
An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors. CHI (2024)
Ningzhi Tang, Junwen An, Meng Chen, Aakash Bansal, Yu Huang, Collin McMillan, Toby Jia-Jun Li
CodeGRITS: A Research Toolkit for Developer Behavior and Eye Tracking in IDE. ICSE Companion (2024)
Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, Haijun Xia
Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. CHI (2024)
Yue Jiang, Yuwen Lu, Tiffany Knearem, Clara E. Kliman-Silver, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, Wolfgang Stuerzlinger
Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces. CHI Extended Abstracts (2024)
Ruyuan Wan, Simret Araya Gebreegziabher, Toby Jia-Jun Li, Karla A. Badillo-Urquiola
CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents. CoRR (2024)
Yuwen Lu, Chao Zhang, Yuewen Yang, Yaxing Yao, Toby Jia-Jun Li
From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX. Proc. ACM Hum. Comput. Interact. 8 (CSCW1) (2024)
Zheng Ning, Zheng Zhang, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, Toby Jia-Jun Li
MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos. CoRR (2024)
Zheng Zhang, Zheng Ning, Chenliang Xu, Yapeng Tian, Toby Jia-Jun Li
PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data. CoRR (2023)
Jie Gao, Junming Cao, ShunYi Yeo, Kenny Tsu Wei Choo, Zheng Zhang, Toby Jia-Jun Li, Shengdong Zhao, Simon Tangi Perrault
Impact of Human-AI Interaction on User Trust and Reliance in AI-Assisted Qualitative Coding. CoRR (2023)
Yuwen Lu, Chao Zhang, Yuewen Yang, Yaxing Yao, Toby Jia-Jun Li
From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX. CoRR (2023)
Zheng Ning, Zheng Zhang, Tianyi Sun, Yuan Tian, Tianyi Zhang, Toby Jia-Jun Li
An Empirical Study of Model Errors and User Error Discovery and Repair Strategies in Natural Language Database Queries. IUI (2023)
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu
Empowering LLM to use Smartphone for Intelligent Task Automation. CoRR (2023)
Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yun Wang, Peng Zhang, Tun Lu, Toby Jia-Jun Li, Dakuo Wang
Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks. CoRR (2023)
Yue Jiang, Yuwen Lu, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, Wolfgang Stuerzlinger
The Future of Computational Approaches for Understanding and Adapting User Interfaces. CHI Extended Abstracts (2023)
Gionnieve Lim, Hyunwoo Kim, Yoonseo Choi, Toby Jia-Jun Li, Chinmay Kulkarni, Hariharan Subramonyam, Joseph Seering, Michael S. Bernstein, Amy X. Zhang, Elena L. Glassman, Simon T. Perrault, Juho Kim
Designing for AI-Powered Social Computing Systems. CSCW Companion (2023)
Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, Toby Jia-Jun Li
VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. UIST (2023)
Hong Shen, Tianshi Li, Toby Jia-Jun Li, Joon Sung Park, Diyi Yang
Shaping the Emerging Norms of Using Large Language Models in Social Computing Research. CoRR (2023)
Simret Araya Gebreegziabher, Zheng Zhang, Xiaohang Tang, Yihao Meng, Elena L. Glassman, Toby Jia-Jun Li
PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis. CHI (2023)
Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, Toby Jia-Jun Li
VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. CoRR (2023)
Yuan Tian, Toby Jia-Jun Li, Jonathan K. Kummerfeld, Tianyi Zhang
Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations. CoRR (2023)
Yuan Tian, Zheng Zhang, Zheng Ning, Toby Jia-Jun Li, Jonathan K. Kummerfeld, Tianyi Zhang
Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations. EMNLP (2023)
Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, Simon Tangi Perrault
CollabCoder: A GPT-Powered Workflow for Collaborative Qualitative Analysis. CoRR (2023)
Yuwen Lu, Ziang Tong, Qinyi Zhao, Chengzhi Zhang, Toby Jia-Jun Li
UI Layout Generation with LLMs Guided by UI Grammar. CoRR (2023)
Chaoran Chen, Weijun Li, Wenxin Song, Yanfang Ye, Yaxing Yao, Toby Jia-Jun Li
An Empathy-Based Sandbox Approach to Bridge Attitudes, Goals, Knowledge, and Behaviors in the Privacy Paradox. CoRR (2023)
Aakash Bansal, Chia-Yi Su, Zachary Karas, Yifan Zhang, Yu Huang, Toby Jia-Jun Li, Collin McMillan
Modeling Programmer Attention as Scanpath Prediction. ASE (2023)
Zheng Zhang, Zheng Ning, Chenliang Xu, Yapeng Tian, Toby Jia-Jun Li
PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data. UIST (2023)
Hong Shen, Tianshi Li, Toby Jia-Jun Li, Joon Sung Park, Diyi Yang
Shaping the Emerging Norms of Using Large Language Models in Social Computing Research. CSCW Companion (2023)
Aakash Bansal, Chia-Yi Su, Zachary Karas, Yifan Zhang, Yu Huang, Toby Jia-Jun Li, Collin McMillan
Modeling Programmer Attention as Scanpath Prediction. CoRR (2023)
Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, Haijun Xia
Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. CoRR (2023)
Jie Gao, Yuchen Guo, Toby Jia-Jun Li, Simon Tangi Perrault
CollabCoder: A GPT-Powered WorkFlow for Collaborative Qualitative Analysis. CSCW Companion (2023)
Nitesh Goyal, Sungsoo Ray Hong, Regan L. Mandryk, Toby Jia-Jun Li, Kurt Luther, Dakuo Wang
SHAI 2023: Workshop on Designing for Safety in Human-AI Interactions. IUI Companion (2023)
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, Mark Warschauer
Fantastic Questions and Where to Find Them: FairytaleQA - An Authentic Dataset for Narrative Comprehension. ACL (1) (2022)
Yuwen Lu, Chengzhi Zhang, Iris Zhang, Toby Jia-Jun Li
Bridging the Gap Between UX Practitioners' Work Practices and AI-Enabled Design Support Tools. CHI Extended Abstracts (2022)
Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, Toby Jia-Jun Li
StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. CHI (2022)
Shao Zhang, Yuting Jia, Hui Xu, Dakuo Wang, Toby Jia-Jun Li, Ying Wen, Xinbing Wang, Chenghu Zhou
KnowledgeShovel: An AI-in-the-Loop Document Annotation System for Scientific Knowledge Base Construction. CoRR (2022)
Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, Ray LC
AI as an Active Writer: Interaction Strategies with Generated Text in Human-AI Collaborative Fiction Writing 56-65. IUI Workshops (2022)
Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, Toby Jia-Jun Li
StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. CoRR (2022)
Toby Jia-Jun Li, Yuwen Lu, Jaylexia Clark, Meng Chen, Victor V. Cox, Meng Jiang, Yang Yang, Tamara Kay, Danielle Wood, Jay Brockman
A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality. CoRR (2022)
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, Mark Warschauer
Fantastic Questions and Where to Find Them: FairytaleQA - An Authentic Dataset for Narrative Comprehension. CoRR (2022)
Toby Jia-Jun Li, Yuwen Lu, Jaylexia Clark, Meng Chen, Victor V. Cox, Meng Jiang, Yang Yang, Tamara Kay, Danielle Wood, Jay Brockman
A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality. CHIWORK (2022)
Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, Toby Jia-Jun Li
Computational Approaches for Understanding, Generating, and Adapting User Interfaces. CHI Extended Abstracts (2022)
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Mo Yu, Ying Xu
It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books. ACL (1) (2022)
Toby Jia-Jun Li, Brad A. Myers
A Need-finding Study for Understanding Text Entry in SmartphoneApp Usage. CoRR (2021)
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, Brad A. Myers
Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. CHI (2021)
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Tran Hoang, Branda Sun, Toby Jia-Jun Li, Mo Yu, Ying Xu
It is AI's Turn to Ask Human a Question: Question and Answer Pair Generation for Children Storybooks in FairytaleQA Dataset. CoRR (2021)
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, Brad A. Myers
Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. CoRR (2021)
Toby Jia-Jun Li, Tom M. Mitchell, Brad A. Myers
Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations. ACL (demo) (2020)
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, Brad A. Myers
Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems. CoRR (2020)
Toby Jia-Jun Li
Multi-Modal Interactive Task Learning from Demonstrations and Natural Language Instructions. UIST (Adjunct Volume) (2020)
Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, Brad A. Myers
Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs. UIST (2020)
Ritam Jyoti Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, Xiang 'Anthony' Chen
Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications. UIST (2020)
Toby Jia-Jun Li, Jingya Chen, Tom M. Mitchell, Brad A. Myers
Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents. CoRR (2020)
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, Brad A. Myers
Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems. Proc. ACM Hum. Comput. Interact. 4 (CSCW) (2020)
Ritam Jyoti Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, Xiang 'Anthony' Chen
Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications. CoRR (2020)
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, Brad A. Myers
PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. CoRR (2019)
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, Brad A. Myers
PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. UIST (2019)
Isaac L. Johnson, Allen Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, Brent J. Hecht
Not at Home on the Range: Peer Production and the Urban/Rural Divide. CoRR (2019)
Toby Jia-Jun Li
End user programing of intelligent agents using demonstrations and natural language instructions. IUI Companion (2019)
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, Brad A. Myers
APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions. VL/HCC (2018)
Toby Jia-Jun Li, Oriana Riva
Kite: Building Conversational Bots from Mobile Apps. MobiSys (2018)
Marissa Radensky, Toby Jia-Jun Li, Brad A. Myers
How End Users Express Conditionals in Programming by Demonstration for Mobile Apps. VL/HCC (2018)
Toby Jia-Jun Li, Amos Azaria, Brad A. Myers
SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. CHI (2017)
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, Brad A. Myers
Programming IoT Devices by Demonstration Using Mobile Apps. IS-EUD (2017)
Toby Jia-Jun Li
End user mobile task automation using multimodal programming by demonstration. VL/HCC (2017)
Yuanchun Li, Fanglin Chen, Toby Jia-Jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, Jason I. Hong
PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1 (3) (2017)
Isaac L. Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, Brent J. Hecht
Not at Home on the Range: Peer Production and the Urban/Rural Divide. CHI (2016)
Shilad Sen, Toby Jia-Jun Li, WikiBrain Team, Brent J. Hecht
WikiBrain: Democratizing computation on Wikipedia. OpenSym (2014)
Toby Jia-Jun Li, Shilad Sen, Brent J. Hecht
Leveraging advances in natural language processing to better understand Tobler's first law of geography. SIGSPATIAL/GIS (2014)