From wearables and smart homes to gaze interaction and learning, these projects use multimodal sensing and time-series learning to understand human behavior in context. They also explore how AI agents and interactive systems can turn that understanding into adaptive support that evolves with people.
Rondo discovers reusable local units and recurring compositions in unlabeled time series. By combining representation adaptation with variable-length motif modeling, it captures temporal structure at multiple scales and updates its discoveries as new data arrives.
TRACE combines ambient and wearable sensing with personalized routine information and LLM-based temporal reasoning to understand everyday activities in smart homes. It uses broader context to resolve ambiguous observations and produce more temporally coherent activity predictions.
EduGage combines physiological and behavioral sensing to estimate momentary engagement during self-guided video learning. It provides engagement modeling methods and a synchronized multimodal dataset to support research on fine-grained learner understanding and adaptive learning systems.
LLMartini helps users compare and combine responses from multiple language models in a shared interactive workspace. It aligns related content, highlights differences, and supports user-controlled composition to bring together the complementary strengths of different models.
This project develops diabetic foot ulcer detection and segmentation methods using naturalistic images from people of color living with diabetes. It addresses underrepresentation in existing datasets and advances more equitable recognition under real-world imaging conditions.
CoReason enables domain experts to inspect and refine AI reasoning for construction safety assessment. Through editable reasoning workflows and visual grounding corrections, it incorporates expert knowledge to improve hazard identification over time.
GenieWizard helps developers discover missing features in multimodal applications by simulating interactions from diverse user personas. It connects natural-language requests to missing APIs through LLM-based semantic parsing and program analysis, making feature discovery actionable within the development workflow.
COMETIC improves smartphone eye tracking by collecting implicit calibration data during cursor-based interaction. It identifies reliable cursor positions as gaze proxies and updates the eye-tracking model during use, reducing the need for dedicated calibration tasks.
CalibRead calibrates eye tracking through natural reading by modeling the relationship between gaze patterns and text structure. It embeds calibration into an everyday activity, reducing setup effort and interruptions.
AwareAuto translates contextual, multimodal user expressions into complex smart-home automations through LLM-based reasoning. It supports conditional branches, temporal constraints, and dynamic parameters while allowing users to review and refine generated rules.
CasualGaze enables object selection through natural, casual glances rather than sustained precise fixation. It models the spatial and temporal characteristics of gaze behavior to infer intended targets and support efficient, comfortable gaze-based interaction.
ReactGenie is a development framework that translates combined voice-and-touch commands into executable programs. By resolving contextual references and composing application APIs, it enables rich, multi-step interactions while simplifying multimodal application development.
WhisperBuds enables real-time whispered speech input using commercial earbuds, allowing users to enter text quietly in shared spaces. It combines multichannel neural noise cancellation with a fine-tuned speech recognition model to deliver fast, accurate transcription while minimizing disturbance to others.
This project explores how people configure and test automations while interacting with the smart-home environment itself. It identifies situated programming workflows and introduces snapshot-based configuration and simulation-based testing to make automation creation more natural.
SonarWatch combines ultrasound and motion sensing to recognize gestures and nearby actions around a smartwatch. By using sensors available on the device, it expands wearable interaction and behavior sensing beyond the small touchscreen.
ConeSpeech enables users to direct their speech toward selected listeners in shared virtual environments. Adjustable, cone-shaped delivery areas support flexible conversations with individuals or groups while reducing interruptions and overhearing by nearby participants.
FaceSight enables hand-to-face gesture interaction on AR glasses using a compact, downward-facing infrared camera. By recognizing hand–face contact, fingertip movements, and nose deformation, it supports rich gesture input and continuous control around the user’s face.
HeadCross enables hands-free target selection in augmented and virtual reality through deliberate head movements across target boundaries. Its recognition method distinguishes intentional selection gestures from ordinary head motion, supporting interaction when users’ hands are occupied.
PrivateTalk activates voice input through a natural hand-on-mouth gesture, detected by comparing audio signals from two Bluetooth earphones. The gesture reduces speech exposure and conceals lip movements while enabling convenient voice interaction without wake words or button presses.