Human I/O: A Unified Approach to Detecting Situational Impairments with Large Language Models

Copyright 2024 Google LLC

Human I/O is a research project that aims to propose a unified approach to detecting a wide range of Situationally Induced Impairments and Disabilities (SIIDs) by predicting the availability of human input/output channels. Leveraging egocentric vision, multimodal sensing and reasoning with large language models.

Please cite Human I/O as follows if you find it useful in your projects:

@inproceedings{Liu2024Human,
  title = {{Human I/O: Towards a Unified Approach to Detecting Situational Impairments}},
  author = {Liu, Xingyu Bruce and Li, Jiahao Nick and Kim, David and Chen, Xiang `Anthony' and Du, Ruofei},
  booktitle = {Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems},
  year = {2024},
  publisher = {ACM},
  numpages = {18},
  series = {CHI},
  doi = {10.1145/3613904.3642065},
}

This codebase is no longer under active maintenance. The follow-up work of this is Sensible Agent:

@inproceedings{Lee2025Sensible,
  title = {{Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agent}},
  author = {Lee, Geonsun and Xia, Min and Numan, Nels and Qian, Xun and Li, David and Chen, Yanhe and Kulshrestha, Achin and Chatterjee, Ishan and Zhang, Yinda and Manocha, Dinesh and Kim, David and Du, Ruofei},
  booktitle = {Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology},
  year = {2025},
  publisher = {ACM},
  numpages = {22},
  series = {UIST},
  doi = {10.1145/3746059.3747748},
}

We encourage future contributors to XR Blocks for a cross-platform, open-source framework for AI + XR innovations. Example demos of Gemini in XR Blocks include:

• Gemini Icebreakers: https://xrblocks.github.io/docs/samples/Gemini-Icebreakers/
• XR Poet: https://xrblocks.github.io/docs/samples/XR-Poet/
• XR Object: https://xrblocks.github.io/docs/samples/Gemini-XRObject/

API Keys

To run Human I/O, set the OPENAI_API_KEY environment variable before starting the server (e.g. export OPENAI_API_KEY=sk-...). The browser never sees the key; all OpenAI calls are proxied through the /openaiProxy route in server.js.

In addition, you need to place your Google Cloud Vision AI credentials (key.pem and cert.pem) in the folder.

Getting Started

Test and run locally on a browser

npm init
npm install express
npm start

View localhost web app on mobile devices

Download and install ngrok on your computer. Start ngrok and expose the port that your web app is running on. For example, if your web app is running on port 8000, type ngrok http 8000 in the Terminal. Open the browser on your Android phone and type in the URL that ngrok provides. For example, https://randomstring.ngrok.io/myapp.

Using Human I/O

You can use Human I/O, you can either use it with your webcam. You can select the webcam device from the cameraSelect dropdown.

In addition, you can run Human I/O on local videos. Put your video (.mp4 format) into the video folder, enter the video filename (without .mp4), and click load.

Blog Post

Human I/O: Detecting situational impairments with large language models

Notice

This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.