Hemanshu Bhargav's Portfolio

Problem

Live captioning is very cognitively demanding-- self-reported measures found captioning to be more stressful than surgery

Automatic Speech Recognition has already replaced captioners in online meetings and post-production events

Broadcasting agencies and governments are not prepared for this inevitable shift

Live captioning is intense-- can you spot the multiple mistakes made by professional captioners?

Methods

Applied knowledge from psychology, cognitive science, computer science and industrial engineering to conduct a study where captioners supervised AI to write captions rather than working alone

Mixed-methods approach with 21 captioners using NASA-TLX, System Usability Scale, Nielsen-Norman Satisfaction Scale, Trust in Automation Scale and qualitative interviews

Time-series-- 3 sessions spaced 24 hours apart to allow for learning

Controlled environment and pre-screened for familiarity

Two independent groups: Novices and Experts-- controlled for skill level

Research Methods — Used a cross-disciplinary approach to develop measurements for 14 constructs in the Human Autonomy System Oversight model (which until now was theoretical).

Findings

Mitigating fear is about expectation management

Captioners desire more control over AI

AI reduced physical and mental workload over just 3 sessions spaced 24 hours apart

Findings — Saw a statistically significant reduction in mental workload, which led me to test and discover relationships not present in the original model.

Recommendations

Human benefit from control over AI

Even if AI can automate the job, manual control should be relegated

Scheduling is one recommended solution

Another is interface based (see Design Mock-up from the side menu)

Recommendations — Based on rigorous analysis, I suggested a model where control and automation is adaptive to user needs. This has far-reaching implications for human use over AI.

Impact

Led the only study on captioners supervising AI in the world

Discovered new relationships between Human-AI constructs and made novel contributions

Suggested new directions of research for human automation research.

Impact — I discovered three new relationships which better explain Human-AI Interaction contrary to theory.

Presentation

The culmination of my work was presented to key stakeholders, including the CEOs of 3PlayMedia and ScribeWire, representatives from broadcasters and providers (i.e Rogers) and advocacy groups

Successfully defended at Toronto Metropolitan University