Capturing the work behind expertise

At a glance

A screen recording shows what happened, but it does not necessarily identify which control someone used or what they meant to enter. This paper introduces DesktopTrace, a collector for turning human-computer interactions into training records that connect the screen to named controls and meaningful actions.

The work examines how application selection, accessibility information and reconciliation of typed text can improve the quality of those records. It also explains why privacy decisions matter when the record is created, rather than only after capture. The resulting account is useful for teams preparing expert demonstrations for computer-use agents: it makes the quality of the action label, and the limits of the captured data, visible before model training begins.

What the paper covers

1. 01

   Named interface controls make recorded actions more useful than coordinates alone.

2. 02

   Application selection determines which actions create records at capture time.

3. 03

   Screenshots and ordinary text fields still require careful privacy review before sharing.