Building infrastructure to transform real mobile behavior into data.

Interaction Mining Data-Driven Design Group, Apple Inc.

ResearchEngineerDesign

Screenshot of Interaction Mining project

How do you organically capture how users interact with their phones?

Moreover, how can we build tools that leverage this data to help designers and developers create better mobile experiences?

This was a question that my research group tackled through the development of ODIM, which started as an interaction mining system for Android devices and ended as a operating system agnostic platform for capturing, anonymizing, anontating, and viewing mobile interaction data simply named Interaction Mining.

Stakeholders

Professionals such as designers and ML practitioners or even individuals have stake in recording and understanding how users interact with their phones:

The ML Practitioner: This user is interested in using the data to train machine learning models that can predict user behavior and improve the performance of mobile applications. They can access large-scale datasets, filter and search for specific interactions, and export data for training models.
The Researcher: This user is interested in analyzing the data to gain insights into user behavior and inform design decisions. They can run small-scale studies by creating cohorts of users, filtering and searching for specific interactions, and exporting data for further analysis.
The Individual: This user is interested in automating their personal interactions and creating custom automations. They can upload their own flows of interactions, annotate them with labels and descriptions, and export them for use in automation tools.

Project Overview

From the start of the project, I worked on:

Development of the Android data collection app
Design & implementation of the web platform
Managing deployment and cloud infrastructure

App Development

This was the first and original basis of the project. We wanted to novely record user interaction data without required specialized devices or lab instrumentation.

Think about how you send reels to your friends on Instagram. You probably scroll for a bit, find a interesting/funny reel, and then send it to your friends. Maybe they'll read it and you guys will message a bit.

This is a very common interaction, but it's not something that would be easily captured in a normal study. In lab setting, you would either have sockpuppet accounts that don't have real friends or you'd have users be forced to trust the researchers with their real accounts, which is a big ask. It just doesn't feel the same.

We want to build something that captures organic interactions in the wild.

To record granular data while maintaing natural interactions, we wrote an Android app in Kotlin that tapped into the Android AccessibilityService to listen for touch events and capture screenshots and UI hierarchy dumps. This allowed us to capture not only the raw touch data, but also the context in which the touch occurred without requiring any special instrumentation or custom devices.

My contributions to the app mainly focused around adapting the AccessbilityService for our needs, as well as building out the interfaces for data anonymization.

Web Platform

Originally, we wanted to build a simple web app to browse the data collected by the Android app. However, as the project evolved, we realized that there was demand for a more robust universal platform that could support data from multiple sources and provide more advanced features for data anonymization, annotation, and analysis. This led us to build a full-fledged web platform with primary support for both Android and iOS data, with the ability to support other data sources in the future.

We built the web platform using Next.js and TypeScript, with a focus on creating a user-friendly interface for two main personas:

The Contributer: This user is responsible for uploading, anonymizing, and anontating their data on the platform. They can upload their recorded traces, redact sensitive information, annontate the screens with labels and action descriptions, and manage their account.
The Viewer: This user is interested in browsing and searching the data on the platform. They can view screenshots and UI hierarchies, filter and search for specific interactions, and export data for further usage/analysis.

My contributions to the app mainly broadly covered both personas and tasks.

Contributer Interfaces

I led a team who designed, implemented, and interated on UI for data upload, anonymization, and annotation. This included building a frame splitting interface for screen recordings, a robust but easy-to-use and refined interfaces for redacting sensitive information from screens, and interfaces for annotating screens with touch interactions, labels, and descriptions. I also collaboratively implemented user account management features, such as registration, login, and data management.

Viewer Interfaces

I designed and implemented the interfaces for browsing and searching the data on the platform, with a focus on providing a user-friendly experience for both ML practitioners and researchers. This included features such as filtering and searching for specific interactions, viewing screenshots and UI hierarchies, and data export functionality.

Deployment & Cloud Infrastructure

Even as a research project, Interaction Mining required a robust and scalable cloud infrastructure to support data storage, processing, and serving the actual web platform. I co-led the design and implementation of the cloud infrastructure, which was built on top of AWS services such as S3 and MongoDB for trace data and app-level/metadata storage, Lambda and Elemental for serverless media processing, and Amplify for hosting the web platform.

In addition to the core infrastructure, we also implemented features such as CI/CD pipelines via GitHub and monitoring/observability tools including Cloudflare, Google Analytics, and Amazon CloudWatch.

TL;DR

I built a platform for capturing, anonymizing, annotating, and viewing real mobile interaction data. This included an Android app for data collection, a Next.js web platform for data management and browsing, and a robust cloud infrastructure to support the entire system.

It was a large, intriguing project to work on, and I'm excited to see how it can be used to inform the automation and design of user interface experiences in the future!