Physicalization of AR/XR

Organization: Carnegie Mellon University
School of Design
Role: Undergraduate thesis
Research, design, fabrication
Built with: Unity · Cinema 4D
Arduino · ESP32 · Raspberry Pi
Laser cutting · 3D printing
Duration: 2022

In 2021 most AR sat on top of the world: overlays floating above surfaces, anchored to objects but never really part of them. I wanted to know what it would take to physicalize it — to push AR into the material itself.

I started by designing a complete AR experience. The first prototype fell apart on tracking stability, lighting, and alignment. The mechanics were the problem, not the story I was telling with them, so I stopped and built six physical tools instead — one per mechanic, each one built to show what that mechanic needs from the physical world.

All six pointed at the same thing. Surface finish, form, and contrast determine whether the digital layer holds, so you have to pick them deliberately.

This was research and prototyping, built by hand, not a product that shipped.

Six tools

Isometric diagram of all six prototyping tools — The six tools — image mapping, 3D masking, shader effects, and two sensor inputs

01Single image mappingSurface finish decides whether tracking holds. Matte print worked; reflective and low-contrast surfaces dropped out.

023D maskingThe camera hides a physical object and puts a virtual interior in its place. Scale and form decide whether the illusion works; a few millimetres off and it doesn't.

03Multiple image mappingSeveral targets sharing one frame. Spacing, target size, and angle change how coherent the virtual layer feels across a layout.

04Shader effectsA glass refraction effect on the live camera feed. The surface behaves as though it has optical properties, so the camera reads it differently without any change to the geometry.

05Light sensorA light sensor on Arduino drives a virtual lamp over Bluetooth. Cover the sensor and the lamp responds, so the daylight in the room is doing something.

06PotentiometerA dial mapped to the speed of a virtual fan through the same stack. You turn it and the parameter moves, with nothing in between.

What I did

Reframed the thesis: Switched from making a single AR experience to building tools for designing AR, once the first prototype showed where the real limits were.
Six prototyping tools: One per mechanic: image mapping, 3D masking, shader effects, and sensor input. Each isolates a single behaviour so it can be tested on its own.
Physical fabrication: Designed and made the objects: laser-cut and 3D-printed forms, printed image targets, and housings built to hold a specific tracking condition.
Hardware and software: Unity and Cinema 4D for the digital layer; Arduino, ESP32, and Raspberry Pi over Bluetooth for the sensor tools; laser cutting and 3D printing for the forms.
Testing method: Surfaced problems that usually only turn up during implementation: tracking loss on reflective surfaces, occlusion breaking when things are misaligned, drift as the light changes.
The argument: Argued that AR embedded in physical material behaves differently from AR layered over it, and that you can design for the difference.

Making them

Testing materials as tracking anchors

Every finish is a different tracking target. I laser-etched the same pattern into woods, papers, and card stocks and tested each one as an anchor. Matte held. Gloss and low contrast dropped out.

Choosing a material is choosing how reliably the digital layer will hold.

A range of laser-etched material finishes tested as image tracking anchors — Finishes tested as image tracking anchors

Form, iterated until the illusion held

For 3D masking the physical object has to match its virtual replacement closely enough that nobody notices the swap. It took twelve cube variants, modelled in Cinema 4D and then cut or printed, before the alignment held.

Twelve cube form iterations rendered in 3D — Form iterations, rendered and fabricated

Cube prototype iterations including 3D printed forms — Form iterations, rendered and fabricated

Modelled and made in parallel

Each form existed twice: once in Unity and Cinema 4D as the thing the camera would render, once on the bench as the thing it had to replace. Building both at once is how I found the mismatches.

A rendered virtual office interior — the content the camera swaps in during 3D masking — The virtual interior the camera swaps in

Wooden cube forms and laser-cut image mapping cards laid out on a cutting mat — Targets and formsThe card set that came out of material testing, laid out beside the cube forms they anchor.

Assembled wooden cube cluster on a cutting mat — Assembled clusterMultiple targets sharing one frame, to test whether the virtual layer holds together across a layout.

All six tools with Arduino components and image mapping materials — The finished set — forms, targets, sensors, and Arduino components

What it found

Across all six, the physical decisions drove the digital behaviour. Tracking dropped when surfaces were too reflective. Occlusion broke when the geometry was off by millimetres. The sensor tools only felt direct when nothing sat between the gesture and the parameter.

None of this is easy to judge at design time, because you can't see it until something is built. The tools were an attempt to find out sooner.

I set out to build an AR experience and ended up building instruments for designing them. That turned out to be the more useful thing to have made.

A virtual interior rendered at architectural scale in a field — 3D masking pushed to architectural scale outdoors

Out of the studio

Outside, the tools stopped being desk objects. The masking mechanic that hides a wooden cube on a cutting mat can also hold a room-sized interior in a field. The failure modes don't go away at that scale, they just follow the light instead.

3D masking tested at outdoor scale — Scale and lighting tests in the field

A dark interior scene seen through the physical box outdoors — Scale and lighting tests in the field