For Fusion Arts I wanted to do something simple to use and a little bit magical. You point your phone at a building, and a whale and a shark swim across the facade above your head. No app to download, no instructions, no faff. Scan a QR code, allow the camera, look up.
What I set out to do
The brief in my own head was short:
Web only, so anyone at the event could join in seconds
Recognise the building facade and pin the animation to it
Transparent video so the creatures sit in the scene, not in a black box
Work across the phones people actually carry, not just my test device
That last point is where most WebAR projects quietly fall over, and this one was no exception.
Using Claude Code to find out what was possible
I came into this knowing the animation side cold and the WebAR side barely at all. So I used Claude Code as a way to survey the landscape before committing to anything.
We talked through the realistic options for browser based tracking and landed on A-Frame 1.5.0 with MindAR 1.2.5 for image tracking. Image tracking was the right call for an event: a printed or built facade is a stable, well lit target, and I did not need full SLAM or object recognition for this job. Worth knowing for later projects that those are a separate, heavier undertaking.
Working this way meant I could ask "what are my options here" and get a straight answer with the trade-offs, rather than losing a weekend reading forum threads. The stack it pointed me at is deliberately boring in the best sense. Vanilla JS and HTML, no build step, deployed straight to GitHub Pages.
A Mockup of what it will look like through the App.
Building it
The core loop was fast. Compile the facade photo into a target.mind file, drop a video plane into the A-Frame scene, position it above the tracked image, and you have creatures floating over a building.
The interesting problem was transparency. WebM with an alpha channel renders beautifully on Android in Chrome and Brave, using a small custom GLSL shader to read the alpha. iOS refuses to play that game. Safari does not do WebM alpha, full stop.
So the app carries two videos and picks at runtime:
Android → house_of_creativity.webm, native WebM alpha, webm-alpha shader
iOS → house_of_creativity_luma.mp4, a side by side luma matte, luma-matte shader
A bit of user agent sniffing decides which path you get. Not elegant, but honest, and it means the same URL just works whichever phone comes out of the pocket.
A few smaller touches made it feel finished. The camera and tracking pause on visibilitychange so it is not cooking the battery while sitting in someone's pocket, and there is a debug overlay I can flip on with a single flag when a device misbehaves.
Along the way I hit the usual WebAR gremlins. mix-blend-mode looked promising for compositing and then hung the loading screen, so that went in the bin. The version number lives in a VERSION string on screen so I always know exactly which build is live on the device in my hand before I trust what I am seeing. When you are testing on real hardware over and over, that little green label saves a lot of confusion.
The outcome
The short version: on Android it does exactly what I pictured, creatures swimming over the facade with clean transparency. iOS is the ongoing puzzle and the reason there are two video pipelines rather than one.
What I am taking forward
WebAR is genuinely event ready now, no app required, and that changes what I can offer clients
iOS and WebM alpha do not mix, plan for a separate luma matte from the start
Image tracking is the pragmatic choice for a fixed target, SLAM is a different project
A no build step, GitHub Pages workflow keeps the whole thing fast to iterate and easy to hand over
If you want a building to do something it should not, you know where to find me.
