Hololens
I was asked to join Microsoft to work on a highly secretive project without a single hint as to what it was other than my background would be useful to the team. Fortunately I took the risk and joined the Hololens team and was blessed to find an incredibly talented Design team and UX Researchers that taught me to think from a UX perspective and set a high bar for what the UX Research and Design workflow could be.
My role as a Creative Technologist was to work with the Design team to come up with concepts that Distinguished Designer Nando Costa would lend his creative vision to and fully flesh out into beautiful works of art that we would then have motion graphics designers bring to life, 3d artists that would convert concepts into design assets and would pass on to me to make the interactions and design into actual working prototypes made in Unity and running on prototype hardware. I also worked alongside the system architect to see how the designs would fit into the overall system design and Operating System that at the time was a very elaborate Unity prototype.
The majority of the work we did on the Design team was a reference design that we'd feed to Engineering who would optimize and re-architect elements to make the real products that would ship. Working with Engineering regularly to make sure our designs were possible and created in a way that we had a path to launch.
Much of the early Design Explorations and Core AR Principals have since become the standard for Augmented Reality and have been adopted by the industry.
Highlighted Projects
Project
Interaction Explorations
Deliverables
Unity Prototypes Design Assets
Role
Creative Technologist
Collaborators
UX Research UX Designers Motion Graphics Artists 3D Artists Producer
Brief
While at Microsoft I was challenged with coming up with a new set of cursors and ways they would interact with various 2D and 3D content using our head gaze and air tap interaction model. First I started off by taking a look at all the various cursors that exist in Microsoft Windows and their interactions with UI relative to their functionality. Then I went around to all the XFN teams and Prototypers who all had developed their own custom cursors for their various projects and prototypes to learn from their early attempts at working with this new interaction model.
I discovered a few novel approaches to cursors like letting the cursor follow the 3D normals of objects and built off the idea that we want the reticle to remain the same size within Screen Space, but not also be a simple 2D screen space locked cursor. After a bunch of explorations I came up with the approach to vary the size of the cursor and let it "fall off" the edge of 3D objects before returning to it's ideal pixel size by scaling up and down the size of the cursor based on distance.
When we did user testing of the cursor options users found the consistent screen space scale to be the least distracting and even though almost nobody realized the cursor was scaling in World Space, they loved seeing the cursor adjust it's rotation and options based on where it was hitting a 3D object, grab handle or other transition points. There were multiple iterations of UX Research Studies and creative development before we landed on the final designs. Along with the cursor UXR testing we also tested a variety of input gestures as another part of the Interaction Prototypes.
In the end the first Hololens launched with this cursor design and what users found to be the most pleasant and understandable experience that gets out of the way and allows the user to focus on the application or experience they care about most.
Project
Spatial Mapping Explorations
Deliverables
Unity Prototypes UX Design
Role
Creative Technologist
Collaborators
UX Research UX Designers Motion Graphics Artists 3D Artists Producer XFN Prototypers
Brief
One of the big challenges that utilized my previous work with Depth Cameras was the concept of Spatial Mapping to not only scan the real world environment but to also make sense of it and create some core Interaction Principals about how AR content should respond and interact with the real world.
We wanted the ability to know where a table, chair, couch, coffee table, walls, ceiling and floors were at along with their dimensions and if they were "empty" enough to place AR content on top of them dynamically without the user having to specify a particular location.
Early explorations of this developed in the concept that AR content should "slide" along the floor when dragged not able to pass through a known real-world surface. Always focusing on making the AR content feel more grounded and part of the real world. The revolutionary fast and accurate SLAM tracking all done within the Hololens meant that we could finally render World Locked Content in a convincing way. Proper occlusion of AR content behind real world objects takes the immersion a step further and adds to the illusion of Augmented Reality.
Having the world be tracked without image or optical markers was also another groundbreaking concept that led to the path of Spatial Anchoring and later Azure Spatial Anchors. Early explorations revealed the need for more tech development of Spatial Anchoring because the further you are from a known anchor point the more the AR content will seem to jitter or swim away from the real-world environment breaking the illusion of Augmented Reality.
One of the Hololens launch partner studios Asobo developed a high-tech detective crime thriller that utilized and showcases the Spatial Mapping features by showing AR characters walking around your environment, sitting on your couch, places objects on your tables and hiding clues around your room on various known surfaces all generatively creative a truly unique experience for each environment.
