TalkFrom

How to Balance Tech, Content & Location for the Ultimate UX in Mixed Reality

Jason Yim

Recorded at GOTO 2017

[Music]
first thing I am NOT a tech guy so I've
been in a couple of other sessions that
are super technical I'm not that guy but
if you want to email your super
technical questions I will get them
answered but definitely if you have kind
of more campaign general type of
questions or strategy questions please
bring them up through the app and during
the the presentation so a little bit
about my history born in Singapore moved
to Hong Kong
I mean lived in Hong Kong and then
actually went to the Bay Area around San
Francisco for high school and then
eventually UCLA graphic design major
[Music]
started a company right out of school
like a digital agency that was web
focused that became an another agency
that was funded by Hans Zimmer who's a
film composer so we're doing a lot of
finally started trigger about 12 years
ago film marketing agency that we have
been kind of pivoted a couple years back
to kind of focus on mix reality and I'll
give you a little bit of reason why I've
been working with Lego for five years we
actually have a small office out here
and lived in yelling if anyone's been to
yelling for like three months in the
dead of winter so my wife almost
divorced me but anyway sorry first day
our project it was in 2009 and we have
three patents around a our play and this
was a my FaceTime call at 2:45 a.m. a
couple days ago with a jet lag so
trigger with a mixed reality agency we
work on kind of premiere type of brands
and clients a lot of them entertainment
base you'll recognize a few of these I
think our office is in West LA and a
small one in Aarhus but the West LA one
about 10 or 11 engineers and then the
rest is kind of like creative producers
account managers and things like that so
the fun thing about the shop is that
it's very kind of like an indenter is
kind of workshop the neat thing about AR
and moving into AR is like there's a lot
of physical items involved so we work
for a lot of toy companies
so our our office is full of Lego and
then of course all the movie posters and
stuff and then there's always new
devices right now so at any time in and
more in our shop we'll have a couple
holo lenses we have a couple of daiquiri
things and then you know whatever
there's just tables worth of like a our
devices that people are trying and stuff
we have 90,000 hours of mixed reality
development that's a big differentiator
I think for us again because we've been
doing it for so long for all these
clients and these are all paid projects
so there's a lot you learn from instead
of just kind of prototyping and demoing
there's a lot learn from actually
putting it out in the wild and seeing if
it breaks or not so part of the format
today is a we'll go through some of
these recommendations and what we
learned and I'll have some good examples
of what we've done right and then a
bunch of examples of what we didn't do
so right but I think is good good
learning as well showcase development
means that we actually get hired by the
tech companies to build something with
with the new code so view forea we've
been their showcase agency since they
were in R&D and every major feature
we've been the one to kind of premiere
it so sometimes we would start working
with the code a year in advance and then
our clients are a good spread of
entertainment to the tech companies and
then kind of the major brands in between
so when we look at mixed reality we kind
of break down the services on the
spectrum so marketing and promotions is
kind of what we're doing for the film
companies trying to get a lot of
attention to 3ei our you know funny
experiences that create social buzz but
but aren't very lasting product
development is what we're doing with
Lego and Mattel where it's like the
project can be a year and a half two
years long we start with concept thing
actually prototyping that then gets user
tested we've actually done 30 plus
digital physical play prototypes for
Lego and these all get kid test it
around the world so there's a lot you
can learn from that like you know when
kids get fatigued what's boring what's
not boring what can be filled in by that
kids imagination and stuff
location-based is what we're doing for
tell company so something you might see
in store or we have a slide that will
talk about the new lego house the lego
museum we have a full mixed reality
installation in that museum and then
enterprise is what we're doing for Honda
so the enterprise stuff is using AR as a
actual design tool which is super
interesting
so actually trying to solve a both a
budgetary problem for them because it's
really hard for them to kind of evaluate
concept cars like each of these cars if
they're if they're to fabricate a car
for approval it can get close to like a
million bucks to fabricate this thing
like interior next year so we they are
can you get people to decide on design
direction just from a 3d model placed in
you know real world location so so our
approach to mix reality like there's a
lot of talk about technology there's a
soul there's a lot of buzz around AR kit
right now but we feel like tech is only
one one kind of corner of it and then
there's also the content piece and then
of course location so and that makes
like the killer user experience so why I
think this this kind of this UX issue
covers so many different different
places we can apply mix reality like on
the consumer side so you'll see these
same kind of things that are factored in
that we have to worry about what is the
environment what is the content what
what is a target that we're looking at
through what is the device both hardware
and whether it's a software platform on
the device and then really importantly
what is the interaction that we're
asking from from the consumer and that
same thing is repeated on the enterprise
side and then also even on the location
categories of things that we have to
balance out when we create a mixed
reality campaign so they start with tech
[Music]
so this is this kind of shows the both
kind of hardware and software things
that we've been working with in our shop
in the past
let's say a quarter if you go back a
year or just over a year that might be
just euphoria you know we're just on
mobile phones single AR platform and now
things are just changing leaps and
bounds so that is something that you
definitely have to keep on top of and it
can be quite difficult but because our
clients are you know we go from
enterprise on this end and then like a
like movie promotion on the other end
the fact is that we'll end up touching
so many different types of tech and it's
both really rewarding like the the tech
guys are super excited about that super
excited about inventing and
investigating but it's also really
draining and it's really expensive so
it's very hard to we can't like we can
build expertise in mixed reality as a
whole but it's it's hard to just focus
on on one like just build our experience
very next client that we might get the
hololens doesn't doesn't solve their
problem so something too interesting
challenge so a couple of things with all
this kind of energy and buzz around AR
kit which is really cool it's great to
see you know both consumers and
developers getting really excited about
what what can still happen one of the
issues that not a lot of people at least
on the consumer side it's not clear is
that you actually still need an app so
you know AR kid is close to the OS it'll
perform better but the actual content
and the experience is all lives in an
app that you still have to download and
that's not only a technical issue and
what the rest of the slides are going to
show you is also there's a there's a
much bigger issue about just getting
people to download apps so as soon as
you put it into an app I think one of
the recent survey said that over 50
percent
I think of US customers have not
downloaded a new app in the last month
you know so it's a how do you
communicate to the user how do you get
them to want that app and then to
download it and you know so creating the
app is just like this tip of the iceberg
and it's a much bigger challenge just
get that app on a person's phone so one
thing that we do is we one thing we tell
our clients is you know to consider an
ambassador driven experience so that
means I get in an event you can have
people with tablets and stuff so the app
is installed on on the ambassador's
device you can cast it up onto a screen
so people coming back and see what the
they are experience is and that's a
great way to introduce people to AR and
get them to understand what the app is
before they download it and then the
other thing too is consider an AR module
so this is something we just did for
Star Wars but the idea is like if you
have instead of creating a dedicated AR
app which you need to drive all this
traffic to if there's already in an
install base you have an app that's
doing something else can you fit the AR
functionality into that so kind of
piggyback off of the user base that
you've already established picking the
right hardware and software so again
like a arcade and AR core is kind of
just for the later phones so if anyone's
doing an actual broad-based campaign
that even though that's the most
exciting buzzword right now that may not
be the right solution for you there
because it's just not going to reach
enough people and their solutions are
not cross-platform yet so what that
means is you have to build it for iOS
and then you have to port it over to
Android or vice versa and then it's
really interesting because it's like
it's it's funny for that there's
probably a few people in the room that
have been doing AR for a long time and
there's some things that are very
polished about AR kit but there's so
many other features in another
third-party AR platforms that do really
powerful things as well and an AR kid is
getting a lot of attention but hopefully
those features will start being added
into into that as well so and then one
thing that's really interesting is like
the when you're picking the the software
platform of the hardware platform
licensing and all the legal side of that
can can be as important as
tech side so that's kind of frustrating
but it is the it is the reality of the
space out there especially when if it's
you know if you're not doing a are mixed
reality for promotion and you're doing
it for actually a product development
then there's a lot of fighting around
like what is the license fee like if you
have a if you have a product that so you
AR is actually part of that play of the
toy you know what percentage of the of
the price of the toy does the AR company
get and then it's like how do you debate
that like what if I could play with the
toy would that be AR but with the AR
it's ten times better is that different
from if you need a R to play with the
toy so it's stuff that I think that that
people are still in kind of working
through I don't think there's an
industry standard yet so but just
something to keep in mind and then this
idea that like things are changing so
fast in mix reality right now hardware
and software that things may change mid
development so for us a our kit popped
up while you know three or four projects
are still in the middle of getting built
so that's that can be an issue you can
either choose to rip everything out and
then and then rebuild it and try to hit
the same date or push a date or in our
case we're doing things where we'll
still launch the original version with
that AR kid and then we'll do an update
after but it's also in some ways that's
good as well because we had a car
project where because of we we did the
a life-size car that you're looking at
in a parking lot or whatever because of
the field of view limitations on
hololens you actually have to be 60 feet
away from the car to see the entire car
so all those hololens videos online that
I just made completely fake just because
like it
they don't show the actual POV from the
device it's quite limited I mean there's
amazing things about it but the
field-of-view thing just makes it very
difficult for something that's that's
giant sized like that and then of course
now you're tracking and try to have
world lock from 60 feet away which is
also challenging anyways but yeah so the
point what that was like
were they are kit we actually managed to
solve some of these things that that uh
that hololens was giving us trouble with
and we could actually restart a project
ahead that had been parked because a
field of the field of view hardware
issues so this is just say showing one
path of how we decide what hardware
software to use so first question is is
it enterprise or consumer let's go
consumer is it ambassador driven or is
it a public app that's a big question
for us so we go down that path is an
existing app that we can stick the AR
module in or and sometimes that's
difficult even when a company has a big
app because the person that owns that
know the innovation group that you're
working with on the AR side so there's a
lot of departments that don't work
together so that that that can be a
tough challenge but let's say we can't
get that and it has to be a dedicated
app is it a target based a our
experience the target means that like I
look at the go to conference sign and
then something pops out of that or is it
just like a art kit and I can place it
anywhere in the room so let's say it's
no target then the choice is mass
consumer or early adopter if it's mass
generated targets you can have someone
use anything find any image like a
signage or magazine or something snap a
photo of that and that becomes the AR
target or you can use three degrees of
freedom three Dolf kind of floaty AR
pokemon style to try to create an
experience but if you chose early
adopter is it Apple or Android if it's
both then you end up with a Orkut and a
port AR core
so anyways all our projects kind of go
through this this kind of flow the one
thing that we would definitely recommend
is like you know typically when when
developers are first getting into a are
like you're so excited to build an AR
that that's the only thing that the the
app does but there's actually a lot of
other things that you can tie AR into
so think of AR as a really powerful
ingredient and that's the best way to
use it instead of a kind of standalone
thing try to connect it with as many
other useful technologies as possible
the interesting thing about AR we just
did a spider-man homecoming we we had an
app that was both had kind of
traditional type of content had Peter
Parker's phone she Ashley could go on
his phone and look through his voicemail
and look through his texts and stuff but
it also had a bunch of AR features and
we were measuring a new analytics
package to see how content would compare
for the consumer and how sharing would
compare and I think AR is new enough now
and it's also unique enough because it's
really hard well for film companies and
stuff it's you can literally take your
IP and place it into a consumers world
and of course same thing for the
consumer they can pull at their favorite
character and bring it into their room
so of course they're gonna want to take
photos and share that and the
interesting thing was like when we
looked at the we had some very simple
sharing features you know like emojis
and stuff like that there were or gifts
that you could send out and then we had
the AR path for sharing which is quite
difficult you have to for that one there
was no AR kit so you actually had to
create your own target place spidey
posts buddy take a photo and then
sheriff share out from there the the
sharing on the a our side was as much if
not more than on the traditional sort of
content side even though I had all those
extra steps so I think definitely take
take use of that so then the other tech
that that they can get
you know a Ark lives well with is you
know live data makes it contextual
there's AI for conversation and stuff
with characters definitely location
awareness transaction all things to make
it more practical and useful so just
some examples indoor and outdoor mapping
so the idea of like the AR app knowing
where you are both outside and then with
VPS which is a Google technology like
actually being inside a store and either
directing you or giving you content
based off of your position in indoors
real-time lighting and reflection we're
working with some with this now and some
projects where it's just pretty neat
because like when you when the AR object
appears or thing appears it's actually
we have a reflection map applied to it
so it actually looks like it's at that
that specific location we on Honda we've
worked on projects which I'll show you
later that actually have live telemetry
from the car itself so we're trying to
do AR and VR in the in the rear seat of
the car in a moving vehicle so it's
pulling like telemetry from the car as
well as like IMU data and we we had
lighthouses because it was on on vibe
and stuff so merging all that stuff
together audio fingerprint this idea of
being able to sync using audio as a way
to know what you're watching and that
that could be relevant to you know TV
shows but it's also relevant to live
concerts so imagine like you're seeing a
giant concert and you know you want to
drag him to fly around the stage like
how do you sync that for everyone
watching the concert and you can do that
actually through audio fingerprinting
and stuff and then AI is something that
we're excited about trying to bring more
life like reaction to our characters so
a good example on the tech side
[Music]
see so this was for last year's CES as a
prototype and now we're working to
commercialize this for Honda it's a
platform quick dream drive so at CES of
the sleds that could you know attendees
good to go but there was also a live
drive version so the live drive version
show it in a little bit as a bunch of
PCs running in the back it's actually
running vibe in the car and the
challenge was basically as taking the
telemetry from the car your movement
inside the car trying to merge that that
set of data together and then have you
sync your movement in the physical world
with the digital world so actually
that's that's pretty challenging and
pretty complicated to be imagine like if
you're in a car and you turn right but
the how do you figure out where you are
in the virtual environment from from
those two motions conflicting motions
happening at the same time and it also
tied into Honda pay which is a visa
partnership so this idea of being able
to you have all your your credit-card
accounts set in your car and you can
actually do purchases and stuff through
the car itself and those kind of funny
games just to show that it was both you
know P oh I navigational stuff and as
well as a
and entertainment opportunities and the
reason why it's uh why they're doing
that is because the car companies are
looking at with autonomous Drive
especially all this captured caught all
this captured time inside the car you
guys have it a little less here like in
LA it's friggin insane you're just on
the freeway for you know however many
hours a day and at that time is like
captured time how do you deliver content
to those people and in VR is gonna be a
big Content player later how do you
deliver VR and if it's if you try to do
VR in the back seat of the car no no you
ever tried that it's like it's you know
you get super nauseous if it's not
synced so yeah I have tried so lessons
learned so some some examples of where
kind of tech we failed on the tech side
I think
jollibee is a fast food company in the
Philippines we did they're kind of happy
meal a our app it was super when it
launched like you know the the the video
trailer for it got like millions of
views millions of likes like it was they
were super excited about it and the the
app itself was actually a quite a fun
create your own music you could unlock
different characters by going into store
you could even take a photo of your own
face and old map it onto a 3d character
that appeared in AR so there's quite a
you know I we felt a polished app but
we're the actual campaign kind of broke
was because of data issues in the
Philippines so the Wi-Fi wasn't in the
restaurants yet so that's one problem so
you you hear about the app from TV or
online and stuff but the app needed a
needed the Happy Meal box so whatever to
work off of which is what you get at the
restaurant so those two from the point
of like kind of awareness to actually
getting the other piece which is of the
Box there's a there could be quite a lot
of
since between that so the ideal solution
is like when you're actually in the
store and you get the Happy Meal box
that's when you should be able to
download the app in and play that game
so it's asking a lot of consumers to say
download something in advance for your
marketing campaign right but because of
super slow speeds in the Philippines and
also no Wi-Fi in the restaurants this
was kind of a you know it couldn't it it
could have been more successful and it
not going with a with a kind of app
approach probably do something more
mobile web for this particular execution
so moving onto content so this let's see
if there's what can pop up in a are
there's 2d content like images UI your
text and stuff 3d stuff like the
characters objects in the space you can
even do like full environments in AR and
then also effects like all the particle
effects that you you need to make
something look really cool and then you
know on the audio side we have we are in
a project right now where it's a hundred
pages of content ever like price 25
characters all voice-over and custom
music throughout the whole thing like
there's a lot of stuff you can push
through an AR app but something to
consider is the 3d side so 3ds like kind
of like the the biggest content piece
for us a lot of times because it looks
really cool in AR it's cool to see it in
physical space and our clients want
their characters in 3d but it can simply
take a lot surprisingly large amount of
the budget and every project that we've
started there's always a glimmer of hope
that the client will have an asset that
you can actually use and the
conversations always start that way like
oh yeah we have 3d files for blah blah
blah and then once you actually get into
the project our head of tech as a you
know his saying is like we have had
0% passed on that actually working all
the way so typically you know we'll get
a film asset it looks amazing
but it's it's a giant film asset so you
have to how do you get that down to work
on a mobile phone or we have CAD assets
from the car companies and stuff like
that how do you make any of that AR
friendly so a lot of times it's it's
almost like reworking the entire the
entire 3d asset from top to bottom so
there's not only like because it's not
only the the modeling it's like the
rigging all the textures how effects are
applied to it or how those effects are
shown so for instance on Star Wars we
had to completely recreate the hologram
effect that you see on film that they
were doing one way and we have to do it
in a complete different way because it's
a AR hologram effect and then the other
thing that makes 3d super expensive is
that because you don't want to do the
person to see the same animation over
and over again you need variety right so
that can get expensive and then the
other piece that's expensive is
completeness of content so if you
imagine like right now you see a lot of
demos let's say hey I can place the
table in AR wow that's super cool
but if that was an actual functional app
for a client you're gonna realize that
like this manufacturer has 50 tables
right or however many cars this guy has
or this client has like how do you that
becomes a big workflow issue and also a
app size issue as well so and then the
problem is like if you're not complete
with that so let's say I'm gonna do it
for you know I can make the latest line
of Honda cars pop up but I'm only gonna
do the Honda Civic in the Honda Accord
because that's that's what I'm doing
right now how do you communicate that to
the end-user how do you let them know
work or the Odyssey doesn't work there's
a lot of kind of friction from that
because from the consumers point of view
they're gonna expect that if one car
works and they're all going to work and
if it doesn't you there's a lot of work
almost apologized for that right so and
you want to yeah you want to head that
off at the pass you want it to be clear
to consumer how to engage where they are
and when it will work and when it won't
work so a good example on the on the
kind of 3d content side anybody watch
Walking Dead cool so for the new season
we did this a our app that basically you
can unlock all these zombies and it's
you know take photos with them fun stuff
like that the the neat thing about is
actually like um you can we have a
fairly frictionless way of unlocking
zombies so when if you're in the US and
you're watching the show it's playing
these Mountain Dew ads during the
broadcast and the app will actually
listen for those ads so when I here's
the audio fingerprint it knows you're
watching the show and when when that
happens then it will give you unlock a
new zombie that week so there's 20
zombies total that you get to unlock but
to show you kind of where we started
from you know we will get so in Walking
Dead there are no and no they're not a
lot of 3d effects for all the other
zombies so it's all practical stuff so
we have practical photo reference they
have done some scans but these are so
this this isn't a good example of a
client saying hey we have 3d scans of
the zombies you get the 3d scan and it's
just like completely covered in all
these artifacts and it's not really you
know it's good for reference but it's
not you can't just take that and then
put it into your AR experience so we'll
start from this kind of reference and
then we'll kind of model and build
everything from scratch on that so some
lessons learned a couple years back on
spider-man 2 we did a big ar app for
that and we were and we also did a kind
of online game as well and we were super
excited because we got we got to
collaborate with the actual spider-man
so this is a the actual stunt guy that
did all the moves for spider-man I think
he stayed on as the
the main Spidey stunt guy since then as
well but we had him for two days in for
motion capture and Sony paid for all
this so for two days we asked this guy
to crawl around and fight people and do
all these special moves that we were
gonna use in AR app so so the crazy
thing I mean a super it was really fun
doing the shoot and then afterwards we
looked at this crazy mocap data and like
it was basically there are two big
things one was that like this is so old
that that are the AR execution that we
had like Spidey was actually a fairly
small character back then there wasn't a
lot of the AR had to be kind of planted
on top of the target you couldn't move
very far away from it
so anyways he ended up being a small
character and all the mocap data that we
got had to be hand kind of exaggerated
just so you could see it at that kind of
small scale and then the other big issue
was again because there's so much data
coming through from from that mocap
capture that a lot of it wasn't really
usable so it would probably had been
faster if we had just videotaped him
moving around and then just used that as
reference then trying to work backwards
from from the from the mocap data so
location so I think this is sometimes
forgotten too because they you know this
idea of like a are just
one-size-fits-all use it anywhere but a
collision and occlusion really break
believability so a lot of the demos that
you're seeing online you know if you see
someone doing a are in the middle of a
field or in like a park or something
like that it's it's the the chances of
you actually using it that way is pretty
minimal so so if you imagine you take
that same experience and you map it to
you're sitting in the office you're on a
bus or whatever you'll instantly break
because you're your characters will be
overlapping physical environment
the available space issue like what I
was talking about on the car side like
that idea of like oh super cool dude
let's do a are in a car dealership you
know half things show up but it turns
out how far do you have to be away from
the car how much in the showroom how
much how much space do you have between
the cars that a person can fit in and
step back and see all that has to kind
of be figured out and then indoor versus
outdoor there's a whole bunch of
lighting conditions I changed because of
that we've done stuff in theme parks
where the lighting will change a lot
throughout the day because also at night
all these lights come on and do all
these different things so how do you how
do you make something recognizable
through a whole you know 16 hour cycle
for for a place like that and then we've
even have had issues with magnetic
interference and stuff like that and
I'll talk a little bit about that later
but some clients might choose something
like even specific down to table size
had to work in on all their tables so we
actually had to get table sizes from
around the world and build the game to
whatever spec that minimum table size
was which was a interesting challenge
yeah so some of the locations that we've
had to kind of tailor an experience to
but I think once you actually have it
dialed in and tailored into that
experience it's it's much more powerful
you know first was just trying to make
it that they'll fit anywhere so a couple
of examples on the location side I just
want to show this this first is a so
when we first started working with AR
and on mobile device like us being Star
Wars fans one of the first things we did
this is a we had a asset from the game
that we're doing for Lucasfilm was like
a online flash game and we made them
gonna walk around the the posted box and
stuff in AR so this is 2012 and we
basically pitched that every six months
to Lucasfilm for five years something
like that and we finally got it into
into the Star Wars app so so we did it
for force awakens we did four characters
in in in the official Star Wars app for
that
and then for the new for the last Jedi
it's a much bigger campaign so this
campaign is twenty thousand retail
locations so for force Friday it's when
the toys first came available to the
public so though if you imagine the
logistics of getting signage into 20,000
stores around the world in different
languages and then those were all AR
[Music]
it's a trap
Thanks there's 15 total characters there
was like basically you'd go back into
the retail location and for that weekend
every single day it would give new
character so you couldn't see them all
on the first day so it's basically
driving parents and kids back in to buy
more stuff but another another tip is
again like the physical world and
putting augmenting the physical world is
a lot for me it's a lot more interesting
than just placing something into an
empty space so this idea of like 3d
recognition so we've done things where
you can look at a toy and then the Lego
characters are you know walking on top
of the the boat or whatever or the shark
is swimming around it like it's it's
very it's locked onto a physical object
there's a lot of magic in that and then
then the next step where that's all
gonna go is kind of machine learning
visual identity stuff so instead of
right now we have to you know program
into the app we take the CAD model for
let's say a Lego toy and it will now be
recognizable by the app itself but we
had to ingest that CAD model first in
the future I don't know how near future
but with machine learning you hold it up
they will recognize that it is a Lego
toy it'll recognize that it's that theme
or whatever and then the right content
will appear so that that's that's quite
an exciting bleep I think so in terms of
kind of you know physical objects this
is kind of at a at a big scale so for
Star Wars the other part of this
campaign was um
we called landmark AR but basically on
that weekend if you're within I think it
was a mile and half radius of these 20
world landmarks
[Applause]
we'd actually put a a our star destroyer
at the proper scale and the sky and
stuff but I think like that's again like
the landmark was important because it
gave you that sense of scale we could
have set it up so that you could just
point it at the sky anywhere and and see
a star destroyer but there was more of a
event-driven push through this and then
all of that there's so much consumer
messaging that you need to do to again
drive the right kind of usage in the app
it has to be consistent across all this
kind of media so before person downloads
you're trying to explain why they should
download this app you know how to
to play with it and then after download
the problem is like no one ever reads
a big challenge on a our right because
sometimes it's a can be quite a few
steps so they're the ways around that is
you know put a lot of videos on YouTube
showing the right way to do it embed
videos walking people through it or we
are doing a lot more projects that have
some sort of and this is better for a
product that has some sort of guided
play so the UI and the UX will actually
walk them through as they go the first
time through the through the app and
here's an example from force Friday just
to show you know from the social banners
then a good lesson learned in terms of
location we did this for the Eiffel
Tower at the Paris Hotel in Las Vegas so
the iPad experience that when you went
up into the Eiffel Tower you got all
this content pushed to you along the way
so it was pretty complex was like the
first time we asked Anna Klein we're
using beacons so as you're waiting in
line and moving towards it you know
using beacons it will tell you start
pushing content to you there was a are
kind of
targets along the way so you could scan
stuff and get more info and more
experiences as you were kind of going up
this Eiffel Tower experience and then at
the very end when you're at the top we
were gonna give a very functional view
of Las Vegas Strip so you could look
around everywhere and all these all this
pop-up information about you know the
name of the casino like it was gonna be
dynamics or like what were the offers
that were happening at all these kind of
partner casinos and locations and stuff
so on a project like that everything was
we were really focused on the AR &
beacon side of things because like
that's where we felt the risk was on the
development side that's where it could
go wrong the most and then the crazy
thing was the compass didn't work at the
top of the Eiffel Tower because it's
like this Faraday cage because of how
it's a building stuff so we're yeah so
we basically had to you know halfway
through the thing with some negotiation
with the client we had to rework what's
gonna happen at the top so it became
basically just a you know 360 view of
actual Paris and then that that way it
didn't actually have to lock to any
position really you're just gonna you
know you don't know you're looking North
versus South or whatever so it could be
a very loose approximation as you as you
turned around so quickly to finish it
off so on the UX side no one wants a
one-trick pony
make sure like there's a it's something
that's worth downloading
how can it be fun in the first minute
the first hour the first week and that's
usually depth of content depth of
experience and I think the the example
that we're really proud of is actually
at the Lego house Lego Museum that just
opened last week I was just there a
couple days ago we did this thing called
a fish tank or fish designer so
basically
[Music]
super-elegant go up you build your own
fish there was those scanners I don't
know sir you saw in the photo on the
side of each tank but you basically
placed the fish into the scanner and it
digitizes the fish you put an eyeball on
it and then a mouth that indicates both
personality and then direction and then
it gets sucked into this digital tank
and it comes alive and basically you can
have like 300 user-created fish in this
tank and there's all this animation that
happens on a cycle like every minute or
so something big will happen like
they'll feed the fish and then all the
fish react to that so this is a super
simple scanning process if we wanted
this magic moment so why I like this
user experience it's like for the kids
it's they're doing everything that they
want to do they don't you don't want to
build a Lego fish in on an iPad I want
to build it out of bricks you know and
then the scanning moment is like just
super short and sweet like this magic
moment when you actually see it digit
digitized and on the scanner itself it
actually fills with water before it gets
sucked into the sucked into the tube
which is really cool and then on the
down side of things fatigue is a big
issue so you know there's a lot of talk
about like a you can play a game
pointing at an empty table like we've
we've done a lot of tests with that
stuff that kind of play in the past like
it is really fatiguing to hold up a
phone in ar4 for that long especially
for younger players and if you're doing
shopping or whatever like you might only
have one hand so any kind of landscape
versus portrait you should probably go
portrait you know where you place the
buttons and all that sort of stuff
matter but it also affects hardware so
AR is using open camera it gets super
hot so you have like heat issues but
also like battery drain issues so lesson
learned from that a downside we worked
with wet a workshop for many years
actually trying to make this this
boardgame product so wet as from New
Zealand they you know Lord of the Rings
fame and awesome team great
designers so we they were designing a
physical game we're trying to do it an
AR digital companion to it at the same
time basically the the board games gonna
launch just as a physical version we
actually user tested and all these game
conferences
everyone loved all the features that we
were trying to do in AR cuz you could
see the battles you could manage the
game that could give you the rules but
the thing that we couldn't solve was
like if you have four players and the
average play time for these board games
was a short game with 50 minutes so long
games were like you know two hours like
you could not get four people to have
their phones charged for like you know
for that for that gameplay like and then
as soon as one person breaks and they've
lost it then it's broken for all the
entire group so just something to
consider and then on a final note you
can't always plan for everything but
there's still like pleasant surprises
and stuff so this was just something
that our client had sent to show us was
like you know at one point during during
the be opening someone made a flag
instead of making a fish and then
everyone afterwards
started copying that flag and no one had
planned for that right like no one you
know but it was really cool I guess just
it's cool that those those experience
those things can still happen randomly
so anyways thank you
[Applause]
so thank you very much Jason I think we
have time for one or two questions I
there's a question on the status or your
opinion on magically as a vaporware or
too complex expensive to allow to
develop for yeah we quickly
automatically Sosa magic leap is a mixed
reality company out of Florida they have
its supposedly using light-filled
technology which is basically the rays
of light beam straight into your eyeball
to see the to see the AR appear and it
requires a it's a see-through headset
and I think why people ask like is it
vaporware is that they got a ton of
funding and everyone's been waiting for
them to actually release something so
we're all hoping that they're pretty
close you know III don't think it's I
don't think it's gonna be vaporware but
I also think that like you know from
what we've seen working on hololens and
and other kind of hardware there are
some things that are physically very
difficult to I mean just out of physics
you can't beat or can't work around that
much so field of view is is an issue and
then also with see-through just
brightness having it work outside is an
issue as well so I think that when magic
leap comes around it'll it'll be amazing
but it's still gonna run into those same
pitfalls that people have so there'll be
conditions like in inside and stuff
where it will look amazing but as soon
as you take it to certain places it will
start to break - yeah cool thank you and
I think we have time for one last
question what to do when the compass
doesn't work it seems you have that
experience and Eiffel Tower is it okay
to for example that the users manually
um you can I mean we we just went with a
different content solution and entirely
we have done stuff with calibration
I think calibration is is a is a good
solution if it's kind of baked into the
experience itself so we were working on
a big concert experience where we had
time for the for the user to find
themselves in the seat before the show
started they could look on stage and
calibrate the AR at that moment and then
the device would remember where the
position of the stage was throughout the
rest of the concert and then still put
the AR in roughly the right spot because
of that but even to do that there was a
lot of coordination with the with with
the show director and what signage has
to be on stage and all that sort of
stuff so relying on the on the consumer
to do it right is it's kind of yeah
it's a bit iffy so yeah try to work
around that yeah okay thank you I think
we're out of time thank you Jason