Get notified about Jesper Richter Reichhelm & Patrick Huesler

Sign up to a email when Jesper Richter Reichhelm & Patrick Huesler publishes a new video

when anything starts that can leverage
the experience of all teams that can be
for them but they are not limited by
their design decisions because we do not
at least unless the team wants to share
a code they start from scratch
so when a new team starts they think
okay this is what the others did this is
the challenges they had how can we make
it better and that leads to an evolution
that is driven by the team's an
evolution and different dimensions so
for example technology vice we started
off doing PHP and MySQL on the back end
that's just four years ago I know we do
service in a line and Scala akka you
name it a huge tension and what we did
they're all driven by these incremental
steps of a new team trying something new
and adding it to the knowledge of the
company and the same old fart actually
started off this very standard stateless
application service in the backend and
nowadays the more complex games to
reduce load on the database it's a
stateful approach and the same holds for
platform we started on
event to Google+ went away and are now
on iOS and Android
but there's one evolution that I want to
talk about today and that's user
interaction it's a bit like game design
how can we enable cooler games that you
play together with other with your
friends the same game more meaningful
and how can we enable this with
technology and again we had some some
steps in the past this is the part that
I will tell about and then I will hand
over to Patrick in the second part of
the talk we started a very simple and
sharing playing together Spencer just
like you see what they did think of
pac-man now I hope you all played
played it's really cool game but that's
not really intellectually played alone
the only interaction is you have a high
score and your friends try to beat your
high score that's it and actually
doesn't solve life's game it's a huge
success but technically speaking it's
not much more advanced and pac-man and
we got to play levels if you have a
score yeah okay there were 145 and here
what was cut down on my points and and
those of my friends so I can compare how
how they did and you can also see how
far they progressed on a map so here you
can see that Patrick Merkel speaker is
just playing level 38 and basically
sucks in the game I'm sorry to say that
it's not done so and technically this is
very simple I mean this is the most
similar throat when I play I write to my
database and a writer table that just
named score or something and then
someone else comes along and reads from
the table so synchronization is easy
because I wide my stuff in someone else
we chose simple a bit more sophisticated
and that's also very common approach
that we have in many games that play in
social networks like Facebook is in
inbox situation think of chess you sent
your move rather than postcard send it
to someone and that someone takes that
card and applies it to its local state
his chess board
it's a turn-based game in the evident
Empire that looks a great game and if
you have not played it you lost
something back in the 1980s and
civilization still the same principle
when I play multiplayer I do my turn and
then I wait while the other students
turn so I put my turn into the n-box
Kings which this is a game that we have
been doing follows the same principle
when other people who's at my village
they can collect taxes at my house then
looks like this now here see was a
circle so here I think this is not yeah
she collected Texas and I surely can
choose to accept or refuse these things
and when I do accept then I apply their
changes are read from the inbox and
applied to my state but I am in control
of my own state and she cannot directly
affect that so it's a very simple system
still again
so how does look like I do my client and
the white in the database it's the same
thing the only difference is that
another player can wipe directly into my
state in a separate area it's a bit like
a cube or depending on the data can be
database table so basically they pushed
something on my cue of my index and I
pop it off whenever I want to further
change simple most games throughout the
day and in facebook architectural you be
involved this a bit the database was a
big bottleneck and we now run mostly on
stateful application service all the hot
memories within the application server
itself but this part about the inbox is
still the same very often it's just a
simple Redis database because it works I
could tell you a lot more on this how we
evolved to two stateful things but this
is not scope of this presentation if you
want to know more please go to this URL
google slash back end and if candy of
presentations are on the topic of
hopping ghost of architecture but let's
get back to user detriment but you
really wanted to have is real
interaction someone let's something fall
on the floor and I pick it up this kind
of interaction things not you have
endure multi-user dungeon did anyone of
you ever played mutt
yes take thought go north kill monster
yeah good old times here 23 years ago I
think I played the start in 1990 and the
descended World of Warcraft it's
probably the most successful game ever
basically the left one is a grandfather
of the right one so this is what real
interaction is about and they did such a
game it's named fantastic frogs and this
is how it looks like you tended lots of
things there and it can also fish that
is if you can't fish I can't but the
nice thing is he can also be visited by
friends so he a friend comes by let me
grab that dart said she doesn't fetch it
so how does this work this is working on
any browser this is working on any
firewall this is just a simple flash
game from technically speaking we both
have our clients both left lines and you
stateful application server this is
written a line
and that process holds our state and
controls the connection back and forth
but there's also a shared process which
administrates and holds a state
falsification in this case the beautiful
landscape we saw and the river so when I
do something I apply change to that
location process and change the state
changes and she can be notified on that
site so it goes from the user goes to
the location to user and then back and
back-channel that's the one thing so I
can be informed by the server for change
that happened to the world and use HTTP
chunked transfer for that that's the
same thing you see when it watched a
YouTube video the client opens the
socket the server sends some vitals but
does not close it and keep sending bytes
over the stream and this works
beautifully even on Internet Explorer
and anything that's much less pain than
depth sockets for example it just works
and then you can do something like he
fizzes a fish and I kick it up so
there's the fish I go over and I grab
the goldfish because I'm a mean bastard
and I kind of fish but she can so the
complex thing with these kind of games
is what if she picked the fish at the
same time who gets it that's the problem
so in what we did then is basically I
have an approach where the client must
never wait on the server because he
expects latency so up to one or two
seconds in this kind of environment so
the client is very optimistic so if my
client things I can pick up the fish it
lets me pick up the fish then the
message is sent over to the server but
maybe she was fast and already picked up
the fish that means when my user process
wants to try to get out the fishin this
singers for the process it gets no he
can't they get it and he peels back and
then the client needs to roll back
everything that happened before and you
roll back the because we have how we do
it is we have a local lock of everything
that has been worked before so every
committed action committed by the server
is put on the lock and then we roll back
basically start the whole session
bouquet the whole session up unto the
point and then stop playing again and
this is how it looks like it actually
took us 30 minutes to record this way
occasion because it's not two second
delay but a lot faster and so this is
very rare and this is how we got away
with it but it works very simple this
was our first step into the realm of
real multiplayer and probably was this
technology could do something like
Diablo if you wanted to unless you have
five thousand players but more like ten
this works
now if you've also done a presentation
one like how this works in detail last
year on technology check it out if you
want to so we basically have seen that
we could do real interaction what we
wanted to do when you time it actually
much faster more like shooting at each
other racing cars in order to do Mario
Kart or Starcraft and that's the whole
sort of different game much more
statement hospital and the real problem
behind is you have a disagreeable system
you have many clients out there
minimums to player clients but maybe
more depending on the game and your
server and they have diverging state
that all looks a bit different because
there's time synchronization and and
what comes through service you have some
players who want to cheat
they're able to be on top of the others
and you have to prevent them from
cheating it's not like a distributed
system everyone cooperates but we have
some people that activity for to disrupt
the gameplay by gaining advantage so we
have to avoid that and we have the
problem like it look good should look
know that compensation not like oh I
spawn Durkins darker than the second
data it appears I mean what kind of game
would be happen it would be fun so
there's plenty of information out there
so great world for example it's an upper
left corner that's open source now you
can look at the source code and there's
many blog posts of how they did it and
it's really cool stuff and whatever
corner counter-strike they have a really
cool wiki rather explain how they did
things especially in a top-level Age of
Empires really cool blog post I think
it's called 1500 office on a 28-point a
kilobit modem or something like that and
it's a really problem because as you
state they have thousands of units how
do you synchronize this and Halo 2 also
lesson really also in presentation form
on some Game Developers Conference how
they explain how they do things and he
also played around with this and we were
pretty confident difficult build such a
game as long as we can rely on a decent
report of the network a low latency of
the network having not too many package
loss and no jitter so everything
smoothly yeah cool there was just one
problem we shifted our company focus to
become a more company so all games were
now develop on mobile first we do all
game development files first and mobile
mean 3G so before the device connects to
the really fast internet you first
connect to 3G and 3GS limited bandwidth
especially on upload that means not so
good late and see it as roughly 100
milliseconds just don't talk just
because it's there we have lots of
packet loss when you move from one
Center to another or if you're on a
stadium with 5,000 other people like
you're probably right now fuse fuji have
and you have lots and lots of jitter
still we said we wanted to do that and
we do that by launching a team and
letting the team build a game and then
see if we can do it so we decided to do
a game that's called Banamine who here
knows Bangla manner
so basically multiple players on the map
Oliver on dropping bombs that bones have
accountant of a second or so when it
explodes and kills player great the last
standing clearance and the nice thing is
one second count'em of a bomb that's a
really nice way to counter neck so how
hello doesn't yep okay can everybody
hear me excellent so yes we set up to
build the Wonder Man style game for
mobile and we just wanted to see if we
can do it because we had never done such
a thing before what does she see now
here is still lobby where we do the
matchmaking so this also has to be in a
casual way and so the
so we made that possible I think it's
still playable in my experience too
latency is up to like a min 400
milliseconds we're just really good I
will try a thing after talk if I can get
it to work with GPS here we will see
about that so how did we build that um
first I did a lot of research because we
have never built such a thing and you
weren't sure how to do it and there are
vast resources out there and you're
trying to figure out what do you really
need to do we don't have time to invest
like a year or more coming up with all
those crazy solutions and frameworks so
we had to go quickly move on forward and
I can tell you that it is possible so we
didn't do any all of these things which
is the ones that we wanted to do and we
took baby steps
so just to figure out what we're doing
what we need to change from monitor
Asian to the next and this is not a
scientific graph it's a very pop science
graph it just tells you that you can get
a far way in multiplayer gaming without
doing the super complex stuff on the
other hand if you want to please your
users do you reach a point where you
have to add more complexity but it
normally happens at the end so let's
look at the first prototype that we
built from the scratch and it's had this
very basic structure yet made by clients
a server the clients would self move
inside move comments on the server and
server would relay them to all the other
clients and also store it internally and
the same way you would place farms and
but and then the server would tell the
clients that a bomb exploded so please
note that we only sending around
commands and not state and there's no
synchronization in here whatsoever for
the game to resynchronize and then you
serve eventually we decide on game over
so how did we build that we use TCP for
starters because it is easy and to get
to and TCP does a lot of the things as a
protocol and we just wanted to make it
very quickly without dealing with UDP
and the problems of greens and we use
chaser shrink doesn't look like a very
typical approach for a multiplayer game
especially if you leave read the
literature but it was close to us not
the simplest thing but an easy thing to
get something working and turns out it
worked very well on Wi-Fi local Wi-Fi
and also over the internet if you had a
really good internet connection so to
verify the concept of Bomberman that was
good enough
however on 3qi he didn't work it didn't
work at all and especially on our CTO s
mobile phone I think I call it iPhone of
doom because he gets to burst network
condition at any point in time you want
to happen and one of the reasons of
stage II is a problem is chitter
not in latency but also in bandwidth
right depth is constant changes in
bandwidth and latency that will have on
a mobile network it is a shared
resources it depends on a distance from
you to the cell tower how many people
are using it which protocol are you
allowed to use have you used up all your
monthly data coulomb do you have to fall
back and she Paris all those things
right drop packets packet loss is a big
problem and so what T's does TCP do for
us to help that it is resending
the the messages so that drop packages
can be delivered and also it is ordering
them because in this architecture we had
order is actually very important though
which daily arrived the actually there's
a decibel problem with the order DeWitt
architecture is but in general that
would break out and it would even try to
optimize for bandwidth and sending rate
right so niggles algorithm would go and
try to optimize your bandwidth at the
expense of latency which is not believe
what you want to have in this kind of
game you cannot buy the we turn it off
with TCP no delay as a setting and also
it controls the sending rate how fast is
going to censor it's not going to
overflow the other link those are very
good features of TCP but not really what
we want especially in this environment
with high packet also so one of the
funny facts the cost that is when
packets would not get received by a
client for a period of time for instance
I see yes / playing on this upper right
corner and a few seconds later all of a
sudden you warps all the ways through
the screen like the million dollar man
then you know okay something got stuck
along the way they could get worse on
really choppy networks it was barely
playable because PC features try to do
its connection wizardry with the result
and nothing so we had to fix that
problem and if you need read the for a
lot of forums you see ok you repeat the
thing you need to do real-time UDP okay
we will wanted to try it UDP keeping
this current infrastructure that is
completely command paste and answering
the big question that I had
can we get you repeated work on 3G
networks I didn't find out the resources
on the internet that actually says yes
you can do it it works because they also
needs to work both ways right the client
sends information to the server and
server responds also on a different port
of course through UDP as well and you
don't know if there are any T firewalls
of your data provider is supporting this
or not if it's letting it through or not
what are the firewall rules so we didn't
know you wanted to just test it up and
to give us a sense of packet loss and to
make experience in developing our own
mutual protocol we decided to implement
commands TCP which is by the way a
complete carbon copy of Glenn Fiddler's
Catherine Gamescom description on how to
build something like that so we started
adding sequence numbering so we would
know the order sometimes we'll just
decide to drop them and also a way of
acknowledging with a column field which
is a 32-bit integer that uses a bitmap
just to flip bits as a1 if you had that
if you have packet number 30 and the
first bit is flipped you would say okay
you received the 30s previous packet
right so you can say which specific
number but just the previous 31 and
since you use monotonically increasing
numbers it gives you a sense of okay
what's this package acknowledged in the
last 30 master if it wasn't you probably
doesn't matter in that scenario so we
would tap and we would add a user ID so
this would allow us to survive network
changes because normally what you do is
you use the combination of IP address
and remote port to identify pliant
however if your player is switching
networks and changed its IP address if
you don't longer no longer know who they
are if they're admits of gaming so by
adding a user ID sort of as a session ID
you know that good user is and also
interesting template sphere network
order like little endian big-endian it
matters because of thing one morning to
figure it out because on Erlang we were
there network order doing things and the
clients for doing stuff in in
little-endian and that causes you
problems if you use integers the wrong
way around
so let's look at what dates we did
ordering yes like knowledge packets yes
to a certain extent presenting packets
no flow control no congestion control
node and it I can tell you it did work
so far what we tested from we tested in
Germany a little bit in the Netherlands
I tested just half an hour ago outside
of the conference center so it seemed to
have worked here as well and a few other
countries I can't speak from all speechy
providers but it generally works fairly
well however the game itself UDP works
the game itself
no not even on Wi-Fi right it's evil
it's just my playbook on speech II
forget about it what was the reason we
are still at the same problem packet
loss and if you look at the protocol
that you're just standing around
commands and and never state and there's
no synchronization built into all the
clients to come to the same concept that
everybody would see something completely
you had inconsistent state all over the
place so after for a competitive game
like dynamo
that's the worst outcome outcome
possible we might as well just not built
a game at all
so we try to fix that and the easiest
thing I could come up with from reading
the literature was well we just use
state updates so the server will now
have particular eights and send state
updates to the clients and the client
should send in their commands to the
server so this way we get to synchronize
and we actually would send full state
ten times a second that sounds a bit
excessive by now why would you send for
state updates when you could only send
Delta updates right now the thing is I
wanted to have it quickly done and
figure it out and so did a little bit of
math and I was worried that end to you
the maximum transfer unit will be a
problem so T's the IP Lee would start
fragmenting my packets again because
before the TCP will take care of that
but you'd if it doesn't so I did a
little math it turns out I could fit in
whole bomber world world state into
under 1500 bytes well you have to
subtract the the headers IP and UDP but
it would be well under what you require
for edge so it was fine so rest in your
architecture client sending their
commands the server's bundle stem
applies them to to its authoritative
States and then sends up the updates to
all the clients but this happens roughly
ten times a second from the server but
also roughly ten times a second from the
client because it happens to walking
speeds of your Bomberman in the way how
we built the game is roughly 100
milliseconds that it takes to go from
one field to
and result with the gameplay itself
worked very well and we were really
happy around it it had still some
problems because we had to deal with a
new concept that is the client had to do
interpolation between frames right so
you just keep get frames frames frames
and you need to figure out ok what
happened here what's the next what's the
difference between it and what do I need
to animate and if you will drop a bomb
and then in the next few states you
would not be there you had to remove the
bomb from the client because the client
that would optimize for usability use a
concept called client prediction so what
you do is if I place rest you walk left
but you do not you do not wait until the
acknowledge from the server that
actually walked left because if you
would wait for 200 milliseconds before
you actually do the move you would do
really choppy gameplay so that doesn't
look very good so you optimize on a
client what that means if your package
got lost you have to correct for it
afterwards so there are a few problems
we have to deal with in solving that but
however the main our game was broken so
fetching high scores flogging in logging
out levelling up in your league
changing your user name all the things
over UDP with no reliability built in
was completely broken so we had to
tackle that next so the architecture
game playing the same we kept that
however we looked at the system we took
a step back and thought okay what else
from apart from the gameplay do we have
and so we have matchmaking and high
scores and changing user data our login
and stuff like that
an a reliable communication protocol I
don't like what I don't want to build
TCP myself that would be insane for this
project why would I even bother to do
this so there's two distinct things we
have reliable transmission and other
reliable transmission
and they did not even have at the same
time right either you are in this world
of reliable communication or on this
world playing the game of on your level
so it's very orthotic orthogonal to each
other so kind of nice so we went that
did it in HTTP instead not even playing
TCP because again it was easy it's very
well understood HTTP inside a company
and it already gives you a protocol how
to query for data how to post data I'm
talking rest here right and it goes well
with Jason and there's plenty of
libraries and you don't have to come up
with the communication protocol on top
of TCP how you organize data on how you
talk to each other and it also fit
nicely into reorganizing how the server
was built before that it was just one
big prototype lump there was sometimes
there was some readies in there and
is moving really fast and this time it
took the time and sort of reorganized
things again and that fit kind of nicely
so the server by the way is built in
Erlang and so you could build distinct
airline services one is talking UDP for
a game logic and running in Wunderland
process running one game and on the
other hand you over HTTP you do the
matchmaking matching of players with
other players to user data changing
registering users logging user in
logging user outs
delivering the high scores calculating
the leaves things like that that will be
on the HTTP side of things and in order
to improve bandwidth we decided to move
away from JSON and also have our actual
game data that's being delivered inside
the game to be binary as well so we took
UDP virtual connection also known as
before meant CCP and we had the payload
in binary as well and here is the
reference this is for the two player
games David a couple of bombs we ended
up with 180 bytes as that paid out for a
full game state of a full word for to
play game so I had 136 bytes
and so if you do the math 10 message is
a second from the server to your clients
like roughly one point five kilobytes
per second and so yes practical my past
my first modem was a 56k modem that
actually has a throughput of seven
kilobytes per second so on the bed
beside me I think we're on the good side
so overall that improved our gameplay
even even better because now we could
sometimes even play with GPRS
connections we'd have very bad fantasy
cheaper recipes you rough equivalent 256
kilo tips in terms of standards so how
in it - okay I go for five all right
more important things that we do is we
make our games simple and easy to
understand so you don't have to be a
hardcore gamer to understand our games
and matchmaking the days where I played
was normally playing counter-strike all
you have to figure out the server but
to choose a server where you want to
play and they have to make sure other
people playing or not can enjoy can I
leave - I make my own server to make my
own game sort of and then are people
coming out there's a lot of
synchronization knowledge involved and
we you don't want to do that you want to
make it just one turn key push a button
right so the first matchmaking how we
match up groups of players to play
against each other was just a FIFO queue
people come in if it's for start a game
it's only two we wait maybe for ten
seconds if nobody comes in we start a
game but there's no logic in it for
optimizing for short game times or
unlucky you just happen to play against
the people that are currently playing
but because they click the button at the
same time not because they're our best
match to play against you right so we
had to come up with a solution for this
triangle that came from product that was
we want to have four play games all the
time it cost
it has to best match in terms of skills
that it's xxp experience points leak and
other numbers that I forgot right now
and also time so that did the waiting
time for plane two gave us of the
exceptionally short this fall and after
some tinkering we came up with the basic
data structure and the basic algorithms
to do that is called hierarchical
clustering it's a set of algorithms that
you can use where you build up a pool
build up a tree and you find elements
that are similar to each other using a
distance function so you can do define
of course the distance talk to yourself
for us it was a mixture between
experience points there the league of
the weekly challenges so you are in the
master or in the rookie league or NH
so xB Challenger League and for instance
location might be as well
b1 and also time I call it a desperation
factor because the longer you wait it
the closer you got to everyone because
then you should be able to play against
others because if you don't do that
really good players for instance will
completely starve in a beginners
tournament because they couldn't play
against anybody if you would not add
time as another fact to this because
they would should must be a terrible
match for everyone else and they were
just sit there alone while everybody
else would be playing so time is an
important factor here as well so let's
go through briefly to an iteration you
would have queueing then we go
we built this tree a couple of times a
second and rebuild this tree and try to
find matches and schedule those matches
yes I think this is actually ordered
they came in
I think yes yeah and so you do the
iterations you schedule them to launch
and then the atom two groups is
scheduled to launch and then you try to
launch that game the problem is of
course in the meantime losers come and
go all the time right and some of them
timeout and so on in a subway in a
tunnel whatever so that changes
constantly but this is how you do it in
several passes all the time which
accidentally also happen to be in our
bottleneck in the game the matchmaking
so then you have a match then you tell
all the clients to start a game a given
type II of the game server and the game
ID and then they switch over to UDP and
start the game on UDP and come back to a
GP so yeah that improved our matchmaking
dramatically we did of course several
iterations tinkering in graphs how many
four players two players games and three
players game we had so we could see how
we optimized every time what he also did
was play with friends so if the city we
desperately could pose hit play local
button and it would ask for to ask to
use our GPS coordinates send it to the
server and it could play against yes per
day over the Internet
so it's a another way of doing that hot
Wi-Fi which is using over-serving using
GPS so debugging this little thing I've
built for us in order to better be able
to debug the game so you can play the
game on the terminal client it's a
full-stack integration so it it talks
HTTP it helps you HTTP the UDP and you
can use your keyboard simulating up to
four players so this is really good also
for the client developers but they
wanted to develop a new feature they
don't need two iPhones you just need
your simulated one iPhone and you
simulate the other players over on your
keyboard using your terminal client that
is very nice for that also you can use
that to develop BOTS to see how my bots
were behaving these boats are later used
for integration testing and performance
testing so we could have fully
integrated BOTS I used that for that as
well and it's actually written in Erlang
and curses in Erlang so you can build
your eyes in Erlang exactly and what was
I saying yeah and we also used that to
belay to develop more BOTS in game BOTS
that we have written that so we could
fill up games you do never have to play
alone he always would be play against
somebody and I can give you did this as
a nice challenge try to figure out how
to build a pot for mom again we started
very simple I just wrote the survival
BOTS that would just try not to die but
coming up with good strategies and
buildings as a bot is a very interesting
challenge and I kind of liked it
and another useful tool are the network
link conditioner where you can kind of
simulate that network networking
conditions but one other option will be
just to use our sitios I found but that
only works and he's using it I don't
know why so you have to be sort to
disappoint my co-host and this is on iOS
on the left if you have developer
accounts you can activate that and the
other one you have to download somewhere
from Apple's website so same thing is in
Erlang our language remote shell login
to server checking what's going on in
real time awesome features reloading
code by 200 games are running without
interrupting those games it's a very
powerful feature in your lines that I
don't know from many other languages and
you can just do that and the built-in
monitoring of the Erlang VM as well
because we got v8 the VX witches to work
and then you can just Rheem and all the
graphs audio out up to your local
machine and see what's going on inside
your machine and then we panicked once
because we thought we'd be using too
much bandwidth in our game turns out he
was 2vx widgets that was causing all the
badness and he turned it off it was next
to nothing
so the secret ah back home so that was
yeah one next step for us one game just
trying something out and adding
something to our knowledge we now know
how there are the limitations and how to
handle 3G so I think we finished that
level but we have not really mastered it
I think we will revisit it so what
Patrick's team did was was adding
something to evolution towards better
using directional games but we are not
there yet
so currently we have two more games
running once more that's all so Patrick
is not working on a tracking game they
have even higher requirements regarding
latency so no bombs with countdowns but
really like fast pace action that's
actually already a great game and we've
also another game running on where the
state does not fit into 200 bytes but
it's more like many kilobytes big so you
need completely different models of
synchronization because you're not able
to send over the four full state but we
are still working on that they're not
yet fully there so this is probably a
story to be told next year as for now do
you have any questions we got something
on the on the app so first one was if
every team starts from scratch with
different technologies don't you lose a
lot of time reinventing the same things
over and over that's a good question um
yes no it depends so no no really
so actually so we don't go completely
crazy just trying out new things the
first things is those people who built a
system they have to run it and they all
don't also have to train up a
replacement if they want to rotate to
the next team so taking this into
consideration is just makes it sure
that's all the responsibility of the
team apps will not go fully crazy
and also look around what everyone else
does and what skills are the rest of the
companies so that kind of like limits
that also it's what we do we use very
often as ideas so you're not starting
from scratch Todd completely so just
typing and the code probably something I
guess made me for weeks in the beginning
but later on
you cannot go point to someone else hey
they did a bug they should fix it no you
did build the code that's your bug you
have to fix it no waiting on someone
else so in the long run it makes the
development speed of each team's more
reliable and development teams the
development speed of the teams does not
decline when we add more teams and right
now we are twenty teams running in
parallel if you would have to coordinate
a one-on-one platform and they each have
to about change requests on how to add
something there would be nightmare so
that is a trade-off what we do is not
perfect but it works for us because we
do these very separate games and you can
just get away from it
so we do it and we have you doing it I
guess you also advance your technology
by doing that because if you have that
shared platform you witness kind of be
stuck with it and people would be
hesitant let's do this PHP in my screen
probably a security question how can you
trust the clients it depends on the game
it depends very much on the game it
depends on the motivation how to cheat
so for these distributed games many
things so a lot of things you can do is
something like encryption server for
example you can to make it harder to
cheat on the client but in the end if
you want to make a system really secure
server and have to know what the client
is doing you have to replicate on the
server and that means double
implementation of logic also another
team they are just doing this by sharing
code or some game with an objective-c
and compile that object is secure by
rewriting some some base class I think
this is LLVM based so that the same code
can run on the server and can validate
the full battle so in the
you cannot trust the client never for
some games it just is not important
enough to tackle it but for if you need
to be sure you have to replicate the
logic on server there's no way won't on
it okay
why not use WebSockets what about WebRTC
your approach is only possible on native
clients or can you get access to tasty
community using something like PhoneGap
or cocoon j/s oh that's a lot so this
was a question I think the last two
would definitely go to you maybe one
that we use first we don't have a lot of
mm web stuff these days so we didn't
consider WebSockets however they've
built a multiplayer version of diamond -
which is a match 3 game we use that
building WebSockets but in that regard
we knew we will all only be running on
mobile devices running natively so we
chose the tools to be able to use
natively that we could use natively and
we have the most control over honestly
I've never used web RTC it just read
about it no you can't but the main
answer is our stuff is mostly running
natively and we never had the
requirement to also be able to play the
game inside a browser so that's also we
this run game is a fishing fantastic
from us we had run a WebSocket test so
basically we piggybacked the very small
WebSocket client on on benna's we just
played for the game so he despaired this
for a few hundred thousand days and the
work to get some real data and I think
from 92% of players have worked really
fine and for the 97th somewhat ok and
for 3% it just didn't work at all
but that was a test you did in 2012 but
it gave us feeling that ok Maps orbit is
fine but some transfers just more
reliable so we went for that one more
question here so with respect to the
fish game
why didn't you build a sink /and
this mechanism instead of having two
players pick up the same thing at the
same time actually SS in the
presentation I mentioned we talked about
this so the first approach we did was
more like a pessimistic approach on the
clients or have kind of like consensus
on I mean the thing is the clients
cannot talk to each other so they have
to talk to the server so the first thing
was basically find some contents on the
server and then check back that would
introduce lack in the game you click
something and then have to wait for the
answer and you don't want to accept that
because actual usability in that
usability means no players no money no
company so the second thing is then and
actually implement it is kind of like
it's kind of like those clients were
optimistic at doing this and when it was
added and Conflict we would split worlds
and each would see their world and
actually had implemented that it's
really nice for the client-side but it
was held to implement and maintain on
the server side so eventually we skipped
it and came to this because it was just
the simplest thing to do to make it work
and condense mechanisms there's a
project going in the background are
called Locker which synchronous game
states are which players thing playing
on on what class what node in the
cluster we had enough enough of contents
hanging there and it was complex enough
any other questions those are all the
questions that came in yeah
I know that was her like okay
notice this version of this was already
running the binary version of it okay so
no trace anymore there was an
intermediate point where we would g-sib
chasing to goofy a little bit but no
chased anymore I think right now working
a gamer even or sipping binaries to get
in this moment I think that from a
anybody else yeah that's a very good
point at this point we don't oh I'm
sorry yeah but we for the matchmaking we
use the client's connectivity latency
for matchmaking and the answer is no we
don't but that's a very good point it
was on our schedule but you know how it
is if prototypes you go like that's
enough move on tonight thing