Talking Netcode With Adam “Keits” Heart

Talking Netcode With Adam “Keits” Heart


(crowd chatting loudly) – There’s been a lot of talk about netcode in the FGC recently and for good reason. As the scene continues to grow and more and more fighting
games are released players need every opportunity
to help sharpen their skills as well as keep their
game’s communities active. And when conditions line up
perfectly online can be great. Tech can be shared, match ups
can be learned, fun can be had but when online is bad, well… (computer stuttering) You get the point. During an interview for
an upcoming documentary on Killer Instinct, I had a
chance to talk to Keits about netcode and just what
goes into its different flavors and the benefits of properly
implementing rollback systems. – [Keits] Netcode is the
most important factor in a fighting game because
the players who play online need to find a match quickly
and that match has to be stable and it needs to feel
as offline as possible. And fortunately for us this
problem was basically solved in 2007 ish by Tony Cannon with GGPO. So we’ve had our framework to look at for how to approach this problem and do it in a really satisfactory way but I think the thing that
most people don’t think about with having good netcode
is actually matchmaking. It’s not just a factor of
can I find a match that has a low enough ping or that
has close to my rank. When you have really good
netcode that can rollback you’re expanding the search. The maximum tolerable ping
probably for most people in delay-based netcode is around 60 to 80. Anything above that and it’s
just unplayable garbage. If there’s any packet loss at
all, it’s unplayable garbage. With rollback you actually expand that. Now you can play people at 150 ping and it feels like offline. And you can play people at 200 ping with some visual artifacts that are weird but it feels like offline. And suddenly my matchmaking
sphere isn’t the two states next to my state. It’s
actually most of the globe. There’s a website where
you can ping any place to any other place and you’d be shocked. It’s very hard to find
anything that goes over 150. You can find stuff that gets into over 200 if you’re really the
other side of the world but like Orlando to London that’s actually probably not over 150. (high-pitched ding) So if you can suddenly support that now when I’m matchmaking I
can care more about the rank of my opponent instead of just “Is this connection gonna be tolerable?” And when you can do that,
that means players who are weaker or newer are fighting
more players who are also weaker or newer rather than
getting bullied out of the game. And when they lose
they’re waiting less time for their next match because
maybe the same amount of people are playing KI or Street Fighter but when Street Fighter is
looking for a match for you it has to start with a
tight circle of low pings and then it has to widen that as it fails whereas KI can just say,
“150 is fine. Let’s go wide. Let’s find you a good rank match instead of a good ping match first.” So that’s step one, right, now suddenly a lot more people are
having a lot more fun just because they’re
getting the right opponents. And we all know this in our hearts all of us hardcore fighting game players. We know that having the right
opponents is a key ingredient to addiction to this
genre; it absolutely is. You have to play people
around your skill level so that you can learn with them. Very few people have
what it takes to sit down as a new playe r against someone like me or a Jwong or anyone experienced and
tolerate that ass beating for long enough to climb through. That’s just a really
small number of people. You have to solve that problem and the way you do that
is rollback netcode. Rollback netcode also solves
a lot of other problems which you can see when you hop online. When you play somebody
from Europe, yeah there’s occasional artifacts or whatever
but it feels like offline and the reason it feels
like offline is because the game doesn’t wait for
the opponent’s inputs. So in delay-based netcode
you’re gonna need to have both your inputs, which you have
as soon as you press them, and your opponent’s inputs in
order to go to the next frame. Let’s say your ping is 90 to your opponent in delay-based netcode that’s round trip ping time round trip So we’re looking at one frame
being 16.6 milliseconds. It’s roughly gonna be
a six frame round trip. So we already have my inputs. I don’t have to send those and
wait for them to come back. I need to wait for my opponent’s inputs so that’s gonna be about three frames. So if you have about a 90
ping and conditions are good you can play a delay-based
netcode game like Guilty Gear with three frame delay. That’s pretty okay, right?
That’s not the worst. The problem is that
because you have to wait for their information to come what happens when the ping changes? And we all know it’s not stable. It’s 90, sometimes it’s 80,
sometimes it spikes to 120. Now the delay has to change dynamically. So the implication of that is
that, “Oh, I’m getting used to the timing of being
three frames behind. Shoot, now suddenly
it’s five frames behind for a few seconds and I’ve
missed what I needed to hit. Oh now it’s back to three okay, right.” And some games display that
in real time at the top other games hide it from you. And then you get into real
trouble when you drop packets. What if the information
doesn’t come at all? And of course the game is gonna bundle the next frame of inputs
with the previous few frames in case that happened
so it can figure it out. But in delay-based
netcode the game is over when you don’t have the opponent’s inputs. We cannot go to the next
frame, so the game stops. And you’ve seen it; we’ve all seen it. The game just stutters
and it chunks out, right? The whole thing just freezes and it stops and it can’t even take your
inputs during those frames. So if you happen to be in the middle of a quarter circle or dragon
punch when it freezes you might miss that direction entirely and you don’t get your move, right? And people will call that “Eating Inputs”. Then you eventually get
your opponent’s data and it does it’s thing to
catch back up or whatever but it just feels like garbage. The delay is changing,
the game is stuttering… You need to have really ideal connections to play with people. People who are strictly wired, no wifi. People who are under 90
ping very specifically. That’s the only way you’re
gonna have a decent match with people. (intense rock music) Then you introduce the
idea of rollback netcode. And how rollback netcode works is that we never wait for the next frame. We still have our delay of
let’s say two or three frames whatever the developer thinks is right so we’re still gonna wait three frames just to wait for things to get to us. So let’s say we’re at 90
ping in rollback netcode, three frames later my
inputs are gonna play because we have that delay of three and I’ve got your inputs
now it’s gonna play. It’s just like delay-based netcode. It’s literally identical. Rollback is delay-based netcode,
but it’s got more features. So what happens when
I don’t get your input or the delay changes? That’s what rollback does differently. Instead of stopping and
waiting for your inputs, what rollback does is it
says, “I have my local inputs. I’m going to the next frame. I don’t care. I’m gonna predict what
my opponent was doing.” And all we do for that prediction is we just replay whatever was
happening on the last frame. If they’re holding forward
it’s a pretty safe assumption that they’re probably gonna
continue to hold forward. Maybe they stopped maybe they didn’t but we’re gonna pretend they did. But in most cases I
would say 80% of the time in a fighting game you’re
in a committed action. You’ve already jumped. There’s nothing you can
input to change that. You’re already in a fierce punch. That’s the next 45 frames of your life. It’s just you doing that
fierce punching animation. Nothing can change that. So if that hiccup happens
to happen during that time, literally no visual
artifacting can happen. It’ll be exactly what the
game thought it would be. But in the odd cases where
somebody was walking forward we thought they would
continue to walk forward but they actually jumped or
they actually hit a button, we’re gonna get their
inputs a few frames later because they dropped them
or because the ping went up and then the game is going to be able to look at the game state,
rewind a few frames to where we didn’t have the inputs
and say, “Did this match? Did what we get match
exactly what we guessed?” And if it didn’t, it
resimulates the game state in one frame back up to the current frame. Totally invisible to the player and then it just renders
the next frame normally. So in most cases this will be like an invisible kind of correction. I’ll do a little posing here for a punch. You’ll have anticipation poses. Your character… Let’s do a Jago overhead. He puts his hand up here almost instantly. He holds up here for three or four frames before he starts to chop downwards. Say we have to rollback because
we didn’t get that input and then we have to resimulate up. So we missed the first two frames of it. So instead of him going
up here for four frames and then chopping down, he
goes up here for two frames (laughs) and then he chops down. That’s invisible to the naked eye. No one will notice that that happened. Some things you will
notice is if a character we expected to walk forward
suddenly goes backwards and their walk speed is really high instead of smoothly going
like this you’ll see him kinda jerk and pop to the other side. But if your rollback is built correctly, it’s very hard to see those
types of things happen. They almost never happen
at 90 milliseconds. You’ll see them tiny bit more but they’re hidden
pretty well at 150 ping. At 200 ping you’ll see
some visual artifacts but you’ll always feel the same. Your end is always three frames. Your combos always work the same. Your reaction time is always the same. You never have to adjust
and say, “Well, I need to DP six frames earlier than I normally do because this guy jumped and
the delay keeps fluctuating.” You can actually learn the
offline version of the game online because of rollback netcode. And it’s an old technology
and it’s not a particularly complicated technology,
it’s just a matter of developers saying, “Yes,
this is valuable to us and we want to do this.” and getting the message out there to say, “Look what this did for Killer Instinct. It has a thriving scene after
development has even stopped. People are finding matches
lightning fast in this game. They can play people all across the world. This is magic. You have to do this.” – [Esteban] If you talk to
any fan about Killer Instinct one of the things they’ll rave
about is the game’s netcode and rightfully so. Even playing today, six years
after the game’s release, matches are found quickly
and playing the game online feels very much like playing locally. So what’s going on under the hood to keep the intense pace of
KI the same even when playing online? – Basically the gameplay
logic is decoupled from the rendering. So we don’t do gameplay
logic and render in kind of the same thing. The gameplay logic can happen without the need to render a frame. The game is always in the gameplay logic. The game can run with
the monitor turned off and never render anything. And it still all happens you
just can’t see it happening. So think of it like that. There’s a version running with
the TV turned off underneath. And then when we get to a new frame that we all agree is the
correct frame to render then we render it. And that’ll mean if we
go to the next frame and we have both players’
inputs just render it. If we go to the next frame
and we don’t have the opponent’s inputs then we make
our guess and we render that. And then if we go to the next
frame and we got your inputs and our guess was incorrect, now we go back a few frames
but we’re not rendering. And then we resimulate we don’t render. We resimulate we don’t render. We resimulate to the new
frame now and then we render. And the player never misses a beat there. It just goes to the next frame but underneath it rewinds and
it gets us back up to current with proper gameplay
logic and just gives us the new state of the game visually. – [Esteban] Earlier this year Tony Cannon made an announcement
that GGPO would become free to use under an MIT License. Now anyone could apply the
rollback SDK in commercial or noncommercial projects
which begs the question, “Could you just throw GGPO
into a current fighting game and get solid rollback netcode, or is it more complicated than that?” – [Keits] You don’t have
to design for rollback from the get-go. You can retrofit it. NetherRealm proved to this. They retrofitted it to a very old engine and they did a great version of it too. There’s a GDC talk
about this as well where NetherRealm painstakingly
goes through how many people and how many months it
took to retrofit this into their games. But the important ingredients are already in most fighting games. Fighting games have to
be the deterministic so that when you put in the same inputs you get the same results. There can’t be any variation there at all. Pretty much every fighting
game on the planet is deterministic and we know this
because they all have replay systems. And those replay systems are
not videos, they’re input files They’re just replaying
our inputs back to us and because those will always
result in the same game state we get to watch a perfect
replay of our match and take control of the camera
and do all the cool things we love to do in our replay modes. And frame step and all that neat stuff. So we know they’re all
deterministic already. What they really need to do
is decouple the gameplay logic from rendering in the next frame. That’s the next big step and
then there are other steps in there to deal with. Problems you don’t have to
think about in delay-based. Let’s do audio, for instance. If I do a fireball and
Ryu yells, “Hadouken!” (dramatic yelling) Cool and then we’re in
delay-based netcode, I do the fireball, it
drops a frame, stutters but it gets to the frame
where he’s supposed to say, “Hadouken!” (dramatic yelling) Okay we’re still good, right? It’s a little stuttery
and weird but it’s fine. In rollbacks let’s say
I’m throwing a fireball and I get to the frame
where I say, “Hadouken!” and I yell, “Hadouken!”
but actually we just got the opponents inputs and
he jabbed me out of it. (painful grunt) So now I actually didn’t throw a fireball. That was incorrect. We have to rollback the game state, come back up a few frames, I’m in hit-stun now I’m not
throwing a fireball anymore. The fireball wasn’t ever real. I never got to the frame
where I yell, “Hadouken!”. I was only almost there but my character is still
gonna yell, “Hadouken!”. (dramatic yelling) It already started yelling it on my screen now we need new logic
to cut that audio off so he goes, “Hadou…”
and then (air whooshing) slice that off, go into hit-stun. But if you don’t build that stuff then you will see like Street
Fighter X Tekken had this. There’s lots of audio artifacts. He’ll yell, “Hadouken!” as if he threw it but he never made it to that
frame in the resolved version of the game. And now you need a bunch
of new code to handle all these things. VFX is the same issue.
You’ll see VFX play twice or be there when they shouldn’t be there. (energetic music) (dramatic yelling) So there’s a lot of
considerations you have to make when you convert over
to a rollback system. It is not free. You
cannot plug and play GGPO. It’s a great framework for network stuff but you still have to do things
on your end to prep for it and to make sure it’s
still all smooth and good. So it’s definitely not trivial but I think what I want
developers to understand is that you will have to spend
money on netcode either way. You’re either gonna do the bad netcode and spend money on it or you’re
gonna do the good netcode and you’re gonna spend money on it. And yes the good netcode is
gonna cost a little bit more money but it’s not like a
choice between spending no money and a bunch of money. It’s a choice between
spending a bunch of money and a bunch plus a little bit more money. So I hope developers will
seriously consider using this framework whether
they write it themselves or use GGPO which just went free license. (dramatic yelling) – I’m afraid your path ends here. – [Esteban] Outside of some major studios and indie developers, most
fighting games today are still made in Japan, a much smaller country with a much better
internet infrastructure. They still use delay-based netcode which may work fine for Japan
but it’s definitely not as consistent in other parts of the world. When asked why these
developers still choose to use delay-based netcode years after
a game like Killer Instinct proved just how good rollback
can be, Keits had this to say. – [Keits] It’s hard to
speak for other studios but you see fighting games
made in the United States have almost all entirely
converted to rollback. Almost every indie game made
here also uses rollback. That’s definitely not a cost factor that’s stopping people from doing this. I think there also might
be a business factor where they say, “Listen,
the netcode we have just really works well for
Japan and you know what? The audience in the United
States they’re buying it anyway. They’re complaining about it
but they’re buying it anyway. Why would we spend any extra to do this?” There’s the legacy factor too where it’s not like they’re
making a brand new game engine when they make the next Under Night game. It’s the next version of Under Night. So this is an existing engine that has working netcode already. That’s done. Now you’re talking about
redoing huge aspects of the game just to get
this new netcode working. So I think it’s certainly
a lot easier and cheaper of a question when you’re
starting from scratch and you’re like, “Not
only is this a new game but it’s a new engine.” Then you can really make those decisions but I think part of the
problem with a lot of the games coming out of Japan specifically is they are reusing the same
engine over and over again. And because the sales are still
so strong all over the world they don’t really see a
business need to revamp that end of it and maybe they
have it in their brain that, “When we finally move
to a brand new engine maybe we’ll do this.” Maybe they won’t though
and that’s what scares me. And until then we’ve had a decade of games that don’t have rollback and
we’ve had versions of rollback that have come out that are
kind of poisoning the well a bit and making people think
rollback is super teleporty and bad because those have been kind of weaker implementations
or broken implementations of rollback that never got fixed. But it’s not like that when
rollback is well done I mean… You need only play Skullgirls
or KI or Mortal Kombat or Injustice or any of
the great indie fighters that have this Pocket
Rumble or Punch Planet. You need only play
those for a while to see that those misgivings are misplaced and that we can have a better
future together. (laughs) – Thank you for watching
our talk with Keits on netcode and fighting games. As mentioned before this
is part of a much larger documentary on Killer Instinct that’ll be out early next year. All made possible by our
patrons like these guys. So if you like what you saw
and want to help us make more consider becoming a patron today at Patreon.com/HoldBackToBlock Til next time. (jazzy piano music)

You May Also Like

About the Author: Oren Garnes

58 Comments

  1. What do you think about rollback netcode? Should more developers use it in future games? Be on the look out for our full length documentary on the development of Killer Instinct early next year!

  2. Great video, agree 100% with all of it… but it's a huge shame you missed mentioning or even showing the BEST modern implementation of rollback GGPO netcode in any fighting game since KI … 🙁

  3. We in the Punch Planet community have been trying to shine a light on the greatness of the netcode built into the game. I've personally played matches with people from Korea, Russia and in South America, (Im in the US, midwest) with much more playable connections than a "5-bar" connection in other games. Playing someone in the EU feels almost like offline, I'm not joking! Its the main reason why I've kept playing this game for over 2 years. Punch Planet is seriously underrated and has some of the best netcode I've ever played since Street Fighter III: 3rd Strike Online Edition. Thank you for making this video. I will share it and hope this spreads like wildfire throughout the FGC.

  4. Reminder that bad netcode sucks in Japan too. In Sajams discussion video on infilaments netcode article, he shows a video of two pro SFV players playing in an online tournament from different parts of Japan having a horrible stuttering match.

  5. This is the most crystal-clear explanation I ever heard on this very interesting topic. 11:23 was a eurêka moment for me, because I already had a grasp of how it works but never realized rollbacks aren't ever displayed, only used for correcting the current state. Excellent stuff.

  6. TFH player here. Love that Esteban added some footage of TFH at the right times, even though Keits himself didn't mention it lol

    No worries though.

  7. It’s a damn shame that this type of net code is not implemented into every fighting game that is release nowadays. Japanese companies have an old school mentality that is really holding back every fighter that comes out of that country.

  8. Can you guys make a video when you compared netcode of other games like for example Monster Hunter World vs Street Fighter V?.

    Monster Hunter have a good netcode but SF V sucks… why is the different so big?

    I have experienced disconnections from MHW but never had a laggy mission to be honest.

  9. I have 1 big problem with this rollback craze, i play tekken soulcalibur and mk11. Mk11 is the one with rollback and it plays like absolute garbage. Maybe it is a ps4 thing but tekken plays like butter smooth by comparison. Its supposed to be the other way around

  10. Thanks for putting this together HBTB and Keits! It was very educational. I've put out a tweet in JP calling for anyone who can potentially subtitle this video for the JP FGC, mine's not good enough. I'm tiny but I hope it helps!

  11. Does he mean that KI has 3 more frame of delay online compared to offline? Or is that 3 extra frame of delay present in both online and offline?

  12. My top 6 fighting games
    1 Mortal Kombat
    2 Street Fighter
    3 Tekken
    4 Dead or Alive
    5 Killer Instinct
    6 King or Fighters

  13. Great video! Every fighting game should have the best netcode there is and i believe that it would make the games much more appealing to more people.

Leave a Reply

Your email address will not be published. Required fields are marked *