Fortnite’s Black Hole Event — Live Service Done Right

This past weekend, my social media feeds were filled with chatter regarding Fornite’s Black Hole event and the hype it generated was overwhelming.

Quote from The Verge:

“Everything started when the meteor, which had been static in the middle of the island since its appearance at the beginning of season X, started to act up, eventually obliterating the entire map at around 2:00 PM EST. Since then, players haven’t been able to do little more than stare at a black hole.”

Fortnite is truly a phenomenon. But more importantly, with Fortnite, Epic is able to execute a near-flawless execution of a proper Live Service Game.

Live Service is about having a long-lasting dialogue with your players. Long gone are days when game developers release a game, perhaps with few Title Updates with Paid DLC in the mix. These days players expect a constant engagement from game developers. Players want fresh content and experience. Slow down on releasing fresh content and you might lose momentum quickly. With the focus on continuous delivery of fresh content, the Games Industry is embracing the DevOps culture.

How was Fortnite able to achieve pulling a precise execution of synchronized community events like the Black Hole (2 pm EST) or Marshmello concert across all platforms and across all hundreds of the game servers?
I don’t know the specifics of Epic Games’ implementation for Fortnite, but I believe the key is in these two key engineering insights:

1. Direct communication with players via game’s own Online Services

Epic invested in building their own Online Services so they can directly connect with their players via the game and build rich and engaging experiences around player engagement.

By connecting all the games using a singular game-platform, across all store-platforms (PC, Consoles, Mobile), Epic is able to enable cross-platform features such as cross-platform play and cross-platform progression. By having a game-platform to rule them all, other specific Game logic that is not readily supported by one or more store-platforms can be introduced. By having their own set of online services independent of store-platforms, they are able to provide unique player experiences to differentiate them from other games.

(if you want to learn more about cross-plaform play and cross-platfrom progression, follow us on Twitter or Linked In for updates to our blog!)

To build a Live Service game you will need to own and operate your own Online Services — especially if you want to build unique online experiences that can leverage crucial backend logic with security, consistency, and reliability. One of the server-side logic that you want to build on your Online Services, is the Server-Side Toggles.

2. Control the narrative via Server-Side Toggles

When you are operating a Live Service game, you need to be able to control the narrative by having the ability to precisely manage what and when things are happening in your game, from a Live Service.

Side Note: As the game publishing industry transitioned from retail to digital, store-platforms introduced title updates (TU) and downloadable content (DLC) as a way to extend the life of a game. TUs are typically used to fix critical bugs and DLC used to introduce new content. Releasing TUs & DLCs take significant time, money, and effort. As each store-platform has its own certification & release process, the complexity is compounded for each store a is distributed through. There are significant drawbacks to relying on the store-platform TUs & DLCs as the solution to deliver a constant stream of fresh experiences to players.

This is where one of the DevOps Tenets makes sense: “decouple Deployment from Release”:

1. Release Patch/TU that containing all the content planned for the next few weeks to all the clients and game servers (Deployment)
2. Use Server-Side toggles to enable/disable certain content or event (Release).
   Bonus is if you build the Server-Side toggle system to be dynamic, where your Live Producer or Designer can “program” when certain things will go Live.
   For example:
   “ActivateChapter1EndEvent” = “$time > 10/13/2019 14:00:00EST”

I believe that this is the cornerstone of enabling Synchronized Live Event across all platforms, such as the past Comet event, or the time-limited in-game Marshmello concert.

Having this set of Dynamic Server-Side Toggle mechanism also allows you to:

• Kill switch certain features or another way to look at it — put certain features behind feature switch.
  An example of this would be Fortnite putting in Cross-Platform Play capabilities behind a feature flag that they can quickly adjust without releasing any patch.
• Gameplay balancing, of being able to nerf certain gun or ability without releasing a patch.

In conclusion, running a Live Service game is all about establishing direct communication with your players. To truly captivate your audience, you need to be able to control the pacing of the content so your game is always fresh, and players keep coming back.

Please reach out to us at hello@accelbyte.io if you have any questions.