Extending AccelByte: To Infinity and Beyond

We don’t believe in a one-size-fits-all approach when building our online services. We know that there will be some customization needed to make them work with your games.

There are two kinds of customization: configuration-based and code-based.

When the logic required varies significantly, it is increasingly difficult to encapsulate complex logic in a config-based solution.

There are a lot of scenarios in which games want to extend, override, or customize:

• I want to add my secret sauce logic for matchmaking
• I want to integrate my own backend logic to do Y
• I want to integrate with third-party solution to do X
• And a ton more!

In all these cases, oftentimes game studios want to put alternative logic on the server side, programmatically.

What Did We Learn From Augment?

We admit this is not the way to go!

Augment was an attempt to support serverless function architecture. We wanted to give game developers a simple way to run logic on the server side. However, the solution lacked a few critical attributes such as debuggability, observability, and predictability. The problem is not the serverless architecture itself but partially due to the use of some immature techs as implementation. The use cases that can be applied are also somewhat limited. In general, game developers want more fully functional development and production environments based on more mature technology.

Why Are We Not Supporting Scripting Languages Like Cloud Script?

Cloud Script allows games to define event callbacks that will cause requests to be sent from the client and executed on the server side.

However, those events can only be defined from a very fine-grained level. If not carefully designed, the events could easily result in excessive calls to the server, resulting in scalability issues and downtime or, at best, inflated infrastructure cost. For those who are not aware of these potential issues, these problems will not be detected until after the game is launched. On top of that, early versions of Cloud Script also suffer from troubleshooting difficulties, lack of observability, and language dependency; among other limitations.

What Is the Problem With Fork and Merge?

i.e. share code, let them build and run locally

When there’s a heavier need to extend or customize our platform, some of our developers' clients would fork and modify the source code on one or more repos of our platform. The forked repo would need to be merged regularly (we release new code every two weeks) to get new features, bug fixes, performance improvement, etc. While this model works well for clients that need deep customization, it is time-consuming and may present other sets of challenges:

• Compilation dependency between the core code and the customized code.
• Since they are sitting in the same project, there is no restriction on customized code to use any other core code or libraries that are available in the core project.
• Must evolve with the core code at the lowest level.
• Any conflicting changes on either side could break the compilation and require manual merges.
• Overheads in constant code merging activities.
• Code refactoring used to be a healthy development activity but now introduces overheads for downstream development as new releases from the core are taken.
• “Avoiding” taking new releases from core, especially around its own release date becomes more common resulting in even more difficult merges later on.
• Another side effect is the coupling between the release lifecycles between the core and the customization project.

Enter: Stable Interface

The solution to this software engineering problem has long existed. What we need here is a stable contract between the core code and the customized logic. In addition, to allow more than one implementation to coexist, Strategy Pattern can be used, which in turn, also relies on the use of a language specific interface.

However, some issues remain. Core and customized code need to use the same language. This is not flexible enough in the game industry. Colocated code cannot be scaled independently. In some languages such as Golang, code from both sides need to be compiled together or at least, with the help of the Golang plugin feature, compiled independently with identical compiler and dependent module versions. We were also dealing with unclear responsibility for colocated logic – troubleshooting requires experts from both development teams.

Why gRPC?

To improve the strategy pattern with colocated code, gRPC can help.

• Can be deployed as microservices
• Allow independent release lifecycle – microservices can be deployed and released separately
• Allow performance testing and scaling to be performed independently
• Techniques like distributed tracing, allow problems to be located easily which is important for cross-group/cross-company collaboration
• Support Streaming (both unidirectional and bidirectional streaming)
• Those additional programming models supported by gRPC fit new classes of functional requirements,e.g. Allow streaming back responses when the request is still coming in from the request stream
• No such standard exists for streaming with REST-based API using http/2
• gRPC uses protobuf as payloads, which provides easy forward and backward compatibility
• New server version can works with older version of client (backward compatibility) while new client version can also works with older version of server
• Effectively decouple the development lifecycles of client (core service) and server (customized logic) even when the protobuf contract is changing
• Provide great flexibility in project management across teams
• Multiple language support – leverage available programming language skill in both teams
• Protobuf is strongly typed so it can reduce runtime surprises
• Efficiency in serialization (size and speed) for cost

What About Server SDK Like Golang SDK?

Server SDK such as Golang (documentation here) or Java is used to talk to other AccelByte services securely using JWT token; so this will work in tandem with the gRPC call.

This is still good for getting current Accelbyte Cloud behavior for further processing. And, still plays a significant role in a few places: async event processing (e.g. analytics) and synchronous call from the customized gRPC server code.

See in Action: Matchmaking V2

We will go through a couple of real examples, looking at how the new architecture enables easy development of customized game specific server logic.

Example one: Matchmaking V2 is one of the most commonly wanted customization targets. First, we identify the interface for abstraction (strategy pattern). Then, we convert the local interface into gRPC protobuf, build an implementation of the interface as an adapter to call gRPC and finally build a gRPC server implementation. The service definition of the protobuf is shown below:

service MatchFunction {
  rpc GetStatCodes(GetStatCodesRequest) returns (StatCodesResponse);
  rpc ValidateTicket(ValidateTicketRequest) returns (ValidateTicketResponse);
  rpc MakeMatches(stream MakeMatchesRequest) returns (stream MatchResponse);
}

First, note that the protobuf service definition is very straightforward and simple. By carefully designing the gRPC interface, the amount of logic you need to implement is minimal. Once the gRPC protobuf is defined and agreed, the gRPC server logic can be iterated separately. With a stable interface using protobuf, you are guaranteed compatibility. Even when the protobuf definition is evolving, you can easily support backward and forward compatibility by following protobuf best practices. This removes any potential problem about project schedules.

Integration testing is also straightforward. When your server implementation is ready, simply register your endpoint to the corresponding AccelByte service (Matchmaking in this example) with our Admin API. This binds your implementation with the core service dynamically.

Also note that you are not limited to the language that is used to implement the core service. You can implement the logic in any gRPC supported languages and, if needed by your logic, use the Accelbyte server SDK to access your data in the Accelbyte Cloud.

Example two: Customized in-game items. By default, in-game items in the online shop are configured statically and appear to be the identical for every player in the game. To attract more players to visit the online shop more regularly, the logic to show the available in-game items in the online shop can be customized in different ways (e.g. exclusive, personalized, daily surprise, random, promotion, newly featured, etc). The selection may be based on information from different sources such as player analytics or other Accelbyte Cloud services such as current item ownership, skill levels, friend-owned items, etc, all accessible from the Accelbyte Cloud server SDK.

Example three: Chat V2. A different gRPC extension point will be provided to allow your own pre-processing of chat messages; for toxicity detection or third party integration.

What We Learned

We learned that what we needed here was a stable contract between the core code and the customized logic. We also learned that in order to allow more than one implementation to coexist, a Strategy Pattern can be used.

How We Want to Help

To help our customers and to improve the strategy pattern with colocated code, gRPC can help our customers through Support Streaming – both unidirectional and bidirectional streaming. gRPC works in tandem with server SDK – such as Golang or Java – to help deploy the strategy as microservices, utilizing multiple language support, reducing runtime surprises, and efficiently serializing for cost.

Ultimately these updates serve to further extend and customize the backend platform for our customers. AccelByte works with studios to ensure visions come to fruition for each game, no matter the unique aspects that may need to be included.

Interested in learning more about our solutions and how they can help you? Request a demo here.