Swapping engines in-flight: How Pragma helped Omeda change backends while live
Swapping engines in-flight: How Pragma helped Omeda change backends while live

Brian Crecente

Swapping engines in-flight: How Pragma helped Omeda change backends while live

On its surface the V0.15 update for Predecessor seemed like another big content drop for the free-to-play multiplayer online battle arena (MOBA).

Omeda Studios used it to add the game’s third original character—a powerful off-kilter dwarven mage named Argus the Dreadlord. The update also peppered the map with explosive flowers, introduced a new in-game currency for unlocking heroes, rolled out some balance changes, and overhauled the game’s item descriptions.

But easily the biggest change to the already-live Early Access game—a seismic shift designed to solidify Predecessor's foundation for longterm and evolving online play, carried out over months of development with the help of backend engine provider Pragma—all happened behind the scenes. 

While players potentially noticed quicker and more responsive matches, they probably didn't think much of it. “If we do a really good job, no one notices we exist,” said Jon Sredl, senior manager of the live production and game operations at Omeda Studios.

Omeda Studios likened the shift between backend online service providers—especially for a game that’s already live and open to players—to rebuilding a plane mid-flight. That it went off without a crash is testament to the skill of the developers at Omeda, a studio that's been testing fate since it launched in 2020.

But it also has to do with the deep knowledge of Pragma, a backend engine provider with a history of delivering the sort of online service support that allows the likes of developers People Can Fly, Frost Giant, Lightforge Games, and Mountaintop the ability to focus on the creative aspects of building their games. In the case of Omeda, Pragma's help also focused in on to very specific areas for Predecessor: multiplayer and pre-launch load testing.

To help Omeda Studios replace the online services for MOBA Predecessor, Pragma “moved mountains,” said Sredl.

Paragon

To understand Predecessor, you need to understand Paragon, developed by Epic Games and released in Early Access in 2016. Paragon was an exciting new take on the free-to-play MOBAs that were drawing so much interest at the time. It used Unreal Engine 4 to create detailed, intriguingly designed characters that battled it out in third-person across a lavish map.

Paragon was a passion project inside Epic, a game created by a team given the chance to try their hand at anything. They decided they wanted to create this take on the MOBA genre with an eye toward injecting more action elements and verticality into the genre.

Then Fortnite happened. 

The sudden, incredible success of Fortnite’s Battle Royale mode in late 2017 eventually led to a decision by Epic Games to end development on Paragon in 2018. Months later, Epic Games announced that it would release all $17 million worth of game assets for free, to be used by anyone working with Unreal Engine 4.

The shutdown and news of the free content sparked a lot of interest, both from experienced developers and Paragon fans.

Death of Paragon

“I was a content creator for Paragon, so that’s how my whole journey really started,” said Robbie Singh, CEO and founder at Omeda Studios.

Singh soon became the game’s top creator—but then in January of 2018, Epic announced they would be closing the game down in April.

Sredl found the game through some of Singh’s videos and later landed work at Epic Games working on Paragon. But his role shifted three days into his new job.

“We had an all-hands meeting at Epic and they announced the closure of the game internally and then two days later we announced the closure of Paragon to the community,” he said. “It was hard. It was hard for the whole team within Epic. A lot of people had been working on Paragon for years and years and it was really a deeply loved product internally.”

When Singh later heard that Epic was releasing all of the Paragon content for anyone to use, his first thought was of all of the different things that could be created with those assets—but then he started thinking about the times he spoke with the game’s developers, suggesting ideas for Paragon.

“Maybe I was right, maybe I was wrong, but now I could actually try and build the game myself,” he said.

Predecessor

Work on Predecessor started in 2018, after Singh discovered like-minded fans of Paragon—Steven Meilleur and Andrea Garella. The three came together through a Twitch stream Meilleur was hosting where the future Chief Technology Officer was trying to rebuild the game live. Two years later, Omeda Studios was founded by Singh, Meilleur, and Andrea Garella, now the game’s level designer. By 2021 the studio had made enough of a name for itself that it received funding from Epic Games' Epic MegaGrants Program.

While the team had Epic's original Paragon assets to work with, they still had to figure out the game’s mechanics, and also rework some of the meta to help make the game a bit more compelling.

And layered on top of all of that was the overwhelming need to have a solid online backend. As a player, Singh had never had to think much about the major changes that would sometimes roil across Paragon’s servers, completely changing the way characters behaved, leveled up, or purchased in-game items.

“We definitely didn’t know how difficult it was to run a live service game,” Singh said. “If you thought just getting the video game to a shippable start was hard, now try patching every few weeks. It’s very tricky.”

Why online services are so hard

While all this was going on, and as Singh, Garella, and Meilleur worked out ideas for their take on Paragon and slowly built up a studio to make Predecessor, Jon Sredl spent five years working at Epic Games running live service games—mostly Fortnite

But when the free online test for Predecessor went live in April 2022, Sredl hopped on and spent the weekend playing.

“At the end of the weekend, I went downstairs to my wife and I told her I was leaving Epic and I was going to reach out to Robbie and get hired at Omeda,” he said. “I saw this as an opportunity to take what I had learned from Fortnite and bring some of that experience to help out this team, because they had captured magic in a bottle in terms of recreating not just what Paragon was, but what Paragon wanted to be.”

Sredl started in September of 2022 at Omeda. While the game worked well, he knew it was going to face some challenges in the not too distant future.  And he knew online services aren’t an easy thing to master.

“Anytime you are trying to take potentially millions of players—all with their own internet connections, all in their own locations around the world—and get them to be able to play a game so that everything other than the gameplay is invisible, that's the ultimate challenge for online services,” said Sredl. “That’s the goal.”

Think of it this way, Sredl said: When you go see a movie at a theater, you want to be comfortable, but you don’t want to think about how comfortable the seats are. You don’t want to think about the lighting. All you really care about is watching the movie and what you got out of it. 

It should be the same with video games. The goal of online services is to be entirely invisible to players, so that the only thing a player is thinking about is the game itself and how much they enjoy it.

The challenge is that you’re trying to create this in a scenario where everyone has their own computers with varying specs, their own networking options, their own internet service providers. Creating an invisible unified system in that scenario is what the game industry has been working towards for decades.

From World of Warcraft to modern-day giants like Riot Games, Valve, and Fortnite, online gaming has grown in complexity with each progression. The industry has also learned a lot along the way, of course—but part of the problem historically is that a lot of those learnings live within a particular studio or developer. 

“They have learned how to do a particular thing well, and I think we're moving into a stage thanks to companies like Pragma where you can take those learnings and spread them across the industry,” he said. “You can take the things that each developer has learned how to do well, centralize that knowledge, and then apply it to every game that's using those services as opposed to everyone having to solve the same problems over and over again.”

Pragma

Late last year, Sredl, Meilleur, and the Omeda team decided that Predecessor needed to change to a different backend service provider. While the existing backend provider allowed them to get the basics up and running, they had hit a wall where adding new online features became extremely slow and difficult.

“We just needed to be able to control our own destiny, and you can't do that unless all of the code is in your hands,” he said. “When you get to a point where you have aspirations to scale this to players around the world and run this as a live service game for years (and hopefully decades) to come, ultimately you need to know that you're relying on your own skills and that you also have the flexibility to make any changes you want.”

That flexibility, he added, allows you to test and iterate quickly. “I think live service games live and die by their ability to respond and iterate, and ultimately you can't do that unless you have full control,” said Sredl.

But before Predecessor could make the switch, Sredl had to do two things: Find a new provider and convince Singh and the rest of the team to do this with a game that was already live for players.

Singh said the idea was definitely concerning when it was first presented. “We had a team working very hard to build the backend while we were basically flying the plane, and that means you have to make certain sacrifices on what you ship in the short term to get that up and running,” he said.

Still, Sredl won everyone over to the idea fairly quickly and then went hunting for their new provider. After spending some time researching the landscape, the team decided to partner up with Pragma on Predecessor.

“When it comes to those provider models, you've got the full service model, which is what we've been working off for the last year and a half. You've got providers like Pragma who offer an engine based on their experiences, and then you build within that framework and that architecture. And then you've got services that just say, ‘Here. Here's the thing. Build with it as you will.

“We thought the middle option, going with Pragma, was really the Goldilocks for us. It offered the ability to take advantage of their experience and their direct support, while still allowing us to build everything we wanted to and write all the code ourselves.”

And for Singh, it brought both Predecessor and the studio one step closer to that invisible online service dream.

“I just wanna build a great game that's fun,” he said. “I don't want to worry about the backend.”

Replacing the engines mid flight

The responsiveness, the speed, everything is just a massive upgrade—and that's without all the improvements that we can now make to the frontend because we have Pragma integrated,” Singh said. “Of course, we were basically taking the engine out of our plane and trying to put it back in while it's in flight, so we were hoping it wouldn't hit the ground before that point.”

There was a lot riding on the work Omeda was doing with Pragma in the five months or so leading up to the transition. And above it all dangled the notion that things could go sideways quickly if everything didn’t go to plan.

But what Pragma offered to the developers, the game, and the players outweighed the risks. And extensive cooperation and planning involving both Pragma and Omeda's internal teams helped minimize the potential for “hitting the ground” or crashing before the switchover.

“We’ve been planning this for a really long time and Jon and the team worked closely with Pragma to ensure it was a relatively smooth transition for our players,” Singh said. “Hopefully they didn't notice anything on their side other than some extended downtime—and then a much more responsive game once they got back in.”

Sredl said the Pragma team was unbelievably instrumental to the relatively short leadup time ahead of the transition. Two Omeda lead developers, Lascha Lagidse and Ben Davison, wrote most of the code with help from a pair of full-time developers from Pragma, while backend engineer Viola Crellin-Davies rebuilt the game's entire matchmaker. Pragma also brought on additional support to do load testing, ensuring the transition would go as smoothly as possible. 

“Doing load testing over the last month has been a big one,” Sredl said. “We brought on some additional support from Pragma and have worked with them very closely to make sure that we are as prepared as we can be.

“Obviously, every developer’s worst nightmare is a launch not going well and the potential impact that can have on your company and your success,” he said. “But Pragma really moved mountains for us and we’re grateful for that.”

Launch and beyond

On March 5, Predecessor's V0.15 update went live, bringing with it a new character, some map tweaks, currency changes and what the team at Omeda described as “matchmaking and backend improvements.”

A longer note to more technical-minded gamers explained that the backend provider that the team previously worked with led to repeated limitations. That's what led to Omeda rebuilding the entire backend from the ground up, giving them complete ownership and control over their services and codebase.

The news about the backend switch was met with mostly silence—exactly the sort of reaction you'd hope for with a service designed to disappear into the background.

“It was a smashing success,” Sredl said. “It all just kinda worked, which in itself is a hell of an achievement.

“Taking on a rebuild of this nature and scale under this kind of time crunch and making the flip weeks before our free-to-play date was a pretty wild move," he continued, "but the talent and effort on the Omeda and Pragma teams was up for the task and we couldn’t be happier with how things are looking.”

Now that the great transition is behind them, Omeda is looking forward to future updates and tweaks powered by their new partnership with Pragma.

“What I’m really excited about is developing all of these things that I’ve been wanting for a while,” Singh said. “Jon and I are both competitive gamers and we are looking forward to having a ranked mode. Being able to build a ranked mode the way we want—without having to conform to someone else’s standards—is great.

“Being able to build all of these other features much more quickly than we were able to previously, that’s the thing that I’m really excited about with Pragma. They’re also really great folks to work with and that makes it all the better.”

Learn more about Predecessor on their website.


About the Author: Brian Crecente runs consulting agency Pad & Pixel, LLC. He founded Kotaku, co-founded Polygon, and was the video game editor for Rolling Stone and Variety. Before his time in video game journalism, Crecente was an award-winning newspaper reporter who covered crime and public safety for more than 15 years.

Other posts

Contact Us

Pragma is currently working with a select group of studios in development.

A great underlying platform is essential in bringing players together. Pragma's technology lets us focus on the creative side of game development while offloading much of the complexity of shipping at scale.

Nate Mitchell

Founder of Mountaintop Studios and Oculus