Hey everyone! In place of the usual monthly blog post, today we’ve decided to do something a little different. Very little has happened so far in January and February, and we’d like to talk about that. We’ve been rather quiet on the internals of our game, so we thought it would make sense to talk in-depth about what we’ve been doing behind the scenes. So why don’t you take a seat and share a slice of Pypperoni pizza with us?
Why is it taking so long for v1.0 to release?
Last month, we explained that our game was now feature-complete. This means that all the features we had planned to be in this version are done, and we are just waiting on bug fixes and more to release the game properly, beyond the Test Server. So, why hasn’t that happened yet? Let’s take a look back to when the Toontown Offline Test Server first opened.
Test Server Security
When we released the Test Server, updates were being pushed out at a very frequent pace. Every day for a couple weeks, there would be a new Magic Word here, a new zone there, among many bug fixes. As developers, we had so much fun being able to push out new content at this pace. Never before did we have the opportunity to work on updates and have them playable for testing as soon as we were done with them. It surely motivated us to continue on. Unfortunately, a wrench was thrown in our gears and we soon had a serious problem on our hands.
Because Toontown Offline is a singleplayer game by nature, we’ve never had to deal with security issues before. We aren’t Toontown Rewritten or Corporate Clash with hundreds and thousands of players- we are Toontown Offline. If there’s a district reset, yeah, it’s a big deal and we try to fix it- but it’s not as detrimental to us as it is for other Toontown projects. By opening our Test Server, we were exposing ourselves to the public for the first time. We’ve never had an official, public server before. Never had to. And only a couple weeks after we opened the Test Server, a few malicious individuals took advantage of that fact and began putting our security to the test.
First, our game was dumped. In layman's terms, this means our game was reverse-engineered and it’s source code was published online for anyone to download. In the past, many different Toontown projects have had their games dumped, and we are no exception. However, what made our case special was the nature of Toontown Offline itself. Typically, all Toontown projects have only the client’s code in their executables. Your client is what you see on your screen- a Toon jumping, the gag select screen, those kinds of things. The server on the other hand is typically in charge of keeping everything in check and doing all the important calculations. For us, however, we need to include both the client AND the server code. After all, you wouldn’t be able to host a Mini-Server or run the game in Offline mode without it. So when our game was dumped, that means those who dumped it had access to not only our client code, but server code as well. This made it much, much easier for exploits to be found and abused.
Second, the Test Server was abused. The only malicious behavior our team has ever had to deal with is the moderation of our Discord server. What we were presented with this time was something new and foriegn to us. With access to our client and server code, many exploits were performed on our Test Server. This was both good and bad. Good, because it meant we could patch them up before officially releasing v1.0. Bad, because we didn’t want it to ruin the experiences of our players. At first, we tried some temporary solutions. Our goal was to make sure nobody was coming into the game, spouting foul language and clogging up the server with bots. After that, we would look at assessing the situation and figuring out a more permanent fix. Sadly, things didn’t go so well. The temporary solutions failed to work well, and for a good few weeks we were left frantically scratching our heads. Imagine trying to plug up the holes in a broken pool, but every time you succeed, a new hole pops up. That was us. As a last-ditch effort, we eventually decided to add a new feature- whitelisting. By adding a whitelist to the game, we could control the flow of people coming into our Test Server, thus preventing exploiters. And ever since we’ve added the whitelist, things have been great. It was unfortunate we had to go this route, as it created an extra barrier of entry for entering the Test Server. Regardless, over 600 people have been whitelisted to date, and we couldn’t be more thankful for your patience regarding it. Lucky for us, whitelisting didn’t just serve our purposes for this one situation, but it will also serve as a neat feature for server hosters too. So now, I bet you’re wondering: What is the permanent solution? Well, I’m glad you asked!
Fixing the problem
The crux of our problem was what we use to compile our game. Most Toontown projects out there use a little something called Nirai. This is a deployment tool used for Panda3D games. Up until recently, this is what we’ve used for deploying our game. The upsides are that Nirai is fast, and was relatively easy to set up. As previously stated, we were able to deploy updates insanely fast when we first released the Test Server. A new feature could be finished, and in under 60 seconds it could be released on the Test Server for public consumption. Unfortunately, a downside to Nirai is how increasingly easy it is to dump nowadays. We could’ve chosen to keep Nirai, but pair it with a code virtualization solution. This would essentially secure the game while being able to keep using Nirai, but it would heavily decrease performance. We didn’t want to have to do that, so we decided to leave Nirai behind entirely.
First things first, we wanted to move Toontown Offline from Python 2.7 to Python 3. Support for Python 2.7 is now over, and whatever deployment would switch to would most likely require us to be on Python 3. So before anything else, we had to move to Python 3. We’ve wanted to do this since June 2019, but have often faced setback after setback.
Our goal was to have Toon Parties fully functional before even starting the switch to Python 3. This took way longer than we expected. As explained in previous blog posts, Toon Parties have much more to them than at face value. As such, we first wanted to see if we could get them finished without having to reinvent the wheel. We reached out to a few of our contacts across the community to see if there was any open-source or free-to-use Party code we could use. After all, Parties have probably been developed by half a dozen Toontown projects by now. We found someone who had previously worked on Toon Parties, and they were willing to let us use their code. Great, we thought, that’s a bunch of time saved right there! Now we can focus on other stuff. So we did. We focused on Episodes and other miscellaneous duties while we waited for our source to get back to us with the promised Party code. Weeks and months went past without much follow-up, and we eventually came to the conclusion we wouldn’t be getting any help after all. So on Halloween, we officially started working on Toon Parties ourselves. It sucked to have to reinvent the wheel again, but it had to be done. A little more than a month later, and Toon Parties were complete at the beginning of December. Finally, now that Parties were complete, we could start the move to Python 3.
Only two weeks after that, our lead developer, Little Cat, managed to convert the game to Python 3 almost entirely by himself. So here we are, Episodes done, Parties done, Python 3 done… now what?
Our first attempt to deploy Toontown Offline with something other than Nirai was with Nuitka, a tool that compiles Python to a C or C++ executable. Unlike Nirai, Nuitka wasn’t really aimed at Panda3D games in particular. To our knowledge, no other Toontown project had even tried it before, so we were stepping into new territory. A few weeks passed, and we were encountering countless errors trying to get our game to run with Nuitka. Some files would take up 100 or more gigabytes of ram when compiling, so we had to do extra work just to fix that and the other issues that would arise when trying to compile. At one point, the executable generated was over 300 megabytes in size. 300! That was unacceptable to us. After that, we were only ever able to get the game to show the initial title screen and nothing past it. Eventually, despite many efforts, we decided to scrap all the progress we made with Nuitka and move on to something else- a big slice of Pypperoni pizza.
Back in June when we were first experiencing our problems with Nirai, we thought about using a compiler called Pypperoni.
This is a project that was made by the developers of The Legend of Pirates Online, or TLOPO for short. TLOPO is the largest private server for what you could consider Toontown’s sister game, Pirates of the Caribbean Online. It closed down alongside Toontown and Pixie Hollow, runs on the same engine, is also written in Python, and uses the same OTP (Online Theme Park) server structure as Toontown. So truth be told, the developers of Pypperoni are well-versed in Panda3D development as are we.
So why didn’t we use it at first? Well, although it sounded great to us, things just didn’t align. Just like Toon Parties, we were promised help with setting it up with Toontown Offline. And again, that unfortunately fell through. As such, we thought that we would have better luck giving Nuitka a go first before coming back to Pypperoni. Well, as you very well know by now, we did end up coming back to Pypperoni. However, this time, we were lucky enough to get a head-start from one of the lead developers of the project. They helped us kick-start the process and we’re now almost done setting up our new Pypperoni deployment.
So far, we’ve gotten the game to run on Linux and Windows, but not Mac. If need be, we may have to delay a Mac release until after the launch of v1.0. We don’t want to delay the release by another month or two just for one platform, while the other two could be playing during that time. We also may end up releasing v1.0 without a feature that was available on our Test Server- Discord integration. This feature is going to need a bit more work before we make it available on the Pypperoni builds of our game. So unless we manage to fix it between now and the release of v1.0, it will most likely be disabled until we can control the issue. Similar to our stance on Mac, we feel that we’ve waited long enough at this point, and would happily release the full version of v1.0 if it means putting one feature on hold.
Where to next?
We still have to do a little bit more testing and finishing up of our Pypperoni builds before we can consider ourselves done with them. Once we’re ready, we plan to release a patch to the Test Server with the brand new build of Toontown Offline, no whitelist enabled. This will be the final test of the game before we close the Test Server and publish the official v1.0 update. Until then though, please hang in there as we work to get that out. We hope you enjoyed reading this special blog post today, especially as it was quite a long read. Until next time, have a good one all!
The Toontown Offline Team