Patreon Issues Fixed

This is a largely technical rant, you don't have to read it. But do know We fixed the Patreon Login issues. Download the latest build to receive changes.

Hello everyone, I am Stillkill, the System/DevOps Engineer for MyRobot. I am in charge of the backend server that serves as the middleman between you and our AI providers, as well as other tasks like tracking AI token usage and most relevant to this write-up, tracking Patreon entitlements. As many of you are aware, we've had major problems with Patreon logins over the past few days. This post is a basic write-up on what happened and what broke, and the very fun day I've spent on this issue.

When you log in to patreon in the game, your game hosts a temporary web server and completes the patreon OAuth process, then sends us a access token that we can use to access certain information about your Patreon account, most importantly your subscription level. The problem stemmed from our back end server receiving literally every other request from the game client BUT this "Token POST". It was absolutely baffling. The ASP.NET backend wouldn't ever log that the request was received. We originally thought that we were saturating our network with our pledge updating service, which is a service that checks every 120 seconds for new patreon subscribers and stores them in our local database. This would be done the only way possible with the crap show that is the Patreon API, by making repeated paging HTTP requests to the patreon API. As of writing we have more than 13k users that need to be paged and processed every 2 mins. Sounds inefficient but in reality it wasn't too bad. The memory consumption was low and was locked to a single asynchronous thread, which wouldn't interfere with any other server processes. (on average it takes 20 seconds for this check to complete, 19 seconds of which is network latency and compliance with Patreon's API rate-limit). So after a while it became clear that these requests were not the cause of our issue. (Pictured, the web requests made to patreon for pledge updates) (edited)

This investigation took a good while, considering that for most of it we assumed that it was a C# problem, not a server/client configuration problem. This is where I noticed that nginx doesn't even receive those Token POSTS mentioned earlier. Well damn, time to change our investigation angle. I began intercepting every TCP packet that came into the server, still no patreon token posts, and we were still getting intermittent reports that authentication would work for 1000 people, but not a different 1000 people. This whole time we had been using the game client as a test vehicle for these patreon posts, and at one point I had accidentally closed mine. So instead of relaunching the game i used my already open instance of Postman, a bit of software useful for testing APIs because it lets you send arbitrary requests that your browser can't. I ran requests with Postman several times and all of them would succeed. I would run the game, fail. It became clear to us at this point that the problem clearly isn't the server, but in the client. This was a bit disheartening because I would have preferred to be able to push a patch to the backend that wouldn't require players to update their games.

I notified b0ss of this discovery and sicked him upon the game client's codebase. He scoured the Umbral Plains, seeking vengence upon the MyRobot Git History that had wronged him. He found that older versions of the game DID NOT have this issue. Excellent, now we at least know that the issue didn't exist before. There is one problem though, the authentication process has not changed once since the game's original release. What did change is the number of requests to the MyRobot backend server. Our primary suspect was the Message Flowers, they are the most often requested data from our backend (yes they're internally called bloodstains on the server, also note how you don't see any authentication requests in these logs). b0ss made a private build that disabled the bloodstain requests and it worked (mostly). The rate of failure went down, but we still had authentication errors during internal testing. After some more investigation, b0ss had narrowed it down to "Voice Pre-Fetching", which is a service that pre-generates certain voicelines that the game will use later. This amounted in a significant amount of requests to the server. Once optimized, the issue vanished during internal testing.

In conclusion, we determined that that bottleneck is client-sided, not server sided. I did a small stress-test on the server and found that it could easily handle our current load. The question still remains, why are the game client's requests so inefficient that it breaks? Is that even truly the issue? Is there some bizarre kernal level networking issue that only a 40-inch white beard 3 times divorced network admin would know to look for? We don't know. We do know that this is our only lead so far into fixing the Patreon issue, and it does seem to work. So we're gonna push it and see what happens. We'll still be investigating, so stay tuned!