I’m a fan of Crowd Control, a service provided by Warp World. It’s an extension for the Twitch live streaming platform that allows viewers to influence the gameplay they are watching. This is done by letting the viewers purchase “coins” that can then be used to trigger specific effects in the game being played by the streamer. These effects range from being helpful (such as extra health or improved items) to malevolent (such as spawning additional enemies or impeding movement) or just silly-but-harmless. Since all games are different, the set of possible effects is game-specific. Regardless of the game and its exact effects, knowing the Chaotic Evil alignment of the internet as a collective, whatever the malevolent side of the equation manifests as per game is typically the more prevalent one.
Since (most) games aren’t designed for this type of audience participation, each game needs its own effect pack, a bespoke piece of software that can talk to Crowd Control, inform it what effects are available, accept incoming effect requests, and provide feedback about whether it was successful.
This past September, I was watching a YouTube video from a charity stream of Cities Skylines. The gimmick was that the player was trying to build a city, but also promised that he would cause a meteor strike for every $1000 raised. I realized that this process, which in this case was manual, could be automated using Crowd Control. Seeing as this charity stream was a success, there clearly existed some largely untapped fling virtual meteors at someone else’s virtual city niche. Knowing that Cities Skylines had an official modding API, surely I could build my own effect pack, right?
Getting Started
Roughly speaking, the way a given Crowd Control effect pack works falls on a spectrum. The two extremes of this spectrum are what I think of as external and internal integrations.
An external integration is one where the effect pack is separate from the game, manipulating it from the outside. A lot of the effect packs for retro console games do this; they alter select parts of the game’s memory as it’s running to change how it behaves. This can be thought of as a sort of viewer-controlled Gameshark or Cheat Engine. This is simultaneously the simpler and more difficult way; it’s conceptually simple to change memory address X to value Y, but finding which address to set to what value to have the desired outcome is the technically challenging part. Additionally, you are fundamentally limited to doing things that the game’s engine already supports.
The other side of the spectrum is an internal integration where you reprogram or otherwise augment the game itself, effectively writing a mod that triggers the desired effects on command. This is significantly more powerful as you are no longer limited to slightly tweaking the existing functionality of the original game; you can actually add new ones. However, it is also more code-intensive and requires one to learn enough about the game engine to integrate your own code with it. The presence of modding infrastructure in the game, as is the case with Cities Skylines, simplifies this type of integration a lot.
So, where do you start with a project like this? Well, let’s outline the basic elements:
- Learn how to mod the game.
- This step varies enormously from game to game. Is there any documentation, tutorials, or example code? Do you have to reverse-engineer everything yourself from scratch?
- Integrate the mod with Crowd Control.
- One might think the building of a Crowd Control effect pack would involve a lot of Crowd Control integration. On the whole, I found this integration process, while slightly bumpy in my specific case, to be the least of my problems in this project.
- Decide what effects to build.
- This is by far the most important part of the project. If the pack has boring effects, nobody would want to use them. If the effects are too chaotic or destructive, the delicate balance between player progression and viewer sabotage is ruined.
- This part must be thought of in less technical terms than the rest. Yes, you still need to base your effects on what you’re capable of implementing, but it’s fundamentally more about game design. You are designing a second game atop the original one; a game that both the streamer and the viewers are playing together. Balancing, engagement, player psychology, frustration, sense of accomplishment, and all the other aspects of regular game design, are imperative here.
- Implement the effects.
- Again, depends on the game.
- Testing, debugging, fixing bugs, testing, debugging, fixing more bugs, etc.
- Oh no.
- Talking to Warp World to get it published.
- Luckily, they’re pretty chill people.
To me, experimenting with modding the game seemed like the reasonable place to start; even the best Crowd Control integration in the world would be useless if it cannot affect the game. I began by reading the modding documentation on the official community wiki and used it to set up a basic do-nothing mod that the game could recognize and load.
Despite having a basic mod in place, I decided to already take a quick step backwards. Even if I were to implement an effect, how would I trigger it without a working Crowd Control integration? Should I start by researching how to add buttons to the in-game UI? Should I reconsider my choice of not starting with the integration? I quickly realized that none of this mattered; I was in an early prototype phase where elegance is irrelevant. I needed something quick, efficient and utterly disposable. My solution? A text file on my desktop that my mod would read a few times per second. If a specific line of text was in it, empty the file and trigger a specific effect. It’s a caveman-level Crowd Control, but it only took a few minutes to write and, most importantly, allowed me to control what my mod did while it was running.
Disastrous Disasters
Now that I had a working mod and a way to externally trigger it to do stuff, I had a solid foundation to actually implement some effects. Given that this entire endeavor was inspired by someone launching meteors at their city, I felt that this was the right place to start. Let’s check the API documentation for how to trigger a disaster.
Oh no.
Now, don’t get me wrong. I absolutely adore this game and I admire what Colossal Order accomplished despite being a smaller studio. I must however come out and say it: modding this game is a nightmare.
Okay, so my first realization was that the API documentation made no mention of triggering disasters. This was strange to me since I had already checked the contents of the official modding API DLL and found the IDisaster interface, which seemed to contain all the basic functionality for controlling them. Was the documentation simply incomplete? Was it written at launch and not updated once disasters were added? Did they intentionally omit it since disasters require a paid DLC? Another possibility is “Because the disaster API is broken”.
Cities Skylines unfortunately has a number of… um… “quirks” that make modding problematic, at least from a beginner’s point of view:
- Limited documentation. The information provided on the official wiki is at best bare-bones and didn’t answer most of the questions I came across. I was also unable to find any significant secondary sources of modding information. A lot of what I learned about this game instead came from reverse-engineering the game and enormous amounts of trial-and-error.
- Limited API. After a closer inspection of the functionality provided by the official modding API, I found it incredibly limited. It didn’t really have a lot of potential for interesting effects. If I wanted anything beyond some bare-bones features, I had to look past the officially supported modding and instead start messing around with the undocumented internals of the game. The modding API instead became a bit of a starting point in my many reverse-engineering expeditions.
- Broken API. Like I mentioned before, the modding API for disasters doesn’t actually work. To be fair, it is technically undocumented.
- Threading. Most calls and alterations will crash the game unless they’re done from the correct thread. The documentation provides some clues about what thread is appropriate for what thing, but I had a hard time even getting the official threading constructs to work reliably. In fact, even the dispatch functionality (which is intended to safely transmit calls from one thread to another) seemed to occasionally break when used from one of the threads, forcing me to perform arcane nonsense like “from thread X, starting a new thread Y whose sole purpose is to schedule an action in thread Z”. Why this worked, I still don’t know.
Okay, where was I? Ah yes, the undocumented disasters API didn’t work. By checking the stacktraces and reverse-engineering how the disaster API is implemented, I realized that the game itself fails to initialize one of its own data structures properly. Hence, the first order of business was to use reflection to forcibly alter private data inside the disaster API. Yes, I’ve barely gotten started with my first effect and here I am already repairing broken data caused by a bug in the original game.
(As a brief tangent, the API allowed me to pick the exact coordinate for each disaster. For the longest time, I had a subtle bug where the edges of each area were statistically more likely to be hit. It turns out that the game lies to its players when it claims that each area is 2km2. In fact, the real area is 1.92km2; the sides are 1920m rather than 2000m. I would hereby like to petition Colossal Order to immediately patch in a 4% discount for all in-game area purchases to compensate for the 0.08km2 not provided.)
With the disaster API repaired, I was finally able to fire a meteor at the center of the map by typing some text in a file. Remotely controlled meteors was basically what I set out to make in the first place, so now what? I can’t release a pack with just a single lousy effect. Okay, let’s start with the low-hanging fruit: the rest of the disasters.
An Argument Against Arguments
The thing about disasters is that once you’ve implemented one, you’re most of the way to implementing the rest as they share a common interface. Each disaster has a type (meteor, tornado, earthquake, etc.), name, position, angle and intensity. Since some disasters have a massive difference in effect based on the specified intensity, I initially thought it would be a cute idea to utilize Crowd Control’s support for parameterized effects. When a viewer triggers an effect, they would be able to choose how powerful it would be.
While variable intensity would be cool, it would have required implementing some kind of dynamic pricing for the effect. After all, if a level 1 and a level 10 tornado cost the same, why bother with the weaker one? Also, the intensity value isn’t even used for some effects, so I’d have to implement two different systems, one for parameterized effects and one for constant ones. Worse yet, in the case of tsunamis, they are so slow and unpredictable that even an upper-mid tier one takes forever and may end up not really affecting anything anyway. Allowing anything but the maximum intensity for tsunamis would probably just be a waste. Besides, even if I allow people to control the strength of some disasters, do we really need 100 different options for how destructive each one should be?
In the end, I came to the realization that by adding parameterized disaster intensities, it only serves to clutter and confuse the set of options. Instead, I created two categories, Minor and Major Disasters and hand-picked which disaster at which intensity should go in which, or both. By having, for instance, a “Meteor Strike” and “Major Meteor Strike”, it’s significantly simpler to price them. Additionally, the streamer could independently disable them if, for instance, the major one is a bit too destructive for their tastes.
As software engineers, we often overthink and overgeneralize problems. There is something called the KISS principle that we frequently have to remind ourselves about: “Keep It Simple, Stupid”. Just because I could generalize the disasters and make them more flexible, it came at the cost of making everything more complicated for myself, the streamer and the viewer.
Social Media Nonsense
I seem to be editorializing the timeline slightly, because there was in fact one effect I began implementing before the disasters. It was the first one I started making, yet one of the last ones to be finished. To test whether I could trigger anything remotely, I used some of the readily available API functionality to post a message to Chirper, the in-game social network where your citizens discuss your city. I knew from the start that I wanted some effect involving Chirper, but I didn’t know what. The naïve approach would be to parameterize the effect so that the viewer could type in a message to be displayed in-game. However, having people be able to send any unmoderated and uncensored message into a game being livestreamed would probably be a horrifyingly bad idea.
After a lot of time of not knowing what I wanted to do for the Chirper-based effect, I eventually settled on the idea of allowing the the viewer to get namedropped in a Chirper message. The presumption here is twofold: Twitch usernames are moderated, and people like to leave their marks on things. To make it slightly more interesting, I tried added a few possible posts in which the viewer could be mentioned, one chosen randomly each time, in the hopes that people would notice, be curious what else it could say, and trigger the effect more times. I eventually decided to take it even further and combine this with my interest in procedural generation. Nothing fancy, just a simple set of nested mad-libs style messages. There are a few basic templates, each containing a few slots where I’d randomly pick one of a few possible options, to generate a quasi-unique message mentioning the viewer.
Aside from Crowd Control, Warp World also produces podcasts. One of them, called I Got One, involves coming up with silly business ideas and inventions. Inspired by this, and the fact that I had just implemented a system that can combine words at random, I also added a chance for the message to be about how the viewer had invented a new product, when is then randomly generated from a set of possible words. With almost 200 possible products, including “self-heating ice cream”, “canned cheese”, and “drone-delivered fire”, my hope was that the random silliness would inspire viewers to try this effect multiple times to see what other things they might have invented.
Fix = Hack + Time
The Cities Skylines game engine occasionally comes off as fragile. I’m sure there is a reasonable explanation for everything somewhere, but deducing what exactly that explanation is by reverse-engineering its failures is difficult. Case in point, I had major difficulties getting the “Expand City” effect working reliably. At various points during gameplay, the player gets the option to purchase an additional, ahem, 1.92km2 of land. Given the normal requirements to unlock an additional area, as well as the cost, giving the viewers the ability to trade their coins in exchange for immediately granting the player another area seemed sensible. Unfortunately, it was also the singularly most difficult effect for me to implement, simply because it would cause random errors. Sometimes, these errors wouldn’t even manifest right away, instead causing some problem a minute or two down the line.
A good developer, assuming they are given the appropriate amount of time and resources, typically investigates a bug in detail. They analyze the code to figure out what sequence of events lead up to the error, and in the end either pinpoint a specific coding mistake or which set of features turned out to be incompatible with one another. I normally try to stick to this approach; figure out what exactly is going on so that I can make an informed decision about what I want to do about it. Unfortunately, it does occasionally happen that the deck is sufficiently stacked against me that I’m eventually just forced to give up. Some call I make will, at random, trigger some chain of events that, seconds or minutes down the line, causes a graphical glitch or some nondescript exception being thrown. This code, which I don’t have access to, don’t have documentation about, cannot easily debug, and has already shown indications of being incredibly thread-unsafe, is nigh impossible to track down complex problems in without an enormous time investment.
As the deadline for when the pack’s release date approached, I had to make a choice. Seeing as I wouldn’t have enough time to reverse-engineer enough of the game engine to find the root cause of the bug, I either had to cut the effect altogether, or find a band-aid solution to the bugs. I eventually managed the latter, in that I found a roundabout way of triggering the effect that, according to large amounts of testing on my end, for whatever reason made the issues disappear. This was done by having a several-second long cooldown before triggering it again as consecutive uses seemed to increase the chances of failure, as well as triggering it from another thread that seemed less prone to issues. I don’t know why, but this way of doing it made me unable to reproduce the bug again. There is certainly a correct way to do this, but lacking access to the original source and proper documentation, finding that correct way can be enormously difficult. I can only hope that the large amount of testing I did was representative enough of how the effect will work in practice.
Power Curve
Given the length of this article, I estimate that approximately three years have passed since I mentioned that making Crowd Control effects has everything to do with game design. As a more software oriented person, I sometimes find it difficult to remember the human side of the equation. One of the effects I made early on lasted for most of the project until, a few days before release, I suddenly realized it was completely absurd.
The effect was simple: “Destroy all power plants”. I initially made it to test whether I could selectively destroy specific types of buildings and left it in as a “fun” effect. Oh no, the power is gone. I need to rebuild my power grid. What a humorous turn of events. It took me a long time to understand the enormous, no pun intended, power gap between the player and the viewers in this situation. Unlike a meteor, which may destroy a small portion of a city and necessitate partial reconstruction, destroying all power plants causes an enormous and potentially long-lasting setback for the entirety of the city unless the player reacts immediately and is able to replace them. Combine this with the fact that most power plants are expensive and you get an extremely overpowered and overly destructive effect. Turns out that this probably isn’t fun.
In the end, the effect was significantly weakened; each use of the effect only destroys a single power plant. It’s still creates problem for the city that needs to be solved, but it’s less disastrous on the whole. Furthermore, for the more helpful segment of the viewers, I also added the opposite effect: the ability to spawn a random power plant somewhere in the city.
In the end, the destructive ability still needs to be balanced, though the last step of the way is typically done at the pricing level. I simply try to make sure that the individual effects are sufficiently granular so as to be more easily managed.
Legally Distinct Snap
Cities Skylines is not just a city building game, it’s also a city simulator. Never before have I been reminded of this fact as much as when I was working on this effect.
The “Infinity Snap” effect was initially called something else, but I was advised to change it for legal reasons. Regardless of its name, its effect should be recognizable to most contemporary pop culture consumers. Despite the simplicity of its description, “Kill half the population”, what exactly that meant went through three distinct stages.
My first implementation was simple. Pick half the population at random, then change their status to dead. It worked perfectly, so why didn’t I keep this version of the effect? Because Cities Skylines is also a simulation. Dead bodies don’t just disappear; managing the dead is a fundamental aspect of the game. You need graveyards and/or crematoriums, as well as having enough of the city budget allocated to hearses. Normally, this works fine even for large cities since only a small number of people would die at any given time. Now, imagine a city of a million people where half of them would suddenly die all at once. As I was playtesting this, it took years upon years of in-game time for all the bodies to be properly buried, and that’s even after a massive investment in additional graveyards and an increased hearse budget.
Oh right, the budget. Did I mention the secondary effect? Taxes. There are half as many people in the city paying income tax, and half as many people working in the factories and shops, and half as many people buying and consuming goods. Most of the commercial and industrial sector collapsed, creating an even more unfathomably large dent in the city budget. Normally, this would balance out quickly by a sudden influx of people moving in, but not in this case. No, guess what? Nobody wants to move into an apartment or house that’s still occupied by a dead body. Getting rid of them became a gargantuan bottleneck for the economic recovery of the city. Realistic? Absolutely. Playable? Not in the slightest. A single snap would effectively and quasi-permanently destroy the whole city. This is unusably powerful.
My second approach did the opposite. Instead of killing the citizens, they are merely instantly removed from the game. Same effect, but without the city being full of dead bodies. After all, in the piece of pop culture that this is referencing, no bodies were left behind anyway; they merely fade out of existence. This worked, but was extremely unsatisfying from a gameplay point of view. Some viewer traded their coins for an effect, and all they got was the citizen count and taxes dropping briefly before the almost immediate influx of new citizens moving into the vacant homes quickly brought things back to roughly the same state as before. Nobody would want to pay for that.
The third approach had the same outcome, but did it in a flashier and more visual way. Step one is to kill half the population, in the normal sense. This is also done in a staggered way; they don’t all die simultaneously, but rather spread out over a short period of time so you can see dead bodies popping up here and there in the city. After a brief wait, where the city is covered in skulls indicating dead bodies, we do the same process again, but instead remove the bodies altogether, making the skull icons fade out of existence, one by one. I found this version of the effect more visually appealing; you could feel that something important was happening to the city. However, since the bodies are removed, the city will still more or less recover before too long as new citizens move in, preventing a complete economic ruin of the city.
The fascinating thing about simulation games is that they often have complex emergent properties that you never consider until you one day lean back and say “Well, that actually makes perfect sense”. It turns out that, aside from a brief economic hiccup following the disappearance of half the population, things don’t go back to exactly the same as they were. The lost individuals actually have things that the new ones don’t. One such aspect is education. Apparently, the game assumes that your city is the only place in the entire world with schools, meaning that whenever I use the Infinity Snap, half of my (presumably well-educated) citizens will disappear and people lacking any education whatsoever will move in to fill in the void.
Pictured above is the result of me using the “Infinity Snap” a few times in a row, waiting slightly between each use. Each time, the total amount of collective education was effectively halved. This turned out to have an enormous secondary effect since my city was well-developed and had many jobs that required higher education. As such, a lot of businesses were forced to close down, impacting the economy. This absolutely makes perfect sense, yet came as a total shock to me as I had never considered these potential ramifications.
I feel like I have inadvertently created the most sophisticated simulation ever of what would happen in a real-world snap scenario.
Oh Right, Crowd Control
It may seem weird that throughout this article about making a Crowd Control pack, I haven’t really talked much about actually integrating with Crowd Control. So, why is that? Well, to be blunt, because it doesn’t really matter.
Yes, the effect pack needs to talk to Crowd Control in order to trigger the effect, but honestly, that’s a technicality. The important part of an effect pack isn’t how you talk to Crowd Control; the important part is what effects you have, how they work, how interesting and balanced they are, and how stable the system is. After that, the integration is merely a formality, and not a particularly interesting one at that. And there is absolutely nothing wrong with that; dull integration between two systems is just a thing that happens a lot in software development. It also doesn’t in any way imply that either of the two systems themselves are uninteresting.
Although, to be honest, that’s slightly misleading. I’m glossing over some of my own experiences in this project in an attempt to prove the general point I’m trying to make. Integration was reasonably, though not perfectly, smooth for me and there are a few Crowd Control-specific notions to keep in mind throughout the entirety of the pack development.
First of all, a few things in the API weren’t completely clear, and the lack of documentation forced me to occasionally ask a few questions to one of the Crowd Control developers. I have since been told that this documentation does in fact exist and that I “just didn’t read it”, so I must have overlooked it in my eagerness to get started.
Second of all, the use of bleeding edge functionality. It was recommended that I use the integration style where the Crowd Control software acts as a TCP server and two-way communication happens through NUL-separated JSON messages. Using this built-in functionality (instead of having to also write the server portion of the effect pack integration myself) seemed like a perfect choice. And it was, aside from me doing it at the exact wrong time. The feature was still under active development, resulting in a time or two where I had to wait a few days for the next version of the SDK to be released that added support for a feature I wanted to use. Still, not a big issue, and one that wouldn’t happen again if I were to make a second pack some day.
However, the singularly most important thing to be aware of is that Crowd Control has a philosophy behind it that the effect packs need to adhere to. Seeing as an effect is triggered in exchange for coins, which in turn are (indirectly) obtained using real-world money, the pack needs to be very careful with its effects. The Crowd Control server requires the effect pack to provide feedback regarding the outcome of the effect. As such, proper error handling is essential; if an effect cannot be run, or it tries to run and fails in any way, the server absolutely needs to know so it can refund the coins spent. (Just gobbling up the coins whenever an error occurs would be a terrible user experience as people would feel cheated.) Likewise, if an event is successful, the server must be told in order to finalize the transaction. Let’s just say that the finally keyword is invaluable for this sort of thing.
If a pack isn’t written from the get-go with the knowledge that feedback to the server must be guaranteed, it can, depending on how the pack is written, be problematic to retrofit it with this functionality after the fact.
Lastly, the publishing of the effect pack. Warp World were made aware of my work on the pack from early on in the project, so the whole thing sort of just flowed naturally. Towards the end, I provided a few betas and release candidates for testing purposes. Once we felt sufficiently reassured that it was working properly, they simply took it from there, set up the appropriate backend stuff and just released it. From my point of view, it was effectively seamless.
Conclusion
So yeah. I made a Crowd Control effect pack. I learned how to integrate with Crowd Control. I learned a bunch of stuff about how Cities Skylines works. I also discovered some seemingly bizarre design decisions in Cities Skylines’ engine that I later realized where actually clever optimizations, which has already inspired the way I write performance-critical game logic. (Let’s just say that there are major fundamental philosophical differences in how you write performant game logic versus the “behemoth-sized enterprise Java”-type systems I’m more accustomed to.)
All in all, it was a fun and interesting journey. Now let’s go and destroy someone’s hard work with giant meteor strikes.