[JRPG Idle] Dev Log #6 - Data is Power!
/Hello friends, and welcome back to the Dev Log!
I’ve been making games for over a decade now, and one of the development patterns that consistently proves itself over and over again, is the power of separating data from code. Whether it’s a big AAA production with hundreds of developers or an indie production with a small team, it just makes sense to treat code and data separately.
I’m not sure what exactly was the genesis of this practice, but personally what draws me to the approach is its cleanliness. The reality of development is that we have very different workflows for determining an object’s behaviour versus determining its properties. If I want to change how long it takes to mine Copper Ore, I don’t want to wade through the code to find where this is set, and I certainly don’t want to have to change it in more than one place.
One of the mantras in software development is that “magic numbers are bad”. However, at the end of the day, your project is always going to have immutable values that have to be stored somewhere (like say, how much max_health an enemy has). The code needs to know that these values exist, but it isn’t too concerned with what the values actually are. And so it’s all the better that they be defined somewhere else, where they can be easily accessed by the parties that are concerned with them, using workflows optimized for that specific task.
An example from Godot’s website, of entering data in the editor
In a previous Dev Log, I briefly mentioned that we will need a robust system for adding-to, editing, and visualizing the game’s data. Of course every approach has its strengths and weaknesses:
Defining objects in code is easy, it’s a convenient way to establish relationships between objects, and pretty human-readable to boot. But this data is a complete nightmare to maintain, or make any changes to its structure.
One potential approach to managing our data, would be to use the editor to set up our objects. Both Godot and Unity offer convenient ways of doing this, and doing it directly in-engine means we don’t need to have some kind of parser or interpretation layer. This will generally give us a nice interface for previewing and editing an object, and makes it effortless to directly reference other objects. If an activity needs to award Copper Ore, we can literally drag the Copper Ore object directly into the variable field!
However, my experience with this type of approach leads me to be somewhat mistrustful of it. Godot’s editor is finicky to begin with, and I don’t really want to increase my reliance on it. The last thing I want is my carefully curated data setup to be wiped out by some whim of the editor (a fate which I’m sure has befallen many a developer, whatever their engine of choice may be).
Perhaps more importantly though, if I have to click into each individual object to see and edit its properties, this doesn’t actually solve the problems we were looking to originally solve - that is to say, making it easier to get a broad view of my dataset, and to make sweeping changes.
Moving our data from the code files into csv gives us a lot more ability to analyze and maintain the data!
So maybe we don’t want our data to be tied implicitly to the editor. Thankfully, receiving data from an external source is a very common practice, with well established patterns. Using formats like JSON or csv to pass data around is pretty standard, and there are a lot of tools out there designed to simplify this process. Making our data portable like this also makes it much easier to patch, or port to a different engine should the need ever arise.
While not my field of expertise, for my purpose, csv feels like a suitable format. Designers will often joke that “the game is made in excel” - and truly, when I think about the most convenient way to interact with this type of data, my heart just yearns for a good spreadsheet.
There’s just a lot to love about this format. It’s incredibly convenient to be able to see my entire data set at once, and the number of data errors this prevents is huge. The fact that it’s already a spreadsheet makes it trivial to pull out the values, visualize the data with charts, work with the numbers to build formulas, and generally model the game’s interactions. Spreadsheets are built for data entry, so updating the values afterwards is also a breeze, even if I have to update every row in a table (it’s usually just a copy/paste).
Recreating my entire data set in excel was fast and painless, but let's not forget - every approach has its weaknesses.
While the strengths of this approach are many, we definitely compromise on some of the readability side of things. Storing our data in a text format necessitates referring to things by ID, which is hard to do without having the table open in front of you. Our data is also kind of “dumb”. If a relationship or an important value ever changes, those updates don’t have a good way to propagate across the data set - which is a very risky proposition if we’re trying to make this data our ultimate source of truth.
Grist offers us a an SQL layer overtop of our spreadsheet, allowing us to create “smart” relationships between objects
Basically, this is a long way of saying that, what we really want is a database. Something with just enough logic to store relationships, and do some of the grunt work for us. It’s a lot easier for me, as a human, if I can tell my data that an activity rewards the object “Copper Ore” rather than “id: 11”. Focusing on the relationships between objects also means the system becomes way more receptive to either of those objects changing in any way. What’s more, if we ever find that we need to use another object’s value directly, we can ensure that value has 1 source of truth, and automatically update any other fields referring to it.
These days there are any number of ways out there of achieving the effect we want, especially for those who know a thing or two about SQL. Unfortunately I don’t really count myself among their number, but one of the advantages of using a commonly supported format like csv, is that there are powerful tools out there even for noobs like me!
Enter Grist. Grist is essentially what we described: a cloud-based service that acts as an SQL layer overtop of your existing csv data. It looks and functions almost identically to a spreadsheet, while offering all the additional functionality that we need. Getting fields from a specific object is very simple, and it even provides a dropdown to basically eliminate input errors. Once we’ve set everything up, because the database stores object relationship rather than the literal value, we can then switch the displayed field from something human readable (like “item name”) back to the “id” field that our game engine expects, when it’s time to export the data.
Grist isn’t perfect, and there are surely many other ways to achieve the same thing (I hear good things about PostgreSQL). However, having now used this setup for about a month, I feel pretty good about how easy my little database has been to maintain and expand.
(countdown until I eat my words…)
Thanks for reading!