Translations Management Systems

Last updated on Dec 28, 2023

Want to hear a pet peeve of mine? CSV files.

A concept that takes a while to shake for those learning GameMaker is the idea that a file format should be chosen based on the ergonomics of how you would edit them. Namely: if you use CSVs, you use Excel to edit them and it's tabular data. If you use JSON, you use a text editor to edit them and it's nested structs. And from that single assumption stems a great deal of misconceptions about what file format to choose an when.

But the ergonomics of a data file format like CSV or JSON are only consequential if you use the most basic and generic tools to handle them. You don't have to use Excel or a text editor. In fact, you should get away from that sort of thinking. It's like thinking that there only exists a single tool out there with which to edit pixel art or sound files. Instead, it's worth remembering that data file formats in general are merely a data interchange format. A way for programs to exchange data in a common format without having to rely on custom data formats, and all the complexities that come from that. 99% of the time you shouldn't even need to look at a JSON file or CSV file, in the same way you don't need to open a PNG file in a hex editor to look at the raw bytes, or a DOCX in a text editor to look at the underlying XML that Word docs are made of.

One specific instance of this a question I often see being asked in the GameMaker Discord: "how should I store translations for my game?" A common answer to that is: "load it from a CSV, you can send the file to a translator and they can edit it in Excel". It's not the worst option. At its core, translations somewhat fit tabular data of Excel, and gamemaker can indeed handle CSV files, and Excel is a piece of software that a lot of people have and know how to use. Compared to editing JSON by hand, this seems like a decent choice. But we shouldn't be comparing that to editing JSON files by hand.

What if I were to say: "the file format doesn't actually matter. If the end goal is to have nice ergonomics for a translator to supply translations, why pick Excel at all?" This is what I want to talk about in this article. Specifically, one example of a choice for translation storage that is better than using Excel.

Accompanying this the following codebase:

Traduora and translation management systems

My very first "real job" was working at a translation agency, building web tools to help a team of translators keep track of translation jobs. One thing I noted back then was that those professional translators used a type of software called a Translation Management System (TMS) rather than Excel. TMSes have a job of keeping track of all the phrases that need to be translated, and lets multiple translators collaborate on the same project, allowing for them to share the workload easily. It also had features that allowed the original writer to add important context notes about the meaning and usage of phrases, to make sure the tone as set correctly.

Today, there are a handful of free or open source TMSes that are suitable for small indie projects. Most of them are written as cloud software, which is understandable given these days that sort of collaborative work can be done in a browser using a web app. The one I want to highlight and talk about is Traduora:

Traduora is a neat open source project that lets you run your own online translation management system. Although it's not as straightforward to run as a desktop app, once you do run it, you have everything you need to invite translators to your project, add passages to be translated, and, importantly, easily export those translations to your game, or in fact, fetch them automatically on load.

I've set up a demo instance of Traduora that anyone can use to test this out: https://traduora-demo.meseta.dev/

Please note: this demo instance should not be used for production purposes. It is for testing and exploration only. I make no guarantees on the reliability and uptime of the instance or the data. Assume the data may be wiped at any moment, or the whole system taken down.

Setting up or hosting your own Traduora instance is a little outside the scope of this post, but if it's something that interests you and you have a gamemaker project that is ready to have this sort of Translation Management, get in touch, I may be able to help out.

Signing up

I've set up this demo Traduora instance just to help people get started. Sign up with any email address.

Creating a project

Traduora lets you create individual projects to keep all your work in. You'd probably want one project per game.

Setting up locales

In Traduora, "Locales" are the languages that your game is in. The first thing you need to do before anything else is set up at least one Locale.

I highly recommend adding English (or whichever is your native language) as a locale alongside the other languages, as Traduora will be able to display that as a "reference language" so that someone doing the translating can see what the term is in English when they make the translation.

Setting up Terms

In Traduora, "Terms" are the individual chunks of language that can be translated. This could range from simply the word "Cancel" as used on a button, to whole paragraphs of dialogue. It's up to you!

At this point you may also want to set up "Labels" to add to your terms. These are small labels to help categorize your terms

Adding your team

If you're working in a team, you can add them. It's possible to set different roles, so different team members are limited in what they can do to the project. Although, please remember: the test instance I set up should not be used for production purposes, only use it for test purposes.

Do the translation

At this point, you have everything set up to be able to start translating. If you head over to the "Translations" section, you can click on one of the languages, and you'll start seeing a list of the Terms you added, and have the opportunity to fill in the translation.

You very likely want to fill in the English translations (or whichever is your native language) first, since these are used as a reference to which the other translations will be made:

Once you have the first reference language done, clicking on another language, you can show those terms alongside the language you're translating:

Exporting the translation

Once you're done with the translation, you can export all the data.

I suggest the JSON option. The resulting file from this export looks like this:

{
    "dialogue": {
        "chapter1": {
            "dave": {
                "0": "OH TO! Je te connais"
            }
        },
        "intro": "Vous êtes arrivé au bureau, vous voyez Dave le comptable devant vous"
    },
    "menu": {
        "button": {
            "cancel": "Annuler",
            "new_game": "Nouveau Jeu"
        }
    }
}

You can easily include this file in your game's additional data, and have gamemaker load in the file using a buffer_load() and a json_parse(). The basic usage is something like:

function read_language(_language) {
  var _buff = buffer_load($"{_language}.json");
  var _json_str = buffer_read(_buff, buffer_text);
  buffer_delete(_buff);
  return json_parse(_json_str);
}

global.translation = read_language("fr");

// somewhere inside the code for the cancel button:
draw_text(_x, _y, global.translation.menu.button.cancel);

Of course, you'd want to add a way to load in one of several languages depending on the player selection.

As you can see, we're using JSON here. We could have exported as CSV too, though it would have required more code that the above to import. But either way, we're not editing the data file by hand, it's merely a data interchange format. The most important thing is that we are using a tool that is eminently more suited to editing translations than Excel is.

But that's not all! The major benefit of a web-based translation tool like Traduora is that you don't have to manually export and handle the translation files. You can do that automatically and have the game fetch the files itself.

Importing/Restoring a Backup

Because my Traduora instance is for testing only, if you do start using it for your project, you should frequently export your terms as a backup. In case the data is wiped, or if you want to migrate your project to a new Traduora instance, you can do it by exporting all your locales, and then re-importing it.

Note: a lot of the properties like the labels and context don't get exported. There is a way to do this through the API though.

Automatically Downloading Translations

To give your game access to your Traduora project, you need to create an API key:

You'll also need your project ID, which is the UUID that's in the URL bar, e.g. https://traduora-demo.meseta.dev/projects/<your project ID>/

Grab my Traduora client library from here:

Here's a typical setup:

// Create traduora
traduora = new TraduoraClient("https://traduora-demo.meseta.dev", "<project id>");

// Authenticate and then backup, and then get the English locale
traduora.authenticate("<client id>", "<client secret>")
  .chain_callback(function() {
    return traduora.get_locale("en");	
  })
  .chain_callback(function(_payload) {
    show_debug_message(_payload);
  })
  .on_error(function(_err) {
    show_debug_message("uh oh");
  });

I'm using a combination of libraries here to achieve a convenient interface for making HTTP requests, including what I call Chains, which are Javascript promise-like callback chaining for Gamemaker. This helps us keep a sequence of asynchronous requests running one after the other.

First we have to authenticate the client with the Traduora service, the .authenticate() method takes the client ID and secret created previously, and makes the authentication request. This function returns a Chain, which we can add further callbacks to, which will be executed when the authentication request succeeds.

The first callback we add to the chain is another request to Traduora to fetch the "en" locale. We return this so that it can be added to the chain.

The next callback will run only when the previous fetch to "en" locale succeeds. The _payload passed to it will be the result of the request, which should contain the same JSON format as you would get if you exported it from the web UI. At this point, you would save that _payload somewhere in your translation handling code.

Using the Chain library gives us a lot of options. For example, we could make multiple concurrent requests to load all of the languages at once:

// Authenticate, and then get a bunch of locales in parallel
traduora.authenticate("<client id>", "<client secret>")
	.chain_callback(function() {
		return Chain.concurrent_struct({
			en: traduora.get_locale("en"),
			fr: traduora.get_locale("fr"),
			es: traduora.get_locale("es"),
			de: traduora.get_locale("de"),
		})	
	})
	.chain_callback(function(_results) {
		show_debug_message(_results.en);
		show_debug_message(_results.fr);
		show_debug_message(_results.es);
		show_debug_message(_results.de);
	})

This code will first perform the authentication, and then make multiple requests in parallel, before running the last callback once all requests have completed.

Finally, in the above examples I've chained the get_locale() method calls after authenticate(). This is to ensure that we only call get_locale() after the authentication completes. However, there are often times when enough time passes between authentication that you don't have to do that.

// Authenticate
traduora.authenticate("<client id>", "<client secret>");

// Some time later
traduora.get_locale("en")
	.chain_callback(function(_payload) {
		show_debug_message(_payload);
	});

Hopefully throughout this article you can see that while GameMaker can load CSV or JSON, those file formats should be considered data interchange formats, rather than for human consumption. And just like we use dedicated tools for art and sounds, we should also use dedicated tools for data.