From the beginning of our work on the technical side of The story of Nek, we tested various approaches to programmers' work, with better or worse results. But at the end of the day, after more than a year in production, we have clarified processes and tools that improve the work. If you're interested in the field of Gothic modding or just like to look "behind the scenes," then you came to the right place, because there is a lot of meaty stuff.
Organization of work
Before we get to the technical aspects, by way of introduction about the organization of work, because when implementing large projects it is even more important than the work itself.
Task tracking
We record the to-do list and track the progress of their completion basing our work management on simple Kanban boards There was supposed to be a Scrum, but it turned out that there are no Fruit Thursdays, and we can't afford to hire a Scrum Master, so we can't sprint (#iykyk). We are using ClickUp to handle these boards because it was the only one that could provide a reasonable set of features for 14 people at a reasonable price (read: for free).
Code repository
For modification code and satellite projects, we use Git as a version control system and keep repositories on GitHub. If you don't know what Git is, it's the kind of tool that allows multiple people to work in parallel on a project by dividing it into branches. You can work on each branch independently and then merge them, preserving changes from both. Usually, the merging of branches is automatic. Still, if 2 people modify the same place, a conflict may have to be resolved manually during the merging, determining what will ultimately go into the code.
We use Trunk-Based Development, so each developer starting a job creates his ‘branch’ based on the ‘main branch’. When he gets the job done, he makes a Pull Request, which is a request to merge his ‘branch’ with ‘main’, and other developers review the changes and if there are no comments, the changes are moved to ‘main’, from which the others can update their branches. This way, we work independently, without getting in each other's way, and the frequent addition of changes to the ‘main’ means that everyone has a fresh version of the code.
Knowledge base
For knowledge sharing, we use an internal Wiki based on Outline, where we keep documentation of the technical aspects of the project, and when we need some tables, we rely on Google Sheets. The whole team uses a Wiki which contains a knowledge base about the plot and characters in the modification so that everyone can get a feel for what they are creating.
Gothic Mod Build Tool (GMBT)
At the heart of the mod repository is the Gothic Mod Build Tool (GMBT) created by Szmyk for Chronicles of Myrtana and improved by us upon the way of bug reporting for our process. We use GMBT to organize the code by separating the MDK (Mod Development Kit), external dependencies, and our code into various folders that we keep outside of Gothic's working directory. GMBT merges the different folders with the code during the build and then copies them to the working directory that Gothic already uses. This keeps the repository in order and allows us to create our directory structure.
When executing GMBT commands, you can define ‘hooks’, i.e. custom scripts launched at a specific point in the process. We use:
tools\clean-compiled.bat @test.pre.assetsMerge, @compile.pre.assetsMerge
The script removes compiled assets (files) from the working directory before the proper build process begins. This is to force the compilation of the entire project again so that the files in the game always correspond to what the programmer is working with. Otherwise, the game would skip recompiling the already compiled contents and load their outdated versions.
tools\copy-union.bat @common.post.subtitlesUpdate
The script copies our Union plugins from the repository to the Data and System/Data folder right after GMBT generates the OU files. This ensures everyone has all the expected versions of the Union plugins.
tools\localize-strings.bat @common.post.assetsMerge
After copying the files to the working directory, the script runs and executes the translation program on them, which I will tell you more about in a moment.
This way, after firing up the 'gmbt test' command, all the necessary files will go where they should, and a test version of the modification will run containing everything we are currently working on.
Translation system
Since the beginning of production, we have been aiming to release The story of Nek in Polish and English, so we have already been looking for a solution in advance. We especially wanted the translations to be independent of the scripts themselves, to keep only one version of the scripts. For this reason, we decided to keep all the texts in the scripts in the form of $PLACEHOLDERS_IN_THAT_FORM:
func void DIA_SO_20001_Gomez_R1Q2_D11_AmbushA()
{
AI_Output(other, self, "DIA_SO_20001_Gomez_R1Q2_D11_AmbushA_15_00"); //$DIA_SO_20001_Gomez_R1Q2_D11_AmbushA_15_00
AI_Output(self, other, "DIA_SO_20001_Gomez_R1Q2_D11_AmbushA_09_01"); //$DIA_SO_20001_Gomez_R1Q2_D11_AmbushA_09_01
Info_ClearChoices(DIA_SO_20001_Gomez_R1Q2_D11);
Info_AddChoice(DIA_SO_20001_Gomez_R1Q2_D11, "$DIA_SO_20001_Gomez_R1Q2_D11_ResistanceB_15_00", DIA_SO_20001_Gomez_R1Q2_D11_ResistanceB);
Info_AddChoice(DIA_SO_20001_Gomez_R1Q2_D11, "$DIA_SO_20001_Gomez_R1Q2_D11_ResistanceA_15_00", DIA_SO_20001_Gomez_R1Q2_D11_ResistanceA);
};At the same time, we create JSON files in a separate directory with the same path as the script, so that the translation program can easily determine which file to modify.
{
"DIA_SO_20001_Gomez_R1Q2_D11_AmbushA_15_00": "Panie Majster! Słyszałeś, że Nekowcy wyciekaja własne skrypty dialogów na blogu?",
"DIA_SO_20001_Gomez_R1Q2_D11_AmbushA_09_01": "ZEJDŹ MI Z OCZU! Tylko ktoś taki jak ty uznałby, że ten dialog jest autentyczny.",
"DIA_SO_20001_Gomez_R1Q2_D11_ResistanceB_15_00": "Aye! Molte volte, komandorze.",
"DIA_SO_20001_Gomez_R1Q2_D11_ResistanceA_15_00": "A co jeśli?"
}The placeholders are replaced after the files are copied to the working directory when GMBT runs our 'hook'. The program, written in Java and compiled into native code by GraalVM, is multithreaded and simultaneously swaps placeholders in several files, making applying translations rapidly fast.
The downside of this solution is that it makes it slightly more difficult for us to implement dialogs because we have to keep the texts in a different place than the code, so sometimes keeping track of dialogues can confuse us. We remedied this by creating a plugin for Visual Studio Code that reads our translation files and displays the placeholder text above the code for us.
BuildRelease.ps1
Using GMBT, we can build the main modification package with assets and scripts. But our project still consists of Union plugins and assets stored outside the repository, so we use more of these packages, and we have to somehow put them together, preferably to a form that is easy to install for testers and later for players.
The BuildRelease.ps1 script written in PowerShell is responsible for building such a distribution package. The whole process is based on a directory template with the appropriate structure, into which we copy the necessary files, i.e. VDFS (.mod) packages containing scripts, assets and unions, and configuration files. The script generates an NH.ini file with the mod definition for GothicStarter and includes useful information about the version, date, and build number in the description.
# Fragment kodu podmieniający dane w NH.ini
$IniByteArray = [System.IO.File]::ReadAllBytes($IniFile);
$IniContent = $CP1250.GetString($IniByteArray)
$Version = $args[1]
if ($null -eq $Version) {
$Time = Get-Date -Format "yyyy.MM.dd-HH.mm"
$Version = "$Time-dev"
}
$IniContent = $IniContent.replace('%VERSION%', $Version)
$IniContent = $IniContent.replace('%BUILD_TIME%', "$(Get-Date -Format "dd.MM.yyyy HH:mm:ss")")
$IniContent = $IniContent.replace('%GIT_COMMIT%', "$(git log -n 1 --no-merges --pretty=oneline)")
$OutputIniFile = [IO.Path]::GetFullPath("./release/output/System/NH.ini")
Write-Host "=== Writing $OutputIniFile" -ForegroundColor Blue
$IniOutputByteArray = $CP1250.GetBytes($IniContent)
[System.IO.File]::WriteAllBytes($OutputIniFile, $IniOutputByteArray)After preparing the entire directory, the script packs it into a ZIP archive, which is ready to send to the recipient. The directory structure is the same as Gothic II, so installation comes down to extracting the archive to the game folder and firing up the modification via GothicStarter. From the programmer's point of view, to generate such a package, you only need to execute a few commands:
gmbt test --full
gmbt pack --skipmerge
./BuildRelease.ps1 TTS dialogues
For writers testing how dialogues "play" in the game, it is much more convenient if dialogues are read aloud, but at this stage of production, one does not even think about starting recordings. Even if one did think, it's hard to record voices when the texts are not yet confirmed. For this reason, we decided to write a Text-to-Speech generator that reads dialogue lines from OU.CSL and for each one generates an audio file spoken by a Google Translate voiceover. The script, after generating the files, packs them into a VDFS package that can be loaded along with the mod.
The code can be found here, but it needs to be customized. The solution isn’t perfect, because due to the use of Google Translate voiceover a lot of time is spent on network communication, and generating dialogs for our case takes several minutes, and will be even longer. A faster solution would be to use a local TTS engine (e.g. Windows) or to generate TTS live in the game, which is implemented, for example, by zTTSDialogues by fyryNy.
Jenkins CI/CD
Once we had a script-based build process, the idea of fully automating the process and setting up a CI/CD server that would build packages automatically after each commit and could, for example, deliver them instantly for testers began to emerge. We started creating a prototype using Jenkins on Windows Server 2022 unfortunately, we had problems running a full compilation of the project due to a mess of dependencies and Jenkins made incomplete builds. We gave up on the subject and manually built all the packages for a while, along with performing the necessary actions in the game to get the assets to compile, by hand. It was possible to live like that, but in the long run, it's silly to admit that you can't make a complete build, and you had to deal with it.
The problem lay in the fact that we had modified the Humans.MDS file with people animations, adding new animations to it, but it turned out that the original file uses several animations that are not in MDK, which blocked us from recompiling the MDS file. Deep research through German, Russian, and Polish forums allowed us to solve this problem finally, so if you ended up here from Google searching for Hum_TurnL_A05.asc, look no further:
Hum_TurnL_A05.ascwe replace withHum_TurnL_M03.ascHum_TurnR_A05.ascwe replace withHum_TurnR_A03.ascHum_Amb_1hRunT0_A01.ascdoes not have an analog in MDK, but you can download it from this thread on themodders.org and replace it withHUM_AMB_1HRUNT0_M01.asc
After correcting these 3 missing animations in the MDS file, we finally managed to recompile it completely, which unlocked our full builds and opened the way to using Jenkins again. Because building modifications require the game itself, we decided to install it in one place on the server and only run the builds there one at a time, which is sufficient.
We used PowerShell again to define the steps that Jenkins needs to perform because it has served us quite well so far, and by offering access to the .NET library, it allowed us to easily run external processes and asynchronously capture their standard output to write out to the logs. The code for those interested can be found here, and the steps performed boil down to:
# Pełna kompilacja assetów
gmbt compile --full --hooks-forward-parameter="$LOCALE" -V detailed
# Test skompilowanych plików jest wymagany, żeby powstały pliki DAT
# --merge=none nie odpala hooka tools\clean-compiled.bat
# Obok działa pętla nasłuchująca na "Skompilowano:*", żeby zabić proces po kompilacji DAT-ów
gmbt test --merge=none -V detailed
# Budowanie paczki do VDFSa
# --skipmerge nie odpala hooka tools\clean-compiled.bat
gmbt pack --skipmerge -V detailed
# Budowanie ZIPa do dystrybucji
powershell ./BuildRelease.ps1 release/NH.ini $VERSION
And there we have it - a mod for Gothic built automatically on Jenkins without the supervision of any protein being.
Myxir
Myxir is a project born more out of fun than necessity. It began by playing around with creating a parser for scripts extracting various information from them, and after realizing that you can extract quite a lot, a tool began to emerge. It's hard to define Myxir clearly because it performs many tasks, but each uses its knowledge of the mysterious Deadalus language. Myxir is a program that analyzes modification scripts and builds a database from them, on which we create various special solutions.
One is a database of tasks available in the modification with automatic numbering, which we use in the names of dialogs for order. Myxir then can associate individual dialogs with a task based on this number, and after analyzing them, it can build a tree or graph of the progress of all dialogs. We don't have a great use for this feature honestly, but it's a cut-through, you have to admit that.
Another, already more useful feature, is a translation tool that can read all the placeholders in a project, categorize them next, and provide a convenient interface for making changes. For dialogue translations, we can also generate a context tree where the translator can see the entire dialogue, allowing the possibility to create a very coherent translation without silly errors due to lack of context knowledge.
Myxir's data model allows for a lot. In the future, we want to prepare a tool to support the organization of dubbing recordings by creating lists of dialogue lines for a given character and tracking the progress of covering them with audio files.
Technically, Myxir is a web application written in Java (Spring Boot) with a frontend in Angular and uses PostgreSQL as a database. The module that imports information from scripts consists of several ANTLR4-based parsers that convert information from code into database objects.
Looks cool?
Good tools and a good process allow us to work efficiently and keep the quality of the results high, so we try to make our lives easier, not harder. If you have a similar attitude, can program in Daedalus, and would like to co-create a major modification for Gothic, please visit our recruitment page and apply. Maybe you'll be the one to come up with something new for our process :)
