It's been a while since our last post, but we're working towards some big updates to share this fall. Today, I'll highlight a few key engine improvements that have helped us develop Thumper.

Visual Scripting

The benefits of a tool that enables non-programmers to hookup effects, animations, sounds, etc. with simple logic are obvious -- every major game engine has something like this. But as a small developer with a custom engine, the cost of building such a complex tool can be hard to justify. You want to make games, not tools, after all. We started this project using a simple property-grid based system and custom object editors, but as the game became more complex, it became unsustainable. Brian was often blocked, waiting for me to create or extend custom objects. So we finally decided to roll our own visual scripting tool. Here's what it looks like.

While implementing, I was concerned with performance and this post on the BitSquid engine's system was a huge help. By focusing on memory layout up front, we have a scripting tool that is cache-coherent and fast. Including the UI, it took about a month to implement. It was worth it.

With this tool, Brian can do much more without my involvement. It's easy for me to create custom script nodes in C -- that's a good way to expose more complicated functionality as needed. I'm tempted to integrate a scripting language (like Lua) as an intermediary between our visual scripts and C, but for Thumper, that's probably not necessary.

Single-step mode

I added a "single-step mode" where you can pause the game/editor and advance it by a single frame with a key press. It's simple, but useful. Brian often uses this to closely examine animation and effects. Thumper has a lot of fast animations and quick interactions, so this helps us make sure everything looks and feels completely solid. While in this mode, the main game loop is not run and the time line is not advanced. That's convenient when I'm debugging and trying to reproduce particular conditions or set breakpoints.

Its simple to use, you just press F6 to enter single-step mode. While in the mode, you press F6 again to advance one frame, and Ctrl+F6 to exit the mode. Here are a few frames of an animation, each advanced by a single frame.

Entities and Instancing

Thumper has many different instanced gameplay objects, or entities, including rhythmic cues, decorative meshes, effects, enemies, etc. Initially, we developed entities in an ad hoc way. Brian would create the raw resources and custom code would allocate memory for each instance and stitch everything together. It was OK for prototyping, but things got gnarly fast.

So I finally built a real entity instancing system. Now each entity has its own resource file with a well-defined interface that can be extended with our visual scripting tool. Draw calls for instances are batched. Efficient memory allocation for dynamic entity instances was a challenge, but I found the sections on game object memory management in this game engine architecture book useful.

This screenshot from our test mode shows the obvious benefits of an entity instancing system.

Debug Visualizations

I created some high-level visualizations of what's going on in the game. Including things like audio playback, animations, tasks, memory usage, etc. This is another example of something easy and simple that's indispensable once you have it. Here's a visualization of active animations and tasks. It's hard to overstate how useful it is for debugging.

Next Steps

We've come a long way, but some big tech challenges are ahead. Over the next couple of months, I'll add automatic draw ordering for our renderer, a "shader graph" for creating custom post-processing effects, and a packaging system for our builds, to name a few of the big features. I'll post more about our tech developments as they're implemented.

I'm Marc, the programmer here at DROOL. This is the first in a series of technical posts where I'll go "under the hood" of our game engine and THUMPER. These will be aimed at game programmers -- non-technical DROOLers might need a quick toke before reading further. We've recently started using URL "links" to our game objects that we can easily share via e-mail. It's simple and probably not a unique solution. But it's been useful and feels like a feature every engine should have. It was quick and easy to implement, but the most time was spent working around some minor annoyances.

Our need for "Object URLs"

Brian (the DROOL artist) lives in Providence, Rhode Island, and I live in Seoul, South Korea. Because of the distance (and 14 hour time difference), our opportunities for liquid communication via voice and video chat are limited and we often resort to e-mailing each other about development issues.

Like most game engines, our assets are divided into files, each containing multiple objects (meshes, animations, sounds, etc.). For our first game, THUMPER, we've accumulated hundreds of files and thousands of objects. During development, there could be an issue with any one of those files or objects. For example, Brian might have a rendering problem and send me an e-mail like this:

SUP FLURY. There is a rendering problem with bug_eyed.mesh in gameplay/resource/bugs.objlib. I'm not sure if there's a problem with material settings or a code issue?

Writing e-mails like this isn't hard, but we send hundreds of e-mails to each other every month. Even in this simple example, there's opportunity for the sender to mistype file and object names. And the receiver has to navigate to the appropriate file, search the file for the object, and open the object editor. Wouldn't it be nice to avoid this error-prone and manual labor?

Now we do, with "Object URLs" that directly reference our files/objects. These links can be pasted into an e-mail by the sender and clicked by the receiver. For example, Brian's e-mail now looks like this:

SUP FLURY. There is a rendering problem with http://drool.ws/gameplay/resource/bugs.objlib?obj=bug_eyed.mesh. I'm not sure if there's a problem with material settings or a code issue?

When I click that link, the DROOL editor is launched, the "bugs.objlib" file is opened, and the editor for the "bug_eyed.mesh" object is opened automatically.

Registering a Custom URL protocol

The first step is to register a custom URL protocol for your editor's executable. For example, the "http" protocol is associated with your web browser. For the DROOL engine, we defined a custom "drl" protocol and associated it with our editor's executable. Full details for multiple operating systems are described here, but these quick steps assume you're using Windows:

Copy the text below into a text editor.
Replace all the drl protocols with your own custom protocol.
Replace C:\\path\\to\\your\\editor.exe with the path to your editor's executable (don't forget to double backslashes).
Save the text as a file with the .reg extension (e.g. custom_url.reg).
Run the file via the command line or by double-clicking in Explorer. You'll have to click through some security warnings.

REGEDIT4
[HKEY_CLASSES_ROOT\drl]
 @="URL:drl Protocol"
 "URL Protocol"=""
[HKEY_CLASSES_ROOT\drl\shell]
[HKEY_CLASSES_ROOT\drl\shell\open]
[HKEY_CLASSES_ROOT\drl\shell\open\command]
 @="\"C:\\path\\to\\your\\editor.exe\" \"%1\""

Since we're a two person team, my "deployment" method was to simply check this .reg file into source control and tell Brian to run it on his machine. Large teams probably already have a way to deploy scripts like this to every team member's machine. Large teams might also have multiple editors installed on each machine (for different source branches or projects). For brevity, I'm skipping over these issues here.

URL Format and Parsing

Now that your URL protocol is registered, you can test it by typing a link starting with your protocol (e.g. drl://test) into your web browser. You might have to click through another security warning, but your executable should be launched and the full URL will be passed to it as a command line argument. Now your next coding task is to parse the URL into a file name and object, open that file, and open the object editor automatically. I'll leave the details to you, but if you already support double-clicking your editor files in Windows Explorer, then you've already done 90% of the work.

My URL parsing is bare bones to minimize overhead. I just used standard C string functions to extract the file and object name and I don't worry about properly encoding/decoding special characters (all our file and object names are alphanumeric without spaces anyway). But I did follow the standard URL "query string" syntax so that if I do use a legitimate URL parsing library, it will be trivial to extract the values. You can use any format, but I prefer something that is short while still being human-readable. Our format is simple:

drl://relative_path_to_file?obj=object_name

Making URL Links "Paste-able"

At this point, I thought I was finished with URL protocols, but I discovered that unlike standard http:// protocol URLs, if you paste a URL with a custom protocol (like drl://) into most e-mail clients (like Gmail), they don't automatically get turned into "click-able" links when you send the e-mail. Of course, you can manually make links with custom URLs into click-able links by using your e-mail app's GUI, but that is not quick and easy enough for me. I just want to paste the link and send it!

After some research, I learned that custom protocol URLs aren't automatically click-able in most e-mail clients for security reasons. Unfortunately, the standard workaround is a bit gruesome. The "done thing" is to use a standard http:// URL that sends the user to a webserver, then the webserver redirects to the desired custom protocol URL. It's sad to complicate our tool chain with a webserver like this, but that's what we've done.

So now this URL: http://drool.ws/relative_path_to_file?obj=object_name

Gets redirected here by our webserver: drl://path_to_file?obj=object_name

Now everything works. There are many ways to redirect a URL, but probably the easiest way is to use Apache's mod_rewrite and a regular expression rule. I recommend that you don't do any actual parsing of your URL in your webserver, just do a simple find/replace and then redirect. I prefer to keep all the parsing in my editor and involve the webserver as little as possible.

UPDATE: As commenter F Fulin suggested on #AltDevBlog, an alternative to redirecting is to use a standard, but out-dated, protocol (e.g. gopher://) instead of a custom one. If your e-mail client automatically highlights links using a protocol (and you don't need to use it for anything else) you can hijack it and avoid redirecting.

More Editor Integration

Once you have this much working, everything else is gravy. Now that we can go from an Object URL to a specific file/object in our editor, the obvious next step is to make it easy to extract Object URLs from our editor. When you right-click objects in our editor, the context menu has a "Copy URL" option. Selecting that copies the Object URL for a particular object to the clipboard.

Object URL copying in action

Other Applications for Object URLs

The convenience of Object URLs has already paid off in other ways. Like most game engines, when we detect an error, we print out an error message to standard output. Including the relevant Object URL(s) in our errors makes them more useful and actionable. Conveniently enough for me, in the Visual Studio Output window, http:// URLs are automatically click-able.

It's typical for a large team to have a server that continuously grinds through game files and objects, potentially producing errors for each object based on certain validation criteria. On a large project, you might have thousands of object-related errors at any one time. From my experience, it's challenging to manage this process. Artists and other content creators might not appreciate the importance of these errors and it's hard to keep the overall error count under control. If you include Object URLs with these errors, I suspect that everyone on the team will appreciate the added convenience and will be more likely to fix their errors promptly.

This was also posted to #AltDevBlog.