Edge of Ability

Some of the most successful projects are at the bleeding edge of their implementor's ability. Absolutely as simple as possible out of sheer terror of complexity.

The sequel to this phenomenon is the universal second system effect.

Intellectual discovery can be a powerful motivator in design decisions. Take care when tasking a developer with a mundane or boring task. People have a way of making uninteresting problems interesting!


I have more to say about this, including a crazy idea about the SR71 Blackbird, Lisp and the Apollo Missions all somehow appearing before their time (like from the future ;) )..

Infinite Flexibility

Think about that and weep.

Which Languages Handle Packages and Libraries Best?

In the post Why Do We Use C++, I made the following statement.

Another, often forgotten strength of C++ and of many traditional modular and modular-turned-OO languages is linking, or more generally, we might say 'package management'. C++ innately offers build-time linking and runtime 'linking' is also usually available (i.e., DLLs). This allows graceful scaling. Tool support for this in C++ is strong and reasonably robust because C++ is so frequently used to build enormous projects. It's possible to build massive projects in pieces in ways that are awkward or impossible in many functional or logic languages.

Simon wrote,

C++ good at linking and package management??? That's the most ridiculous thing I've heard all week. This is one of the weakest aspects of C++. Where to start:
* no standard ABI for compiled code. ("Don't mix compilers")
* no standard for managing namespaces and packages like in Java.
* Very hard to evolve a class and keep binary compatibility. Lots of obscure compiler depend rules need to be followed to do this. ("Change a .h file, rebuild everything that uses it")
* dynamically loading code modules has to be done manually. Java, Python and other languages load classes on demand.
* C++ code can call C code, but any other kind of mixed language development is a nightmare.
* easy to break features like dynamic_cast when loading share libraries. (My fun debugging adventure this last week...)

I think Simon makes some excellent points here. So I'd like to ask the community.

What is the best language (and what are the best language features) for linking/loading and package and library management?

Why Do We Use C++?

One of the recurring questions (especially when confronted with concurrency) is 'why do we use C++ for game development?' This entry was extracted from Concurrency in Game Development since it's really a separate topic all its own and that one was overlong anyway.

Why Do We Use C++?

Now before I hear the platitudes about how [insert favorite non-C++ language here] is better anyhow and would fix everything in one shot, let me ask you how feasible it would be to develop multi-million line software on multiple, often brand-new, platforms simultaneously with extreme performance expectations, leveraging mega/gigabytes of shared legacy code and without having to port, develop and/or maintain one's own compilers and toolsets with this superior language. If the game development industry turns to non-C++ languages to solve this, it will be slowly and painfully. And, most likely, it will be an industry-wide move and not just one studio or another, although some must lead the way. Honestly, I would never expect C++ to go away completely (just as assembly has never really gone away), and some future approaches may just be layered on top of C++ (Bigloo, Intel Ct, OpenMP), or use it when performance is critical (Java's JNI).

C++ is flexible and fast. With sufficient (possibly enormous) effort, it can do almost everything any language can do. It can function both as a high-level multiparadigm language and as a low-level portable assembly language. What makes C++ (and C) so widespread is its unrestrictive nature. This is widely seen as a negative, but in the real world, being able to abuse your language to get what you want from your machine is of crucial importance - especially when performance is a primary concern. That said, you may only want to abuse your language say 2-5% of the time. The rest (95-98%) of the time, you'd like some nice, type-safe, bounds checking, memory-managed, interactive, memoizing language, giving you a >100% increase in productivity. That we have settled on C++ indicates that the 2-5% is so crucial that we're willing to sacrifice the rest for it. In the game industry, I think that's a fair statement.

Another, often forgotten strength of C++ and of many traditional modular and modular-turned-OO languages is linking, or more generally, we might say 'package management'. C++ innately offers build-time linking and runtime 'linking' is also usually available (i.e., DLLs). This allows graceful scaling. Tool support for this in C++ is strong and reasonably robust because C++ is so frequently used to build enormous projects. It's possible to build massive projects in pieces in ways that are awkward or impossible in many functional or logic languages. (See comments, some strong points about other languages package handling vs C++'s and Which Languages Handle Packages and Libraries Best?)

Still, writing solid C++ code even in the absence of multithreading requires a mastery of nearly the whole language, making it dangerous for inexperienced developers. Even merely adequate C++ coding is heavily reliant on learned idioms that are not a part of the language, and therefore unenforceable. For example, that you are allowed to return a pointer or reference to a stack-allocated object from a function, or that you are allowed to overflow a string buffer have cost the world untold man-hours and dollars. And yet for all its ills, C++ ranks near the top of the most useful (or at least used) languages ever explicitly designed (including Esperanto).

(4/11/08 - adapted from one of my comments in reddit.)

I should stress again that one of the more crucial issues surrounding language choice is the set of tools provided to us by the console vendors. C++ wasn't adopted until late in the console world because C++ standards weren't well supported by console vendors. Tools have always been poor on consoles compared to the PC (this has changed somewhat with the 360), and good quality C++ compilers were rare in previous console generations, but C compilers were available (athough they, too, came late - with the original PlayStation or Saturn, I believe?).

The PC gaming world hasn't seen this sort of lag. Game developers are eager to adopt new technologies. Other technologies have been unavailable on these platforms.

This is an exploratory post (as they all are..) and is subject to change.

Concurrency in Game Development

Effectively developing concurrent software for modern machines with multiple cores is perhaps the greatest technical challenge we'll encounter in some time. Our current approaches just aren't up to the task of creating robust multithreaded code.

Why Do We Use C++?

Please see the linked post for some thoughts on this topic. Before we look directly at concurrency, let's look at what brought us to where we are.

Categories

It’s reasonably well established that categorization is a fundamental human strength. You might say that feature extraction and classification are hardware-accelerated in the brain. Even at levels far below the conscious (i.e., the visual cortex) information is categorized before it even becomes 'thought' to us. In fact, categorization is so fundamental to human thought that assuming categories themselves are real objects has been a universal illusion. In “How the Mind Works”, Pinker states that nearly all cultures initially adopt a ‘folk idealism’ as a result of this. Applying this to the boundary of man and machine communication, Object Orientation has evolved as a straightforward way to map categories (type, class, etc.) and instances of those categories onto machine architectures that deal primarily in a few primitive and largely undifferentiated numerical types.

Abstractions are complexes of categories and their interactions. Understanding complexity in terms of hierarchies of abstraction is something that people do really well. Modular Programming and, to a greater extent, Object Orientation attempt to give us tools to work at these levels of human competence - with, of course, some consequences in terms of final performance.

The predominant programming paradigms attempt to map the way people think onto the way machines operate. Let's turn our attention to concurrency and see if we can stretch this a bit further.

Concurrency as Time vs. Space

In software development, one of the things I’ve noticed is that people are much better at understanding space than time. This is why we map out time in timelines, MS Project files and a million other ways in spatial form. What this means to programming is that anywhere you can map out state in terms of space, the result is far easier to comprehend. For a clear example of this, see Google's MapReduce.

This is the core issue with concurrency. It’s very difficult to understand the possible states of concurrent systems because they happen in time. Many of the abstractions that can help with concurrency (such as Functional Programming, Message Passing, etc.) are useful because they essentially transform a state-heavy process into some equivalent but more understandable spatial map whose design appears much more static.

Open Questions

In some ways the game industry is at the vanguard of multicore development on consumer machines with the Xbox 360 and the PlayStation 3. The amount of time and effort that we spend finding and fixing multithreading bugs is terrifying. While the hardware companies move toward doubling the number of cores with each processor generation, software companies will be reeling.

Clearly, C++, as it is currently, is not well suited toward developing software on highly multicore machines. Ideally, we’d have a language, extension or paradigm that would allow us to map thread concurrency onto easily understood (that is, spatial) language constructs that discourage or prohibit the kind of multithreading errors that currently waste hours and hours of our time. Right now, we don’t have to worry how many registers there are on the processor when we write C++ code. Similarly, whatever language/paradigm we’d want to use would, at the compile stage, optimally generate code for the number of cores available to it on the target platform.

So would some sort of functional language be best? C++ with functional extensions? Erlang? Haskell?

What about programming models based more on hardware description languages such as VHDL and Verilog, which are inherently spatial? Would they map more effectively to multicore machinery since hardware languages describe processes which are inherently asynchronous?

In any event, we software developers have an interesting road ahead.

The Impossible Dream #4

(Updated 11/23/2007)

Where Have I Been?

It's been months since I've blogged. They've been productive months, though. See, in early October, another beautiful baby girl joined our family! It's been a rush having another child. It's incredible how you can still be completely awestruck as if it were the first time, each and every time. :)

Prior to that, vacation.

Because I had some time off, I slipped in some hours on my Impossible Dream. I hadn't really considered blogging about it yet until I stumbled across Project Steel & Glass at The Cluttered Desk that motivated me to write an update.

Before leaving on vacation this summer, I'd made some headway with Pair, my embedded script language. Especially, I'd made some strides compiling it to more efficent bytecode, etc. It's in a sort-of usable state. Still, some things like lexical closures are not 100%. I'll leave blogging about that until later.

Basically, much of the material pass (no illumination yet) of the game is rendering. It looks pretty raw and repetitive with respect to content right now. There's not yet much differentiation in city sections. No niceties such as shadows or even basic lighting yet. And, jeez, how am I gonna do trees? It's time, though, to switch gears for a bit and prototype the gameplay. As you might infer from the screenshots, the protagonist will be capable of flight. I've got my Xbox 360 controllers hooked up to my PC and I can fly all around town.

(Zaragosa, Mexico [fictional])


(Puebla, Mexico [real])


Please, I'm Just One Man!

So I must enlist the machine to generate a great deal of content automatically, albeit offline. This approach, of course, runs the risk of a high degree of repetition, but I think with some caution and tool refinement this will turn out great. I'm really happy with how auto-generation has gone so far. It's actually starting to vaguely resemble the residential city streets of central Mexico, which is, of course, the idea.



Anyhow, I'm pretty pumped about it. Given some time (lots) and little-by-little refinement, we may have something pretty cool here.

NCAA 08's Video Highlights

I'm pretty proud of this one. ;)

Thousands of user-created videos from NCAA Football 08!

http://www.easportsworld.com/#canvas_TopVideos

I posted this on 4/12/07:

I've been working on a little something special in NCAA Football 08. An NCAA Football 08 First Look article by GameSpot details the surprise. So now that it's public..

".. Once you've found a highlight you're particularly fond of, you can then choose to create a highlight of that play. This lets you choose from multiple camera angles to take a snapshot or segment of game-generated video. From there, you can either save these photos or videos on your hard drive or upload them to share with friends. .."

About the middle of last year, I was asked to do some R&D for this technology. Not long after that we were building it. Still, it came really close to being cut on a number of occasions. There were times when it seemed impossible to satisfy all of the below constraints and still produce a video of acceptable (or any) quality. I had taken some time off for a bit of surgery as NCAA was finalling and came back to find some bugs in my .flv encoder. So, I was a little paranoid that some random, unforseen issue might render the whole feature unusable in the shipped game.

Constraints

There were some wicked constraints putting this one together including -

• Encoding while rendering the game at 60fps/X360 and 30fps/PS3 without dropping frames.
• Keeping final file size small enough to not overwhelm the servers and to keep our bandwidth costs down.
• Final visual quality given the other constraints.
• Memory during encoding. Early in the cycle, I gave a rough estimate of 25MB free memory in-game to encode to .flv. It turns out we had to do it in under 500K in-game, and we did that by offloading some tasks until later and some other tricks I won't - and shouldn't - go into ;).

Compromises

• No sound. This one's hard to take but we just didn't have time. Remarkably, on the web, I haven't seen anyone complain about this. For next year.
• 15 frames per second. Yes, we wanted thirty, but we could only do that with a smaller frame size. Believe me, the trade off was well worth it. Also for next year.
• Video dimensions. We would have liked a larger frame size but we just didn't have the machine resources or network bandwidth to spare.

All in all, I think we struck a reasonable balance and it works. It's exciting to see people this jazzed about it! This technology is now quite the buzz around EA and next year we'll be seeing it in a lot more titles.

Will this technology and its descendants usher in the Web 2.0 era in console sports gaming?

I hope so!

Update Jan 3, 2008

Check out #2 on this list at pastapadre.com.

.. Launching with the release of NCAA Football 08 [EA Sports World] was an immediate success as fans of the game utilized the screenshot and video uploading features and it became one of the most positively responded to new additions to any sports game in recent memory. ...

For video highlights and optimizations geared at getting our football games to 60fps on PS3 starting with NFL Tour, I was nominated for the 2007 Outstanding Achievement in Engineering award for EA Sports - Tiburon. :)

A Developer's Life Stages

I ended up writing a novel as a comment on Jamie Fristrom's blog post on Speculative Generality. Read his entry. I'm reposting my comment here.

Developing The Simplest Thing That Could Possibly Work

In Forrester Research's 10 Mistakes in Software Development, #3 is overscoping a solution and #9 is jumping into development without enough research, and we've all done it. :(

A friend consulted at a large transportation company in the Kansas City area. They'd been gearing up for a massive conversion from their existing AS/400 systems to a completely Java-based system. Remarkably, in the early stages of the project, they decided to wrap the entire JDK in their own custom framework and decreed that all developers use these wrappers exclusively. As a result of this, they obtained -

• little or no added functionality,
• more limited abstractions than the raw JDK,
• minimally tested code underlying everything,
• the requirement to train all incoming Java developers on their framework and,
• worst of all, the obligation to maintain many more thousands of lines of code.

I've observed that developers tend to progress through a number of stages in their career. These are like the stages of grief: inescapable. This is because each stage produces a set of skills and philosophies that are eventually synthesized in the seasoned software developer.

The Underengineer

The underengineer is elated at his ability to get things done quickly and be productive, disdaining others (usually the overengineer) for what they see as excessive rumination and for producing an excessive amount of code for a given task. You can count on these guys to get stuff done. Will it be done 'the right way?' Only if by accident. Underengineers are often unable to discern 'the right way,' often because they've spent little time reading and maintaining other people's code.

A developer may begin to come out of this stage when he tires of fixing or writing the same code (in different places) over and over again. Or perhaps he'll be propelled into the second stage by an enthusiastic reading of Design Patterns or a sudden grasp of UML.

The developer at this stage is fascinated by accomplishing the task at hand and it is this stage in which he learns how to code and debug.

The Overengineer

The overengineer knows that he can solve any single development task and has become fascinated by possibility. Abstraction is magical. How can any piece of code be used to solve multiple tasks? Paradoxically, this usually ends up producing more code for any one task. While the underengineer shakes his head at this, the overengineer knows that his approach will produce much less code for an entire set of similar problems. Meanwhile, the overengineer looks at the underengineer's hackery with contempt. The question the overengineer rarely asks himself, though, is will there be a large enough set of similar problems to justify this effort? And so he abstracts everything in the name of flexibility.

The overengineer is sometimes shocked to realize he's become much less productive in terms of functionality than before. He can't seem to get anything done! Still, he produces as much (if not more) code and suddenly becomes capable of building much larger systems than he could as an underengineer.

This is the stage in which the developer learns architecture and the value (and painfully, the costs) of abstraction. This is the stage of the architects of the transportation system conversion discussed above.

What kills the overengineer? Maintenance. Dependencies. Broken abstraction. Realizing that the flexiblity he built in is unused and real systems change in ways he could never have anticipated. At some point, and probably many, the overengineer will get a phone call at 2am asking him to come in and fix an issue that breaks his whole architecture. After a few of these heartbreaking moments, his world view begins to change again.

The Seasoned Developer

The seasoned developer is skilled in coding and debugging from his time as an underengineer and is a skilled architect from his time as an overengineer. What's more, he's come to understand the following:

The primary reason for abstraction is to simplify implementation for a given set of requirements.

If abstraction doesn't simplify, ditch it.

The seasoned developer also realizes that the solution domain for a given problem may extend beyond technology. People often try to use technology to solve social or organizational problems. Conversely, people frequently attempt social or organizational solutions for essentially technological problems. Recognizing these incongruities before they become a software design and implementation can help to avoid doomed projects.

The organizational problem in the transportation example above is that software architects were assigned to do a job before anyone knew what job they were to do. The proper response would be to do nothing - or better, to work on a different project until the problem domain was fully understood.

Instead they chose speculatively general busy work, incurring a much higher cost.

The Future of Processor Hardware

Many-Core Processors, Languages for Multiple Processors and Reconfigurable Computing


Many-Core Processors

Multicore processors are giving way to many-core processors with Intel estimating the number of cores doubling with each processor generation. The paradigms of the most commonly used programming languages are ineffective with respect to these types of processors. So what will happen?

Languages for Multiple Processors

Modified C++/Java# with futures, promises, (Flow Java)?

Functional languages are inherently more parallelizable than imperative languages. Will we see a resurgence of these types of languages?
- Concurrent Haskell/Concurrent Clean
- F#

High-Performance Reconfigurable Computing

Hybrid systems combining conventional (Von Neumann) architectures with FPGAs and other forms of reconfigurable hardware are starting to mature, but with radically different models of "software" development.

Is High-Performance Reconfigurable Computing the Next Supercomputing Paradigm?

From Wikipedia

Paradigm Shift

It is called the Reconfigurable Computing Paradox, that by software to configware migration (software to FPGA migration) speed-up factors by up to almost 4 orders of magnitude have been reported, as well as the reduction of electricity consumption by more than 1 order of magnitude - although the technological parameters of FPGAs are are behind the Gordon Moore curve by about 4 orders of magnitude, and the clock frequency is substantially lower than that of microprocessors. The reasons of this paradox are due to a paradigm shift, and are also partly explained by the Von Neumann syndrome.

EASTL

EA genius and colleague Paul Pedriana published a paper to the C++ Standards Committee detailing his EASTL, an EA version of the Standard Template Library that provides a number of efficiencies with game development in mind, but that are nonetheless applicable across other software domains.

I've always been a fan of the STL in general, and since I've been at Electronic Arts, of EASTL. It's nice to see some of these innovations get out into the world.

Horrific Default Behavior/Failure Modes

(Updated 7/10/07)

Horrific Default Behavior

Today I wrote a batch file called clean.bat and put it in a directory on a different machine. It contained -

del /f /s /q *.obj
del /f /s /q *.ilk
del /f /s /q *.pdb

I had the directory up with Windows Explorer and '\\machine_name\c$\dev\..' was in the address bar. I saved the file into the directory of the other machine. Pretty straightforward, right?

So, thoughtlessly, I double click on clean.bat. Without ado, the batch file starts executing with this message -

'\\machine_name\c$\dev\..'
CMD.EXE was started with the above path as the current directory. UNC paths are not supported. Defaulting to Windows directory.

C:\WINDOWS>del /f /s /q *.obj

Holy crap! Fortunately, I saw it immediately and broke out of it. Even more fortunate, I don't think C:\WINDOWS has any .obj files.

But, if I had wanted to clean .exe files instead (which I frequently do), my machine would have been utterly hosed.

That, my friends, is horrific default behavior.

Failure Modes

Last year, our water heater died. It was pretty old, I think. The problem was the way that it died - by catching fire. That was pretty disturbing.

So the next day, while I was at work, the repairman came out to replace it. Rebekka asked him why it caught fire. The repairman explained, "that's how you know it's broken."Really? The next time my water heater fails, I'll know by following the fire trucks home to see my subdivision in cinders. "Time for a new water heater," I'll say to myself.

And how will I know when my nail clippers need replacing? Oh that's right, from the mushroom cloud over Orlando.

Building Materials #1

The Material Abstraction

In computer graphics, Material is a fundamental concept. Like a sheet of really special stretchy wallpaper, it is the collection of elements and properties that describe the visual characteristics of a surface. This encompasses texture, color, transparency, reflection, refraction and much more. Even so, it does not usually say anything about the geometry whose surface is described by a Material.

While Material is a useful abstraction for us, as software engineers and artists, it maps poorly to modern computer graphics hardware. The Material abstraction wildly cross-cuts a large number of subsystems in a modern renderer. As a result, most easy-to-use direct implementations of this concept you're likely to encounter are slow and/or rigid. That being the case, it's a great computer science problem to examine, and any solid, easy-to-use, reasonably fast and flexible solution is useful.

Relationship to Geometry

Let's define a low-level Geometry type as simply a list of vertices and an associated description of the vertex layout. Let's call a description of a vertex layout a Vertex Format. We'll talk about these vertex layouts at length later. The vertex layout is necessary because each vertex in a Geometry can be annotated with additional information, such as texture coordinates, the normal vector of the vertex, color, or pretty much anything you want.

Since the visual appearance of a geometric surface is described by a Material, any higher level geometric abstraction such as a mesh will refer to Materials as well as Geometries. An individual Geometry will be associated with a single Material, although multiple Geometries may point to the same Material.

Unfortunately, a Material needs to know the layout of the associated Geometry. In fact much of the time, the layout and the additional information included in each vertex in the Geometry is in determined by the choice of Material associated with it. This is an unpleasant reverse dependency that tightly couples Geometry creation with a Material.

Relationship to Textures

Here, let's broadly divide the Textures category into two parts - dynamic and static. Dynamic textures are those that can be written to and modified at runtime. An example might be a frame of a video during playback. Static textures are those that we probably load from disk. They don't ever change.

Textures may have different uses. The most common, of course, is that a texture represents an image to be wrapped around a surface. However, in the age of Shaders (see below), textures can represent any two+ dimensional array of data and are frequently used to store a vector (often a per-pixel normal vector) at each pixel.

A Material can refer to static and dynamic textures.

Relationship to Shaders and Shader Constants

Shaders come in different forms. Generally, there are vertex and pixel shaders. A shader is a small program that runs on the GPU for each vertex or pixel. These days, the classic distinction between the two is blurring. In the past an assembly language that was compiled into the hardware-dependent GPU machine code was the predominant way to program shaders. Currently, higher level languages such as HLSL and Cg are more common.

Shaders are tightly coupled with the vertex layout of Geometries and the number and intent of Textures in a Material. Shader Constants are the primary method of game-to-shader communication and as a result, they tend to impose a great deal of coupling between the shader design and the design of the code that invokes it. Some shader constants really are constant (PI, for example), some will be bound to attributes of the Geometry. Others may be related to world or game state. In a lighting shader, the constants may hold the location of the five nearest light sources. Shaders usually require that the current world transform be passed in as shader constants.

Applying a Material

So how do we apply a material? It depends on what the material specifies and a lot of other things as well. We'll look at some possibilities in the upcoming parts.

Render Components for a Given Material Pass

Compiling Haskell

I've played with the Glasgow Haskell Compiler only a bit. Compiling Haskell to, say, native code is quite a chore. The GHC compiles to a number of special intermediate languages between source and machine code.

Haskell > Core (Hindley-Milner typed lambda calculus) > STG > C/C-- > native machine code

That's a lot of work.

Small Tools

(Updated 7/10/07)

Part of the purpose of this blog is to remind myself of best practices I've discovered through the years I've been developing software. One of these best practices was brought to mind again in a recent conversation with a colleague.

Small, Combinable Tools

Unix, of course, is built around this philosophy. In essence, it is an extension of the practice of modular programming to applications. It allows us combine the functions of applications in an automatic or semiautomatic way. These are generally command-line tools, which may make them inconvenient in certain circumstances. Nevertheless, with a judicious layering approach, it is frequently possible to capture the best of both worlds - command-line combinable tools underneath, user-friendly GUI on top.

UnxUtils

If you use Windows and you develop software, it is absolutely imperative that you download and use UnxUtils. They are native Win32 ports of many standard unix utilities, and they can be extraordinarily useful. Add them to your PATH. Oh, and if you're not comfortable in unix and, by extension, with these utilities, let me encourage you strongly to learn to use them well. You'll thank me later!

Windows SysInternals

Windows SysInternals. You may not need these beauties often, but when you need them, you need them badly. I believe that few (or none) of these are command-line operable (I may be mistaken).

Windows Utilities

Junction for symbolic directory links in Windows. Get it.
PathMan for path management, and a ton of other good stuff here.
Unlocker, a fabulous (GUI) utility for unlocking files, by Cedric Collumb.

Build Utilities

Ant and NAnt for builds.

Freeing pov2mesh

My pov2mesh utility is currently distributed under the shareware model. I've pretty much decided to release it as freeware. If I do this, I will refund the registered users and remove the registration code from the application. Refunding will be easy, because there are so few registered users! I will not release the full source now, although I may in the future. I've also developed a number of other small tools that I will slowly be polishing (a little, not much) and releasing here sometime.

ASPack

($29) No, it's not a fanny pack. This executable packer is phenomenal.

Not So Small Tools

UltraEdit

($50) My primary text editor. It loads fast and I love its Find/Replace In Files. TextPad's a close second.

LTProf

($50) An easy-to-use, very lightweight profiler.

Araxis Merge

($129) I don't know how I'd live without it.

MilkShape

($35) Basic, inexpensive low-polygon 3d modeler than can import/export almost anything.

Ultimate Unwrap 3D

($50) Unwrap is great for texture mapping.

Paint Shop Pro

($90) Of course, as every game and web-developer who prefers Paint Shop Pro to Adobe's heavyweight (and expensive!) Photoshop knows. Now that Corel owns it, hopefully it will fare as well and not become as bloated (or expensive!) as Photoshop.

Gnu Prolog

Need a good Prolog implementation for expert system development or curiosity or anything else?

In the rare moments when I use Prolog, I use GNU Prolog.

The Impossible Dream #3

This is, of course, going incredibly slowly and that's just alright.

Script

I'm exploring an interesting architecture. I'm starting to use my script language Pair to script core parts of the game. Pair is Lisp/Scheme-like and very compact (the VM executable [compressed] is 28k without many external functions!). I designed Pair a few years ago with gaming in mind. The original intent was to use it to script many (hundreds or thousands) simultaneous events without having to make everything a state machine. Pair is multithreaded at the virtual machine level and not at the OS level, yielding very lightweight threads. To accomplish this, an explicit frame stack is maintained for function calls, and they are not implemented as C++ function calls underneath. Like Stackless Python which takes a similar approach (and is also used in gaming for its facility in concurrency), it means that Pair can (and does) support continuations because the call stack can be directly manipulated. One of these days I'll put up Pair and source.

On the Side

In the 80's in front of a house in the Kansas City, Missouri area a man built a boat. It was an incredibly tall half-built steel boat in this guy's yard. Every once in a while we'd drive by and see people working on it and a bit of progress here and there.

One day, my dad decided to stop and talk with the guy, with us little kids in tow. My brother and I took the opportunity to see the beast up close. And it was freaking amazing. To us, it might as well have been a Saturn V.

Many (and many of the best) software developers I've known have had little (or large!) projects on the side. Interviewing developers, I've sometimes asked about their side projects. It's a question that can yield great insight. Where does their passion lie? What problems attract them? What someone does with free time is what they really want to do, regardless of what they might let on in an interview.

It's an indicator of passion and of a drive to create. I've recently come to understand that these side projects also fuel the passion for our day-to-day work and inoculate against burnout. As developers, we have the power to create something from nothing. For some small portion of our time, to follow our own muse can be an exhilarating act.

Some companies, notably Google, have institutionalized this idea, giving developers a certain percent of time during office hours to work on their own projects. I'd expect Google to reap a significant innovative advantage with its policy.

What compels about software development is only partially solving puzzles and, for me, is about the joy and power of creation. Unfortunately, this power is, in part, illusion and the illusion is intensified by software's abstract nature. It nearly always seems far easier to write software than it actually turns out to be.

I'll demonstrate this illusion. Which requires more effort in man-hours: building a bridge across a medium-sized river or developing a game such as Spider-Man 3 or Madden 07? How about the Empire State building or Windows XP? I don't know the answers to these questions, but I don't believe that I can fully trust my intuition to provide them. In any event, good software requires enormous, and univerally underestimated, effort.

Maybe one of these days, I'll have another kooky neighbor, this time toiling away in his garage or barn building a Space Shuttle or something like in that goofy movie "The Astronaut Farmer" (which I have not yet seen). I'll know and he'll know and the fire department and the old ladies from the HOA will all know that it will never lift off. And I'll sit in the garage with that guy, and put the LOX and kerosene tanks together. And it will be freaking amazing.

(Updated 7/11/07)

(Rebekka and I watched The Astronaut Farmer last night and now I wish I hadn't referred to it. I'd hoped for something good, like Apollo 13 or October Sky but Astronaut Farmer was godawful. The premise was utterly absurd and the profoundly self-centered protagonist Charlie Farmer reminded me of an Allie Fox (from a great movie The Mosquito Coast) barely disguised under a veneer of sentimentalism. The best part? The rocket launches from their barn. As in a barn made of wood and filled with hay. And speaking of wood, during one of the bloopers, Billy Bob Thornton asks, in jest, if this was a film directed by Ed Wood. I was wondering the same thing.)

Senzee's Developer Interview Test, Q. 1

You have two hours to complete this exam.

1.) Without looking at the following questions, quickly estimate how long it will take you to finish this test correctly. Please do not let your estimate exceed the time limit given for this test.

..

..argh..

Moving Parts

This is where I intend to catalog (all?) the components of a commercial game. I will periodically update this post.

Why?

I've bought and read many books on game development. The bad ones are generally 1000 pages of C++ source code. The good, useful ones tend to fall into two categories.

• Narrow and deep (books that equate 'game engines' and 'rendering engines', for example)
• Broad and shallow (books that contain light sections on rendering, sound, networking, data structures, etc.)

Still, even books in the broad category aren't that broad. The reason? Books can only be so thick. My point really is that the practice of game development includes a vast array of subspecialties and I'd kinda like to enumerate them here. Hopefully with input from my readers (that's you, Mom) that I'll add. So this list will be ultra-broad with no depth whatsoever. Depth will have to wait for subsequent posts.

Now, not every game will have or need all these systems, but it's crazy just how many most games will need. So, for right now, this is a brain dump and I've only just begun..

Offline

Content Creation - tools - geometry, mesh creation
Content Creation - tools - geometry, mesh creation - offline CSG tools
Content Creation - tools - image creation
Content Creation - tools - motion capture cleaning
Content Creation - tools - higher level 'world' assembly, including assigning gameplay attributes to objects
Content Creation - tools - sound creation/mixing/production
Content Creation - tools - mission/level creation
Content Creation - tools - statistical data, skill/inventory data and attributes
Content Creation - tools - state machine creation
Content Creation - tools - script debuggers, interactive interpreters for testing
Data Management - managing generated code
Data Management - asset dependency management
Data Management - geometry processing - ie. adding attributes, triangle stripification
Data Management - texture processing - texture packing, format conversions, bit depth conversion, etc.
Data Management - asset databases (SQL, XML)
Data Management - metadata collection about assets, data mining for global performance enhancement, data mining for content creation automation, etc.
Data Management - version control systems
Code Management - Code Generation - from asset databases, other offline sources
Code Management - Build System - standard way of building the game for all developers
Code Management - Library/Component Management - system(s) for cataloging, versioning and incorporating libraries and reusable components
Code Management - version control systems
Testing Systems and Methodologies

(much more offline/pipeline side to come; the game pipeline gets far less attention than it deserves, often with disastrous results)

Runtime

System - threading libraries/primitives
System - error handling, logging, assert handling
System - string handling, regular expressions
System - elementary data structures, STL, etc.
System - fast math primitives
I/O - File - standard file i/o
I/O - File - file compression/decompression
I/O - File - loading game assets (meshes, textures, scripts, game data files, etc..)
I/O - File - low-level game asset caching
I/O - File - asset 'packaging' into larger files
I/O - File - object serialization support
I/O - Network - lobby
I/O - Network - server side
I/O - Network - event handling (see Events below)
I/O - Network - publishing player data to online services
I/O - Network - web access services
Localization Support - text databases
Memory - efficient game asset caching
Memory - memory pooling systems
Memory - garbage collection systems
Events - event handling/dispatching systems
Events - 'replayable' game state management
Events - game state represented as loadable/saveable data stream
Events - game state policies - random/deterministic, etc..
Events - game 'flow' management
Sound - Mp3, WAV, Ogg, etc. playback
Sound - runtime mixing, filters
Sound - in game capture
Sound - stereo, 5.1 surround sound
Sound - 3d audio
Input - console controller
Input - force feedback (output)
Input - spatial (Wii, SIXAXIS)
Input - mouse
Input - keyboard
Input - other (Guitar Hero, EyeToy, DDR, Donkey Konga)
Graphics - rendering - geometry processing
Graphics - rendering - geometry processing - progressive meshes and LODs
Graphics - rendering - geometry processing - efficient spatial subdivision
Graphics - rendering - shader(s), per pixel lighting
Graphics - rendering - scene graph management
Graphics - rendering - GPU based utilities for non-graphics
Graphics - rendering - geometry processing - in game CSG operations
Graphics - geometry processing - human animation
Graphics - geometry processing - motion capture
Graphics - rendering - decaling
Graphics - rendering - particle systems
Graphics - rendering - dynamic skyboxes/skydomes
Graphics - images - loading/encoding image formats - DDS, TGA, JPG, etc.
Physics - collision detection
Physics - collision detection - geometry processing
Physics - collision handling, mesh processing for non-rigid object effects
Physics - ragdoll animation
Physics - wind, environmental effects
Gameplay/AI - script
Gameplay/AI - state machine systems
Gameplay/AI - classical AI algorithms
Gameplay/AI - events - collisions, sphere of influence effects, NPC line of sight, NPC earshot
Gameplay - Cars - Damage Models
Gameplay - Cars - Driving Models
Geometry - Modeling
Geometry - Inverse Kinematics
Geometry - Rigging/Tagging/Markup
Geometry - Animation
Geometry - Procedural Animation (ie. Cloth, Hair, Jelly/Spring Models, etc..)
Geometry - Facial Performance and Lip Sync'ing
Rendering - Textures
Rendering - Shaders
Rendering - Animated textures/maps
Video - codec(s)/playback - MPEG, MJPEG, WMV, VP6, Bink, Madcow, etc..
Video - color space conversion
UI - rendering
UI - event handling systems, architecture and data
UI - script
UI - some games have a large amount of UI (ie. Madden, with its enormous UI)
UI - HUDs
Data Management - embedded SQL database (not just for plain-old IT anymore)
Embedded Control - script languages
Control - in-game gameplay tweaking utilities, runtime assets updating support

(more..)

Online Game Services

Online - lobby
Online - server side
Online - (deterministic) player position prediction and tracking (online multiplayer)
Online - publishing player data to online services
Online - web access services
Online - billing tracking for online services (online multiplayer)
Online - player data storage
Online - accessibility from PC website

(more to come, but I'm sleepy now and I want to go to bed..)

Optimal Bytecode Interpretation

I always feel like I should blog more about engineering related stuff, since that's what I do all day.

For now, my earlier plans for bytecode to C++ compilation are on hold as we're looking into more immediate gains from aggressively optimizing the interpretation. One of the downsides of the hybrid C++ compilation/bytecode interpretation approach that I have in mind is that it will require a workflow change. The C++ will eventually need to be compiled and then linked to the executable. There's a pipeline/workflow bubble there that's not appealing. Still, I think it would be fantastic to get that built for critical core libraries of script (utilities) that don't change much.

Optimizing Interpretation

Our bytecode is that of a dynamic, and (as a result of the structure of the instruction set) terribly slow language. Unfortunately, we use an off-the-shelf compiler, so we can't infer much that was in the original source but fails to show up in the bytecode output.

Did some memoization, in this case caching of variable and function lookups, with excellent (awesome!) results.

A clever universal hashing scheme for both special strings and user strings opened up a lot of possibilities.

Close attention paid to string handling yielded good results.

Thread synchronizaton primitives were killing us. We usually try to keep our assembly language to a bare minimum for portability sake. This translates to a handful of primitives to improve PS2 performance. But to rid ourselves of our most time consuming locks, we applied asm to a couple of critical spots where we need to read in, modify, then write out whole cache lines as atomic operations on the PowerPC machines (PS3, 360). This avoids the expense of a more heavyweight lock such as a mutex in those places and gives us back a lot of performance for multithreaded operation.

Anyhow, we've significantly improved the bytecode execution performance, which is something the game teams have been asking (begging!) for for ages.

Of course, it's never fast enough.. ;)

Bytecode to Native Compilation

At work, some of the software I develop uses a bytecode interpreter. And, as always, we need better performance from the whole system. So I'm looking into bytecode-to-native (in this case C++) compilation. I've done this before, with embedded Lisp-based languages and there are a number of compilers available that do this for Java (GCC has a back-end for this), C#, Lisp and its derivatives (Bigloo for Scheme, for instance). Compiling to C or C++ is great as it serves as a sort of portable assembly language and it's possible to leverage further the fine optimization skillz of modern C/C++ compilers. I'll report on this when I make some progress - if I don't get pulled off onto something else.