Pair #1

Finally, I'm putting up what I have for Pair, my script language intended for homebrew game projects. Pair is designed to be a small, embedded, acceptably fast (probably not yet the case) script language that will ultimately allow multiple concurrent VM (and not OS-based) threadlets, generators and continuations.

I don't have a lot of time on my hands, so stuff like this takes quite a long while. Now, there's some messed up stuff going on here. You'll have to forgive me that or wait until it's refined more. For example, scoping isn't right at all. The (let ..) expression isn't how it's supposed to be in a lisp-type language. It defines a variable for the rest of the outer scope which isn't right. I'll fix that. Continuations aren't currently supported, but the underlying machinery is mostly there as well as the mechanism for running multiple lightweight VM threads although this is not accessible from the command line program. Which is probably good, since I'm not sure how well that works with the garbage collector at this point. Also, there's a lot of just general messiness going on. There are a bunch of warnings at compile time and comments are sparse. These things will be fixed eventually. I've compiled these with Visual C++ .NET 2003. There are two parts here.

Version 0.01.00

The Pair Compiler
The Pair VM

See the Code Use Policy if you are interested in using any of this for your own purposes.

The Impossible Dream #4

(Updated 11/23/2007)

Where Have I Been?

It's been months since I've blogged. They've been productive months, though. See, in early October, another beautiful baby girl joined our family! It's been a rush having another child. It's incredible how you can still be completely awestruck as if it were the first time, each and every time. :)

Prior to that, vacation.

Because I had some time off, I slipped in some hours on my Impossible Dream. I hadn't really considered blogging about it yet until I stumbled across Project Steel & Glass at The Cluttered Desk that motivated me to write an update.

Before leaving on vacation this summer, I'd made some headway with Pair, my embedded script language. Especially, I'd made some strides compiling it to more efficent bytecode, etc. It's in a sort-of usable state. Still, some things like lexical closures are not 100%. I'll leave blogging about that until later.

Basically, much of the material pass (no illumination yet) of the game is rendering. It looks pretty raw and repetitive with respect to content right now. There's not yet much differentiation in city sections. No niceties such as shadows or even basic lighting yet. And, jeez, how am I gonna do trees? It's time, though, to switch gears for a bit and prototype the gameplay. As you might infer from the screenshots, the protagonist will be capable of flight. I've got my Xbox 360 controllers hooked up to my PC and I can fly all around town.

(Zaragosa, Mexico [fictional])


(Puebla, Mexico [real])


Please, I'm Just One Man!

So I must enlist the machine to generate a great deal of content automatically, albeit offline. This approach, of course, runs the risk of a high degree of repetition, but I think with some caution and tool refinement this will turn out great. I'm really happy with how auto-generation has gone so far. It's actually starting to vaguely resemble the residential city streets of central Mexico, which is, of course, the idea.



Anyhow, I'm pretty pumped about it. Given some time (lots) and little-by-little refinement, we may have something pretty cool here.

Trampolines

Developing embedded languages for applications in the past, the most time consuming part always seemed to be wrapping the multitude of foreign language (ie, to/from C++) function calls of interest with script-compatible functions.

One of the goals of Pair is to make working with other languages - especially C/C++ - very easy. So here we'll take a different approach. This time we'll generate trampolines to call foreign functions (for example, from a DLL) and do the boxing/unboxing of values for Pair to use. Trampolining has a number of different meanings but many of them center around the idea of generating code at runtime to call a function with special requirements that aren't known until runtime. In this case, we're directly generating the machine code necessary to set up the stack frame and call a function using either the __cdecl (for the C runtime library) or __stdcall (for the Win32 SDK functions) calling conventions.

This allows us to do things like the following -

(let
  ((beep  
     (import "kernel32.dll" stdcall uint Beep   (uint uint)))
   (system
     (import "msvcrt.dll"   cdecl   uint system (string))))

(system "dir c:\\root")
(beep 500 1000))

C++ code -

nativecaller.h
nativecaller.cpp

Building Materials #1

The Material Abstraction

In computer graphics, Material is a fundamental concept. Like a sheet of really special stretchy wallpaper, it is the collection of elements and properties that describe the visual characteristics of a surface. This encompasses texture, color, transparency, reflection, refraction and much more. Even so, it does not usually say anything about the geometry whose surface is described by a Material.

While Material is a useful abstraction for us, as software engineers and artists, it maps poorly to modern computer graphics hardware. The Material abstraction wildly cross-cuts a large number of subsystems in a modern renderer. As a result, most easy-to-use direct implementations of this concept you're likely to encounter are slow and/or rigid. That being the case, it's a great computer science problem to examine, and any solid, easy-to-use, reasonably fast and flexible solution is useful.

Relationship to Geometry

Let's define a low-level Geometry type as simply a list of vertices and an associated description of the vertex layout. Let's call a description of a vertex layout a Vertex Format. We'll talk about these vertex layouts at length later. The vertex layout is necessary because each vertex in a Geometry can be annotated with additional information, such as texture coordinates, the normal vector of the vertex, color, or pretty much anything you want.

Since the visual appearance of a geometric surface is described by a Material, any higher level geometric abstraction such as a mesh will refer to Materials as well as Geometries. An individual Geometry will be associated with a single Material, although multiple Geometries may point to the same Material.

Unfortunately, a Material needs to know the layout of the associated Geometry. In fact much of the time, the layout and the additional information included in each vertex in the Geometry is in determined by the choice of Material associated with it. This is an unpleasant reverse dependency that tightly couples Geometry creation with a Material.

Relationship to Textures

Here, let's broadly divide the Textures category into two parts - dynamic and static. Dynamic textures are those that can be written to and modified at runtime. An example might be a frame of a video during playback. Static textures are those that we probably load from disk. They don't ever change.

Textures may have different uses. The most common, of course, is that a texture represents an image to be wrapped around a surface. However, in the age of Shaders (see below), textures can represent any two+ dimensional array of data and are frequently used to store a vector (often a per-pixel normal vector) at each pixel.

A Material can refer to static and dynamic textures.

Relationship to Shaders and Shader Constants

Shaders come in different forms. Generally, there are vertex and pixel shaders. A shader is a small program that runs on the GPU for each vertex or pixel. These days, the classic distinction between the two is blurring. In the past an assembly language that was compiled into the hardware-dependent GPU machine code was the predominant way to program shaders. Currently, higher level languages such as HLSL and Cg are more common.

Shaders are tightly coupled with the vertex layout of Geometries and the number and intent of Textures in a Material. Shader Constants are the primary method of game-to-shader communication and as a result, they tend to impose a great deal of coupling between the shader design and the design of the code that invokes it. Some shader constants really are constant (PI, for example), some will be bound to attributes of the Geometry. Others may be related to world or game state. In a lighting shader, the constants may hold the location of the five nearest light sources. Shaders usually require that the current world transform be passed in as shader constants.

Applying a Material

So how do we apply a material? It depends on what the material specifies and a lot of other things as well. We'll look at some possibilities in the upcoming parts.

Render Components for a Given Material Pass

The Impossible Dream #3

This is, of course, going incredibly slowly and that's just alright.

Script

I'm exploring an interesting architecture. I'm starting to use my script language Pair to script core parts of the game. Pair is Lisp/Scheme-like and very compact (the VM executable [compressed] is 28k without many external functions!). I designed Pair a few years ago with gaming in mind. The original intent was to use it to script many (hundreds or thousands) simultaneous events without having to make everything a state machine. Pair is multithreaded at the virtual machine level and not at the OS level, yielding very lightweight threads. To accomplish this, an explicit frame stack is maintained for function calls, and they are not implemented as C++ function calls underneath. Like Stackless Python which takes a similar approach (and is also used in gaming for its facility in concurrency), it means that Pair can (and does) support continuations because the call stack can be directly manipulated. One of these days I'll put up Pair and source.

On the Side

In the 80's in front of a house in the Kansas City, Missouri area a man built a boat. It was an incredibly tall half-built steel boat in this guy's yard. Every once in a while we'd drive by and see people working on it and a bit of progress here and there.

One day, my dad decided to stop and talk with the guy, with us little kids in tow. My brother and I took the opportunity to see the beast up close. And it was freaking amazing. To us, it might as well have been a Saturn V.

Many (and many of the best) software developers I've known have had little (or large!) projects on the side. Interviewing developers, I've sometimes asked about their side projects. It's a question that can yield great insight. Where does their passion lie? What problems attract them? What someone does with free time is what they really want to do, regardless of what they might let on in an interview.

It's an indicator of passion and of a drive to create. I've recently come to understand that these side projects also fuel the passion for our day-to-day work and inoculate against burnout. As developers, we have the power to create something from nothing. For some small portion of our time, to follow our own muse can be an exhilarating act.

Some companies, notably Google, have institutionalized this idea, giving developers a certain percent of time during office hours to work on their own projects. I'd expect Google to reap a significant innovative advantage with its policy.

What compels about software development is only partially solving puzzles and, for me, is about the joy and power of creation. Unfortunately, this power is, in part, illusion and the illusion is intensified by software's abstract nature. It nearly always seems far easier to write software than it actually turns out to be.

I'll demonstrate this illusion. Which requires more effort in man-hours: building a bridge across a medium-sized river or developing a game such as Spider-Man 3 or Madden 07? How about the Empire State building or Windows XP? I don't know the answers to these questions, but I don't believe that I can fully trust my intuition to provide them. In any event, good software requires enormous, and univerally underestimated, effort.

Maybe one of these days, I'll have another kooky neighbor, this time toiling away in his garage or barn building a Space Shuttle or something like in that goofy movie "The Astronaut Farmer" (which I have not yet seen). I'll know and he'll know and the fire department and the old ladies from the HOA will all know that it will never lift off. And I'll sit in the garage with that guy, and put the LOX and kerosene tanks together. And it will be freaking amazing.

(Updated 7/11/07)

(Rebekka and I watched The Astronaut Farmer last night and now I wish I hadn't referred to it. I'd hoped for something good, like Apollo 13 or October Sky but Astronaut Farmer was godawful. The premise was utterly absurd and the profoundly self-centered protagonist Charlie Farmer reminded me of an Allie Fox (from a great movie The Mosquito Coast) barely disguised under a veneer of sentimentalism. The best part? The rocket launches from their barn. As in a barn made of wood and filled with hay. And speaking of wood, during one of the bloopers, Billy Bob Thornton asks, in jest, if this was a film directed by Ed Wood. I was wondering the same thing.)

The Impossible Dream #2

Previous: The Impossible Dream #1

Ground Rules

Mantra 1: The mechanic first, then the game.

It's no fun putting blood, sweat and tears into an unfun game. We structure the game around a solid mechanic - prototype first to find addictive key element(s) and elaborate. Tag is one key element we've identified.

Mantra 2: Build a game, not an engine.

I've seen a number of home and indie game projects go south stuck at the "let's build our engine" phase. Nothing overly general - we build into the code what we need to do the trick. I'm not saying there's no place for useful abstraction here, I'm saying that a general purpose rendering or game engine is not our goal.

Mantra 3: Fit the design to resources at hand.

Building an open world game at all is our first violation of this. Still, we need to embrace this reality instead of fighting it. Operating with restricted resources can force us to come up with creative solutions to problems we otherwise wouldn't have looked at. Of course, it may also prevent us from finishing anything ever.

Mantra 4: Don't underengineer or overengineer, but if you must choose one, underengineer.

Why? Because underengineering is easier to fix.


A Unflinching View

Let's start the impossible dream by looking, somewhat realistically, at our available resources and the strengths and weaknesses of our position.

Resources

• A few development hours a month divided among game design, software development and content creation.
• A powerful target platform (PC).
• Development tools - Visual C++ 2003.
• Direct3D and its auxiliary libraries (see No need for portability in Strengths, below) .
• Third party content creation tools - Milkshape 3D (registered), Unwrap3D (registered), POV-Ray.
• Open source and free resources on the web for game development.
• Free content from other sources that can be legally used in our game.

Strengths

• Strong software development skills.
• A powerful target platform (PC).
• No need for portability.
• A great deal of freely available software for offline and runtime use.

Weaknesses

• Poor content creation ability.
• Few hours to dedicate to the project - it'll be done in like one hundred years.
• Little monetary investment.

Strategies

Make the most of our particular set of resources:

• Simplicity - a simple, compelling core game. No minigames, nothing like that.
• Prototype the game mechanic first.
• Automation - machine generated content, mostly offline, possibly at runtime.
• Reuse/scavenge (to a point) freely available content generation tools and software.
• Reuse/scavenge (to a point) freely available software for the runtime.
• Seek out free content that we can legally use in game.
• Creatively adapt the game design, including character and plot, to cater to our strengths.
• Use a hooded figure for the main character, allowing us to skirt the animation issue for the main character. We can use a simple segmented model for NPC's. We can animate the hood & clothing of the main character with cloth animation, which is CPU-intensive, but nearly content-free.
• Use public domain (classical, folk music?) music for the soundtrack.

Oh, and yes, there's so much more to come.

The Impossible Dream #1

True Story

In Mexico I met a poor farmer named Miguelito who lived in the little town of Metepec (Atlixco), Puebla. Many years before, in a vision, the Holy Virgin of Guadalupe entrusted him to build a cathedral, high up on the hill above the town. She promised him that he would not die until he'd set the final stone. He quickly enlisted the help of the townspeople and over 15 years constructed a small, beautiful hilltop church. At the time Miguelito showed me the rust-colored church it was nearly complete and he explained that this meant that his time was near. His mistress had been generous enough - Miguel was 104 years old!

My Impossible Dream - My Own Open World Game

So, is it possible, working a few scant hours each week on evenings and weekends to build a (simple) open-world game by myself? We shall see. I've occasionally been working on the game since about a year before I came to EA but I think I'll make a more public spectacle out of it on this blog - to keep me motivated.

So why do I even want to do this? Because it's impossible.

Some prototype screenshots
Some generated content and concepts

Next Time: in the next installment of The Impossible Dream, we'll look at what this game will be about and how we will use automatic content generation to build our world.