If NetFlix Rents Games?

I really love NetFlix. So much that I've considered ditching many of the DVDs I own. And since starting NetFlix, I have bought very few.

NetFlix renting games has been rumored since its inception. Other services, such as GameFly have stepped in to try to become the NetFlix of games. But while NetFlix is mainstream, I suspect that GameFly's subscriber base is far from it - and it is an additional charge and an additional service if you already have NetFlix. Unlike movie makers, game makers don't have an equivalent to a 'box office' take. So if NetFlix were to ever decide to rent games, while initially great for the consumer, I believe it would profoundly impact game developer's revenue. Which would, in the end, be bad for the consumer given the skyrocketing cost of game development.

So how do we, as game developers, develop a NetFlix-proof business model? Is it all MMOG's from here on out?

PlayStation 3 Cell Came Before Its Time

The Microsoft Xbox 360 platform is now profitable and Sony's PlayStation 3 is gaining parity with the Xbox 360 in the marketplace. We're about halfway through this console cycle (these cycles typically last about five years) and we can reflect on technology choices with the benefit of some hindsight. The ramifications of some of the more obvious choices, for example PS3's Blu-Ray vs. 360's plain-old-DVD, are well known. Microsoft's background as a operating system and compiler developer provides the 360 with perhaps the best development tools ever to grace a non-desktop system. We'll look at that in another entry. We'll also want to compare the Xbox 360 GPU and the PlayStation 3 GPU.

For now, however, I'd like to look at the Cell. In doing so, I'd like to ask and try to answer the question, "Why has the quality of PlayStation 3 titles lagged behind that of the Xbox 360 titles?"

The Cell Processor is Amazing

From the info I gathered, the PS3's Cell should have a peak numerical performance of about 205 gigaflops. From Wikipedia, I gather that the 360, on the other hand, has peak performance of 116 gigaflops. The PS3's Cell has one regular PowerPC core (with two hardware threads) called the PPU. It also has seven SPUs (one is reserved for the OS), which are streamlined vector processors that are fast as hell, although functionally limited compared to the main PowerPC core. They have their own memory at 256K apiece. Half of that is available to the developer. The 360's Xenon processor, on the other hand, consists of three conventional PowerPC cores almost identical to the Cell's PowerPC core, for a total of six hardware threads. So, if you're like most people, you're probably thinking, "Wow, the PS3 blows the 360 away!"

Except that it doesn't.

Developing Games is Hard

I just read this at .mischief.mayhem.soap about the size of the Assassin's Creed team. Actually, I think that's about the same for Madden NFL. Modern games require an enormous amount of content, some of which must be reflected in code. Games on multicore consoles are much more difficult to debug. Making games is simply much more expensive that it used to be. I heard a colleague remark that he saw that one PlayStation 1 version of Madden had 30 files. Now, it's thousands of files, millions of lines and scores of developers.

Developing for the Cell is Hard

When we first saw the specs for the Cell, I thought, "it looks like it was designed to play movies." And in playing movies, the PS3 will blow the 360 away. The Cell was designed for performing the same kind of vector calculation on large numbers of small chunks of data. Essentially like a GPU, but more flexible although less parallel. There's a lot of this sort of thing in games. Some physics, audio decoding/encoding, movie decoding/encoding, some graphics/geometry processing. Still, there are a lot of things that aren't well suited for this model. Among those would be language interpretation (we do a lot), gameplay code, sparse data manipulation, and a lot of really mundane and common software engineering things that are inherently sequential. Sorting a list of pointers that must be dereferenced to be compared, for example.

Explanations of why the 360 performs so well compared to the PS3 usually start out like, "Well, the Cell looks better on paper, but.." Developing for a 360 is like developing for a multicore PC. Take your existing code, multithread it and know that it'll work on the 360 and upcoming PC-like multicore machines with little changes.

You can build PS3 software exactly the same way using only the PPU and its two threads, but that will yield almost exactly one-third of the power of the 360's processor. In fact some early titles that were developed first for the 360 did just this. This was not as bad as it sounds though. A lot of early code for these machines was not thoroughly multithreaded because much of that code was directly ported from older Xbox and PS2 code. In these cases, it used just a portion of the 360's power anyway.

Now we're pushing the capabilities of the 360 and trying hard to get the PS3's six fast but special-purpose SPUs to do what we can do with the four remaining general-purpose hardware threads on the 360. It's hard, though certainly possible to achieve parity. Still, it requires a lot more code and expertise just for the PS3 (usually from our best developers), which typically means much less overall developer productivity.

The Cell Looks Like the Future

Unfortunately, as delightful as it is to develop for, the approach taken with the 360's Xenon processor simply can't scale. At some point, with some number of traditional cores, memory access and synchronization are going to break down or become so contentious that any benefit of a larger number of cores will be nil. So the Cell looks like the future. A small number of traditional cores with a large number of stream processors can yield phenomenal performance in the right proportions. I predict that some member(s) of the next console generation will have a processor with 8 traditional cores and perhaps 256+ stream processors. 256 sound ridiculous? Consider that an ATI HD 2900 XT GPU has 320 stream processors.

Why do I think the Cell was premature? It did not fulfill the possibilities of traditional multicore processor before moving on to a hard-to-develop-for, though potentially more powerful, architecture.

In any event, it's clearly a foreshadow of processor architecture to come. Still, the difficulty of achieving parity between the Cell and 360 Xenon when game development is already at its most taxing really isn't doing the PlayStation 3 any favors.

Fable 2 News (GDC08 Preview)

In San Francisco last week (more on that later), I heard Peter Molyneux talk about Fable 2. See, I love the first Fable, and I'm really excited about this new arrival.

He announced four new features which are on Wikipedia already, so for many of the details you'll want to head there. As a part of the demo, Peter demonstrated his onscreen counterpart, The Molynator who, like a solid 10% of the audience, was female. Fable 1 had just one possible (male) protagonist. He announced that The Molynator married (after meeting at a courting spot, or something like that) five years ago. They'd found a home in the slums of Bowerstone early on, then moved to a country residence a few years later. The young couple had a young child in that time. Yes, Molyneux said, she experienced all the joys of virtual pregnancy, including the growing belly.

Pregnancy

He said he didn't know if the big belly would actually make it in the game. Something about a nine-month pregnant woman slicing open a dragon or troll with a broadsword strikes me a little inauthentic. Especially if you've ever seen (or been) a nine-month pregnant woman trying to get out of a chair. Also, might it be bad form to have a pregnant woman defeated in battle?

Imagine you choose to have a family in Fable 2. Molyneux strongly hinted that your family will become a part of the narrative. He glossed over this one really fast, though. Your children, your spouse kidnapped, murdered? Avenged? Who knows?

Co-op play inspired by LEGO Star Wars, but much more dynamic. Rebekka and I can play Fable 2 together! So now we won't have to worry about her getting that big belly again. ;) Well, until we finish it. He didn't say anything about the length. Hope it's longer than Fable 1.

There's no overhead map. Instead, there's an innovative breadcrumb trail showing you where you've been most.

Check this out. No money for completing quests? Arghh. You either have to gamble or be hired as a henchman to make money. That just doesn't make any sense to me. I have to get a job to make money? That's way too much like the Sims. Which means way too much like real life. I'm not much of a gambler either. The weird thing is I already have a job. It's the one that I'll use to pay for Fable 2. And I like it more than gambling or being a henchman.

One of the gambling games, Keystone - a roulette like casino game - will be coming onto Xbox Live prior to the release of Fable 2 so that you can make your money before you need to spend it.

I'm excited to play it. I can wait though. I've still got Assassin's Creed and BioShock to work through (not necessarily in that order), so The Molynator's got plenty of time.

More info on the presentation.

Xbox 360, PS3 Games Unplayable on Future Hardware?

Earlier I posted about concurrency being a crucial issue in today's and tomorrow's games.

Recently, my son has been playing the original Xbox game Thrillville. The game is solid on our original Xbox, but crashes with disturbing frequency on the Xbox 360, particularly during loading. This leads me to suspect that the timing differences between the original Xbox and its emulation on the 360 exposed threading issues that were present but simply did not occur because of the timing characteristics of the original console. Console game developers generally test exclusively for the single configuration per target platform. The only way we find threading issues, it seems, is through testing. So if testing fails to reveal real issues, games are shipped with them. This is mitigated, of course, by developing games for multiple multithreaded platforms (ie, Xbox 360 and PlayStation 3) with the same code base.

Emulating Concurrent Code

This is one kink in a larger problem. The problem is developing a body of 'literature,' whether games or other software, that will be usable on future machines. Take, for example, the classic game Ultima VII. It is a beautiful and vastly influential role-playing game rendered unplayable by modern machines because of its hardware voodoo. (We also may occasionally need to use software such as old versions of WordPerfect, VisiCalc or Lotus 1-2-3 - and perhaps from non-PC machines - to read crucial, but old, data stored in those formats.) For such situations we typically resort to emulators such as the excellent DOSBox or AppleWin for maximum compatibility.

It is very difficult, if not impossible, to maintain timing-level compatibility in emulators. Older games that relied heavily upon machine-specific timing are now pretty much broken in emulators. For that matter, they're generally broken when the next generation of hardware arrives. Multithreaded code that contains hidden bugs (which is pretty much all code, multithreaded or otherwise) is implicitly and heavily timing-dependent. It's one thing to emulate an Xbox game, with typically very low levels of multithreading on an Xbox 360, and may be quite another to emulate Xbox 360 games on an Xbox 720.

As a result, I suspect that far more Xbox 360 and PlayStation 3 games will be unplayable on future hardware than Xbox and PS2 games on the 360 and PS3.

NCAA 08's Video Highlights

I'm pretty proud of this one. ;)

Thousands of user-created videos from NCAA Football 08!

http://www.easportsworld.com/#canvas_TopVideos

I posted this on 4/12/07:

I've been working on a little something special in NCAA Football 08. An NCAA Football 08 First Look article by GameSpot details the surprise. So now that it's public..

".. Once you've found a highlight you're particularly fond of, you can then choose to create a highlight of that play. This lets you choose from multiple camera angles to take a snapshot or segment of game-generated video. From there, you can either save these photos or videos on your hard drive or upload them to share with friends. .."

About the middle of last year, I was asked to do some R&D for this technology. Not long after that we were building it. Still, it came really close to being cut on a number of occasions. There were times when it seemed impossible to satisfy all of the below constraints and still produce a video of acceptable (or any) quality. I had taken some time off for a bit of surgery as NCAA was finalling and came back to find some bugs in my .flv encoder. So, I was a little paranoid that some random, unforseen issue might render the whole feature unusable in the shipped game.

Constraints

There were some wicked constraints putting this one together including -

• Encoding while rendering the game at 60fps/X360 and 30fps/PS3 without dropping frames.
• Keeping final file size small enough to not overwhelm the servers and to keep our bandwidth costs down.
• Final visual quality given the other constraints.
• Memory during encoding. Early in the cycle, I gave a rough estimate of 25MB free memory in-game to encode to .flv. It turns out we had to do it in under 500K in-game, and we did that by offloading some tasks until later and some other tricks I won't - and shouldn't - go into ;).

Compromises

• No sound. This one's hard to take but we just didn't have time. Remarkably, on the web, I haven't seen anyone complain about this. For next year.
• 15 frames per second. Yes, we wanted thirty, but we could only do that with a smaller frame size. Believe me, the trade off was well worth it. Also for next year.
• Video dimensions. We would have liked a larger frame size but we just didn't have the machine resources or network bandwidth to spare.

All in all, I think we struck a reasonable balance and it works. It's exciting to see people this jazzed about it! This technology is now quite the buzz around EA and next year we'll be seeing it in a lot more titles.

Will this technology and its descendants usher in the Web 2.0 era in console sports gaming?

I hope so!

Update Jan 3, 2008

Check out #2 on this list at pastapadre.com.

.. Launching with the release of NCAA Football 08 [EA Sports World] was an immediate success as fans of the game utilized the screenshot and video uploading features and it became one of the most positively responded to new additions to any sports game in recent memory. ...

For video highlights and optimizations geared at getting our football games to 60fps on PS3 starting with NFL Tour, I was nominated for the 2007 Outstanding Achievement in Engineering award for EA Sports - Tiburon. :)

Fun, Fun, Fun

..until her daddy takes her GTA away?

One of the things that have become even more abundantly clear in the transition to next-generation consoles is how the concept of fun is quite divorced from what we consider a quality game. Quality games must be fun, of course, to a point. They don't have to be too fun, though, and originality in this arena carries an enormous and intolerable financial risk. High-end games must be beautiful to look at, slick, contain vast quantities of content. Frequently, they require online experiences. These types of games must sell millions to recoup development costs. Because risk is so enormous, there is a lot of copycating of successful models.

Let me provide an example. I loved GTA III, liked GTA Vice City, found GTA San Andreas boring and True Crime, Saint's Row and even Crackdown dull, dull, dull. Fun usually requires novelty (or nostalgia), and there's no more novelty left in stealing cars and ramming them into other cars and pedestrians until they explode (the cars and the pedestrians).

(I am a sucker for licensed content, though. I hate to admit it, but it's true.)

Of course, this is something that everyone involved with games knows. But sometimes it's useful to state what's already widely understood.

(Updated 7/10/07)

Grand Theft Auto IV

Ok, I know what I said about GTA-type games getting boring. Still, GTA IV looks awesome. And of course I'll have to play it.

If I get bored by it I guess I'll only have myself to blame..

Moving Parts

This is where I intend to catalog (all?) the components of a commercial game. I will periodically update this post.

Why?

I've bought and read many books on game development. The bad ones are generally 1000 pages of C++ source code. The good, useful ones tend to fall into two categories.

• Narrow and deep (books that equate 'game engines' and 'rendering engines', for example)
• Broad and shallow (books that contain light sections on rendering, sound, networking, data structures, etc.)

Still, even books in the broad category aren't that broad. The reason? Books can only be so thick. My point really is that the practice of game development includes a vast array of subspecialties and I'd kinda like to enumerate them here. Hopefully with input from my readers (that's you, Mom) that I'll add. So this list will be ultra-broad with no depth whatsoever. Depth will have to wait for subsequent posts.

Now, not every game will have or need all these systems, but it's crazy just how many most games will need. So, for right now, this is a brain dump and I've only just begun..

Offline

Content Creation - tools - geometry, mesh creation
Content Creation - tools - geometry, mesh creation - offline CSG tools
Content Creation - tools - image creation
Content Creation - tools - motion capture cleaning
Content Creation - tools - higher level 'world' assembly, including assigning gameplay attributes to objects
Content Creation - tools - sound creation/mixing/production
Content Creation - tools - mission/level creation
Content Creation - tools - statistical data, skill/inventory data and attributes
Content Creation - tools - state machine creation
Content Creation - tools - script debuggers, interactive interpreters for testing
Data Management - managing generated code
Data Management - asset dependency management
Data Management - geometry processing - ie. adding attributes, triangle stripification
Data Management - texture processing - texture packing, format conversions, bit depth conversion, etc.
Data Management - asset databases (SQL, XML)
Data Management - metadata collection about assets, data mining for global performance enhancement, data mining for content creation automation, etc.
Data Management - version control systems
Code Management - Code Generation - from asset databases, other offline sources
Code Management - Build System - standard way of building the game for all developers
Code Management - Library/Component Management - system(s) for cataloging, versioning and incorporating libraries and reusable components
Code Management - version control systems
Testing Systems and Methodologies

(much more offline/pipeline side to come; the game pipeline gets far less attention than it deserves, often with disastrous results)

Runtime

System - threading libraries/primitives
System - error handling, logging, assert handling
System - string handling, regular expressions
System - elementary data structures, STL, etc.
System - fast math primitives
I/O - File - standard file i/o
I/O - File - file compression/decompression
I/O - File - loading game assets (meshes, textures, scripts, game data files, etc..)
I/O - File - low-level game asset caching
I/O - File - asset 'packaging' into larger files
I/O - File - object serialization support
I/O - Network - lobby
I/O - Network - server side
I/O - Network - event handling (see Events below)
I/O - Network - publishing player data to online services
I/O - Network - web access services
Localization Support - text databases
Memory - efficient game asset caching
Memory - memory pooling systems
Memory - garbage collection systems
Events - event handling/dispatching systems
Events - 'replayable' game state management
Events - game state represented as loadable/saveable data stream
Events - game state policies - random/deterministic, etc..
Events - game 'flow' management
Sound - Mp3, WAV, Ogg, etc. playback
Sound - runtime mixing, filters
Sound - in game capture
Sound - stereo, 5.1 surround sound
Sound - 3d audio
Input - console controller
Input - force feedback (output)
Input - spatial (Wii, SIXAXIS)
Input - mouse
Input - keyboard
Input - other (Guitar Hero, EyeToy, DDR, Donkey Konga)
Graphics - rendering - geometry processing
Graphics - rendering - geometry processing - progressive meshes and LODs
Graphics - rendering - geometry processing - efficient spatial subdivision
Graphics - rendering - shader(s), per pixel lighting
Graphics - rendering - scene graph management
Graphics - rendering - GPU based utilities for non-graphics
Graphics - rendering - geometry processing - in game CSG operations
Graphics - geometry processing - human animation
Graphics - geometry processing - motion capture
Graphics - rendering - decaling
Graphics - rendering - particle systems
Graphics - rendering - dynamic skyboxes/skydomes
Graphics - images - loading/encoding image formats - DDS, TGA, JPG, etc.
Physics - collision detection
Physics - collision detection - geometry processing
Physics - collision handling, mesh processing for non-rigid object effects
Physics - ragdoll animation
Physics - wind, environmental effects
Gameplay/AI - script
Gameplay/AI - state machine systems
Gameplay/AI - classical AI algorithms
Gameplay/AI - events - collisions, sphere of influence effects, NPC line of sight, NPC earshot
Gameplay - Cars - Damage Models
Gameplay - Cars - Driving Models
Geometry - Modeling
Geometry - Inverse Kinematics
Geometry - Rigging/Tagging/Markup
Geometry - Animation
Geometry - Procedural Animation (ie. Cloth, Hair, Jelly/Spring Models, etc..)
Geometry - Facial Performance and Lip Sync'ing
Rendering - Textures
Rendering - Shaders
Rendering - Animated textures/maps
Video - codec(s)/playback - MPEG, MJPEG, WMV, VP6, Bink, Madcow, etc..
Video - color space conversion
UI - rendering
UI - event handling systems, architecture and data
UI - script
UI - some games have a large amount of UI (ie. Madden, with its enormous UI)
UI - HUDs
Data Management - embedded SQL database (not just for plain-old IT anymore)
Embedded Control - script languages
Control - in-game gameplay tweaking utilities, runtime assets updating support

(more..)

Online Game Services

Online - lobby
Online - server side
Online - (deterministic) player position prediction and tracking (online multiplayer)
Online - publishing player data to online services
Online - web access services
Online - billing tracking for online services (online multiplayer)
Online - player data storage
Online - accessibility from PC website

(more to come, but I'm sleepy now and I want to go to bed..)

Explorers and the Game of Tag

I'm partial to games of exploration. The Ultimas and their relatives were my early favorites. These days, I play games like Grand Theft Auto, Godfather, Spider-Man 2 and Oblivion.

Why? It's the joy of discovery - the pleasure of finding things out. So what makes a great open world game? In the context of video games, an open world is a toy upon which a game can be built. So first, what makes a great open world?

Big, not too big

It's no secret that we measure the size of our universe in the time it takes to cross it. The world shrinks with the inventions of the railroad, the automobile and the airplane. Unlike other open world games, the world of Superman Returns seems in turns enormous and tiny, depending on whether you are walking or flying at supersonic speeds. When a world is too big, we can spend all of our time simply getting around. Too small and it seems cramped and unimaginitive.

There must be some novelty to every distinct area or region. The 1987 game The Faery Tale sported a world of over 17,000 screens. There were so many repeating patterns of landscape and it took so long to cross that it was hard to stay interested in exploring the world. In an interview with Game Informer, Radical, in creating Hulk: Ultimate Destruction, specifically addressed this issue of variety.

Mechanics

The mechanics of a world are crucial to its experience. They are the laws of the game's universe. It is important that the game itself cannot subvert these rules. In GTA, this includes the way cars move collide and interact, that any car can be rigged with bombs and the general behavior of the people and objects that inhabit Liberty City. Another important mechanic in GTA is police response to player actions. The world of GTA is rich with mechanics largely orthogonal in purpose. It is this richness that makes GTA great. In Spider-Man 2, the central mechanic is web-swinging. The world of Spider-Man 2 is designed to be experienced through web-swinging and so it is an inseparable element of Treyarch's Manhattan. In GTA, Spider-Man 2 and Superman Returns the speed at which one is able to move through the city is variable and its 8control is in some mechanic anchored in the game world itself. This provides a sense of enormous freedom. In some of these games, this speed increases as the player progresses through the game.

Even so, at this level, the open world is still a toy. Now let's ask the question, what makes a great open world game?

The Childhood Game of Tag

Of course, GTA 3 is the prototypical open world game. Recently, I started playing it yet again and it still may be the best. What makes it fun? The underlying element is tag. The ancient children's game is incredibly simple, infinitely flexible and has strategic depth. Tag benefits from richness and complexity in a large world. The ideal tag world has nooks and crannies in which to hide from predators and from which to ambush prey. Layered on top of that, in GTA, there a number of other game mechanics, many - most? - of which are themselves variations on tag.

These layers of tag and alternating roles of hunter and hunted in GTA 3 stand in contrast to Spider-Man 2's races which also aim to capitalize on a large in-game world. Unfortunately, racing is much less fun than tag. Even though it's one of my favorite games, Spider-Man's races don't appeal to me. What appeals to me about Spider-Man 2 is the absolute euphoria of swinging and the crazy missions (levels?) like Mysterio's burning theater and the Statue of Liberty.

Tag (from Wikipedia)

Players: 2+
Rules complexity: Low
Strategy depth: High
Random chance: Low
Skills required: Running, Hiding, Observation

Some People I've Met (and Some People I Haven't) Whose Work I Like

Earlier I mentioned Richard Garriott as someone I'd like to meet. Jamie Fristrom emailed in response to that and sent this interesting story.

In junior high, I discovered Hackers, by Steven Levy which I must have read a dozen times. Steve Wozniak became my hero. I wanted to build computers.

Then it was games. I wrote to all of the programmers who converted Times of Lore to PC asking how they accomplished the overhead scrolling over the massive world. Herman Miller responded with a detailed description in C and assembly and recommended that I get the Programmer's Guide to PC and PS/2 Video Systems which I guess is going for $0.24 on Amazon these days. Armed with that information, the book and Turbo C, I started writing games like crazy.

So in college, a friend, Alex Kapadia, and I conceived a game called Undead: The Tibovian Era. I sent an early copy to the fast-growing shareware company Epic MegaGames and a few days later, received an excited phone call from the now legendary (in game circles) Tim Sweeney. Tim sent me an email (which in those days I dutifully checked on the Vaxen at UMKC) saying, "I'm looking forward to seeing more of Undead! It's the best 'first' game any author has shown us." Not bad, eh? This was a year or two before Unreal and I like to believe that my Undead inspired the similar name of Tim's magnum opus. Who knows..

Instead of finishing Undead, I spent a few years in Mexico. Kids these days. When I returned to Kansas City I found employment for a while doing bilingual customer service at a well known electronics company. At that time, I found myself sitting across the aisle from Niki Sullivan, one of the original members of Buddy Holly's Crickets! In his honor, the call center played nothing but Buddy Holly. Like an idiot, in my months there I was too awestruck to ever ask him anything about the days he wandered around the country participating in some of the seminal moments of American pop music.

And, of course, I've met a number of equally talented people in places I've worked such as Keane, VML, H&R Block, Sprint and EA Tiburon.

Without further reflection, that's the story of some people I've met (and some people I haven't) whose work I like.