Depiction and Representation

“Let Me Bore You About Videogames” — Part Two

Depiction and representation are terms from the philosophy of art which we can use to talk about videogames.

What can a videogame show us, and how? Can videogames show us things that other storytelling media can’t? What’s Final Furlong got to do with it?

Namco’s Final Furlong (1997)

The concepts of depiction and representation

I’m cribbing this mainly from Image and Mind: Film, Philosophy and Cognitive Science by Gregory Currie. Chris Crawford also has a good, long discussion about this stuff in the context of games.

Artefacts can be used to refer to things. For example, a written word, ”KANGAROOS”, can be used to refer to kangaroos, or a picture of a kangaroo can be used to refer to kangaroos.

Some referring artefacts — like written words — demand interpretation according to learned rules or conventions; others — like pictures — simply resemble the thing they refer to.

Recognising that a picture of a kangaroo is of a kangaroo calls on just the same capacity as recognising that a kangaroo is a kangaroo. That is, if you can recognise a kangaroo by sight, you can recognise that a picture of a kangaroo is of a kangaroo. The ability to recognise that the word "KANGAROOS" refers to kangaroos is, in contrast, disconnected from the ability to recognise that really encountered kangaroos are kangaroos.

When we grasp what a picture resembles in the same way that we recognise the resembled thing itself, the picture depicts its referent. We can only grasp the referent of a word, however, through learned rules or conventions, which are removed from our capacity to recognise the referent itself, so the word represents the referent.

Really simplistically: something represents when you have to know how to interpret it; something depicts when you can just tell what it is intuitively.

Quick, get back to videogames

Now we can differentiate expressive media in terms of what they can depict. A painting can depict the experience of seeing a horse. A song can’t. A song can represent seeing a horse, but not depict that. A song could depict hearing a horse, which a painting can’t depict. (It would be a weird song.) More generally, images can depict visual experience, and sounds can depict auditory experience. Cinema, then, can depict both visual and auditory experience.

We can go further. Cinema can depict extended movements, because a film can lead us to judge that a fictional horse is running around in the same way that we judge a horse to be running around in real life. A still picture can’t depict extended movement. A comic strip can represent extended movement, but only thanks to conventions about how to read one panel after the other, so that is not depiction.

Is depiction better than representation? Sometimes. Chris Crawford gives a good example:

In general, I think it’s safe to say that most spatial problems are better communicated with depictional images than with representational words. For example, imagine yourself at a particularly tight moment in Doom, with monsters all over the place, spitting fireballs and shooting and clawing at you. An image or two can communicate this desperate situation with perfect clarity, but a text description of the situation would be tedious and slow to digest. Indeed, could you imagine Doom played in real time with a pure text display? “The door opens to reveal four imps, two cacodemons, and a hell knight. The first imp is 12 meters away and 28 degrees to the left of your sightline; the second imp is 15 meters way and 22 degrees to the left of your sightline…” Before you finish reading this tedious description, you’d be dead!

Chris Crawford in Representation Versus Depiction

He goes on to point out that representation is used for the health meter and inventory in Doom, since there, text communicates simply and quickly. He also makes an aside which misses the point slightly, in an informative way:

But this does not justify leaping to the conclusion that depiction is always superior to representation. Try depicting depreciation, or libertarianism, or overload, or indigestion. Sure, you can probably come up with some long-winded, tedious sequence of icons or images the get the idea across, but will they really communicate the notion faster than the word itself? I think not.

That “tedious sequence of icons or images” would not depict. If a picture of a kangaroo refers to a kangaroo, then that is depiction. If I tell you that the same kangaroo picture means “horse” then that is representation and not depiction. Things don’t depict just by being pictures; they depict by being recognised with recourse to the same capacities which would enable us to identify the thing depicted. (OK, my simplistic kangaroo silhouette may be more of a representational symbol than a depiction, so a bad example, but if you think that then you understand, so we’re alright. Good job.)

So is this about videogames now?

Images can depict some stuff; sounds can depict some other stuff; films can depict some pretty interesting stuff. What can games depict? Just as film gets to include all the depictive potential of images and sounds, games get all that and more. Once we see what else games can do, we can start asking which games play to the strengths of the medium.

There are three types of thing that videogames — uniquely among mainstream media — can depict: control and decision, physical interaction, and social interaction.

Control and decision

Nintendo’s Super Mario Bros. (1985) Manual (Image source)

How do I come to think that I, as Mario, decide to run right, and not that I just do so by chance or predetermination? I do so not by reading the manual, but by using my normal, natural capacities that come into play when I determine that actions have consequences, that different actions can be chosen between, and that I have made a decision to do one thing and not another. That’s depiction, and depicting decisions is possible in no mainstream medium besides videogames.

Consider a prose story written in the second person: “You look at your email again and decide to make more coffee.” I can interpret that and imagine that it is my decision to make coffee. I even start filling in some gaps — that perhaps my emails are boring and I am procrastinating. It’s effective and engaging, but this is a mere representation of a decision, where videogames can depict one.

Can we counter that games don’t depict decisions, but actually just contain decisions? A 1:2.4-scale plastic model of a horse is not a horse — it just depicts one — but a decision to run right in a game, really is a decision to run right. No problem: imagine a picture of a room, on the wall of which is a picture of a horse. Now we have a picture of a picture of a horse. That depiction of a picture of a horse in fact just is a picture of a horse. That’s true, but it clearly also depicts a picture of a horse — which we can determine from its context (hanging on the wall of a room). The same goes for my decision playing Super Mario Bros. It is a decision, but it also depicts a decision.

Physical interaction

Capcom’s Steel Battalion (2002) controller, for realistic mech piloting. (Image source)

You see! This is about Final Furlong! It’s interesting that specialist videogame hardware (steering wheels, bongos, fishing rods, skateboards, horses, etc.) is readily accepted as being part of the medium, whereas rumbling cinema seats, sprayed water, and air jets are dismissed as odd quirks bolted onto the core of what cinema really is, and reserved for amusement parks. Some arcade games simply couldn’t exist without a plastic horse.

Anyway, turning a wheel is like turning a wheel, and that experience is depicted by the game when a wheel is involved, but only represented where you use a control stick to move hands on a wheel on the screen. Videogames can depict physical interactions like this, and do so uniquely among mainstream expressive media.

I’ll probably go on about it in a separate post, but Ico’s hand-holding tug is a masterful example of depicted physical interaction.

Social interaction

Social interaction in multiplayer games is not just simulated; it’s real. Talking to a friend while watching a film is also real, but interacting with other players is part of a game, and in many games it’s an essential part.

The real social interactions in games can be used to depict fictive social interaction in a story. The social relations of ally, rival, or opponent can readily be both real and part of the fictional world of the game. Tabletop roleplaying games share this capacity for the depiction of interpersonal interaction, but I’ll continue my theme and say this capacity is unique to videogames among mainstream expressive media.

To show how social activity in a game can be both real and fictive, consider the possibility of playing a game in an antisocial way. Certain behaviours, such as out-of-character verbal abuse are clearly antisocial, but clearly not part of the game. Where the game ends can be hard to pin down, but other antagonistic, antisocial behaviours could be legitimate parts of a game in appropriate circumstances: for example, forming gangs, holding grudges, gloating, or crashing a funeral and murdering all the mourners.

In contrast, you can’t watch a film in an antisocial way unless you do something other than watch the film, like throwing popcorn or kicking a seat-back. Mores, norms, and community are all within reach of games’ depictive powers.

Where we can go next

Representation in any expressive medium is flexible and powerful, and only limited by the imagination of the artist, but the varying capacities for depiction are essential and inescapable features of artistic media. Music and paintings cannot depict the same things; dance and sculpture cannot depict the same things; cinema and installation art cannot depict the same things.

Videogames have unique capacities for depiction. They can depict decision and control (choosing actions), physical interaction (riding a horse), and social interaction (cooperation and competition), in addition to sounds, images, movement, change, etc.

If I have a normative thesis around all this, it’s that good works of art should demonstrate awareness of the potential of their medium, and use some or all of the medium’s resources skilfully without carelessly neglecting others. Observing the depictive capacities of videogames is how we start to see when games are pushing the boundaries of the medium, and when they are conspicuously stopping short of the boundaries as a means of artistic expression. It’s another way to work out why great games are great.

Examples will have to come in another post. I have so much more to say about games it’s maddening. Why did I even start writing about books?

🎮💡