CaC Image-Based Rubric Revamp Statement
August 12, 2020 | Effective September 1, 2020
Fellow fakers,
For some time now, the image-based CaC rubric has been the subject of various concerns, mainly stemming from inequities in how it treats beginner fakers vis-à-vis veterans. These inequities are real; I've noticed many of them myself from the judging end of things, and they've been brought to light time and time again by various members of the community hoping for change.
With these and similar concerns in mind, I've spent a great deal of time over the last month-and-a-half looking at potential ways to revamp the rubric. After careful discussion with the current text-based judge @FourteenAlmonds and with the veteran image-based judge @bbninjas, I believe we've reached an optimal solution.
You can read my letter explaining our reasoning for changing the rubric, as well as the changes we intend to make and how we intend to implement them, below (spoiler'd for length):
Some recent events have caused me to pay closer attention to the way we’ve historically judged things with the rubric, and I believe increasingly, for image-based in particular, that it is far from a perfect system. I want to preface the rest of this statement by saying that I know not everyone agrees with any one view on this, and it’s my hope that whatever system we end up using in the future, that it have the support of as many people as possible.
Furthermore, we’ve used the current system for a long time—since 2015, as a matter of fact, for image-based; that was when we shrank C&O from 20 points to 15, and added the Aesthetics category. Why should we change now?
In part, I’m bringing it up now because of some discussions we had a couple of months ago that thrust some issues with the CaC into the limelight. In particular, those discussions highlighted that the current rubric may not be an accurate reflection of the things we value most in card-faking, and also may not treat all entrants equally, disproportionately punishing “beginner mistakes” and disproportionately rewarding long years of experience with Photoshop or similar applications. Let’s break some of that down.
I would say, with more than three years of CaC judging behind me, that the two ideals we strive for most in card-faking are (a) creative innovation of new effects, or combinations of effects, in exciting and/or thought-provoking ways; and (b) adherence to the general conventions, or at least staying within certain parameters of plausibility, set by the Pokémon TCG over the years.
These ideals hold true both for text-based and image-based fakes — and in image-based fakes, they also hold relevance to a card’s aesthetics; the way the faker chooses to present the card visually. I’m going to call them Innovation and Adherence to avoid confusion, but they’re really Creativity and Believability; two sides to a coin that have to be balanced in making the ideal fake.
Innovation and Adherence get most clearly represented in C&O and B&P, respectively, but they’re inherent in other categories as well. Wording and F&P are pretty heavy Adherence checks, as they deal with making sure you’re all referencing actual cards for the nitty-gritty details of your effects. You’re allowed some leeway for custom blanks, rewarding Innovation there with (usually) full credit, but the main purpose of those categories is to check Adherence. And then there’s Aesthetics, which certainly deals with both, but is predominantly to reward aesthetic Innovation.
At first glance, based on how these supercategories fall into place, they seem pretty well-balanced. 20 points to Innovation (C&O and Aesthetics) and 30 to Adherence (Wording, F&P, and B&P) — that’s the same spread you see in text-based.
But the supercategories have subdivisions, and they are clearly not given equal weight. Balancing, which I would argue is the most important part of adherence to conventions, is only given 5 points!
What can we do about this? We could just weight every category equally — 10 points to each, and that would keep the 20/30 split you see above. But there’s something in me, maybe just an old, outdated judging ethos, that makes me think certain categories (C&O, for one) should be worth more than that.
This also brings me to the other point from the previous discussions — that beginners are disproportionately punished and old-timers disproportionately rewarded. Yeah, there’s some sense to that, seeing as people who have put years of hard work into improving themselves as image-based fakers should naturally get higher scores, but honestly? That’s not what we see.
Take F&P, for example. This one category can make or break a CaC score. If entrants don’t have the right connections to get a cheap-’n’-easy font guide, they have to spend hours making one by hand — or, often, just doing DIY placements for every single card. Who can blame anyone for not wanting to do that, and just resorting to drag-and-drop eyeballing instead?
In contrast, we have “Believability and Playability,” the category that represents the incredibly important skill of accurately balancing a card for a particular era or metagame, being worth only 5 points. Beginners frequently get full credit for this category, even if they get a relatively low score overall due to the other factors, but those 5 points don’t make much of a difference, in the end. On the other hand, old pros sometimes make mistakes with balancing, making a card that actually ends up being quite over- or under-powered, but because they generally nail the wording, fonts, and such, they still get scores in the mid-40s. In other words, as a category, B&P does not matter very much compared to the others.
That’s not to say that there’s no purpose to 5-point categories; I would say that Aesthetics as a 5-point category works very well. It’s kinda a nice bonus for people who do put in the time to do fancy holosheets or pretty custom blanks. But it doesn’t break the score for people who don’t.
Now I’ve got a proposal, so as to rebalance the categories in a way that makes sense. I’ve put them in order from greatest number of points to least, though there’s also an argument to be made for grouping them by “Innovation-heavy” and “Adherence-heavy” categories.
Essentially, I’ve just taken 5 points from each of “Wording” and “Fonts and Placements” and assigned them to “Believability and Playability”. This keeps the 20/30 split for Innovation and Adherence, which I think is a good idea to keep, as it has historically worked well for the text-based rubric.
Ultimately, this rubric looks a lot closer to the text-based rubric overall — you could look at it like the text-based rubric, with 5 points taken out of C&O for Aesthetics and 5 points taken out of Wording for F&P. Personally, I think that’s a much better way to do it.
This proposal reflects C&O and B&P being the categories most intrinsically linked to our core ideals of Innovation and Adherence. It reserves 15 points for technical specs, and 5 points for the “aesthetic bonus,” which I think is fair. One could also argue that F&P is more tied to Aesthetics (just a more technical side of it) than it is to Wording, and I think that’s also well reflected in this spread.
How would this new distribution of points be judged?
Well, B&P would become a more nuanced category, for one thing. I judge C&O with a set of “markers” for what each point value means — like, previously seen effects with a fun (often synergistic) twist tend to rate about a 10/15. I have a similar set of markers for B&P, but they’re not nearly as detailed. If the category expanded to 15 points, I would expand my own private criteria to more closely match C&O in scope.
Often, docking more than 5 points from F&P happens because the entrant just eyeballed all their placements. Again, I do not believe this should be punished as much as it has been historically. Under the new system, incorrect placements will cap out at a 5 point deduction. (In general, though, I plan to be more lenient with this and generally encourage people to make their own DIY font guides or seek them out from the community.)
Wording won’t change much; it’s very rare that anyone earns less than 5/15 under the current system anyway.
And that's it! Those are the details on what we're changing, why we're changing it, and how we're going to implement it. If there's anything else I can explain from this section, don't hesitate to ask!
Furthermore, we’ve used the current system for a long time—since 2015, as a matter of fact, for image-based; that was when we shrank C&O from 20 points to 15, and added the Aesthetics category. Why should we change now?
In part, I’m bringing it up now because of some discussions we had a couple of months ago that thrust some issues with the CaC into the limelight. In particular, those discussions highlighted that the current rubric may not be an accurate reflection of the things we value most in card-faking, and also may not treat all entrants equally, disproportionately punishing “beginner mistakes” and disproportionately rewarding long years of experience with Photoshop or similar applications. Let’s break some of that down.
I would say, with more than three years of CaC judging behind me, that the two ideals we strive for most in card-faking are (a) creative innovation of new effects, or combinations of effects, in exciting and/or thought-provoking ways; and (b) adherence to the general conventions, or at least staying within certain parameters of plausibility, set by the Pokémon TCG over the years.
These ideals hold true both for text-based and image-based fakes — and in image-based fakes, they also hold relevance to a card’s aesthetics; the way the faker chooses to present the card visually. I’m going to call them Innovation and Adherence to avoid confusion, but they’re really Creativity and Believability; two sides to a coin that have to be balanced in making the ideal fake.
Innovation and Adherence get most clearly represented in C&O and B&P, respectively, but they’re inherent in other categories as well. Wording and F&P are pretty heavy Adherence checks, as they deal with making sure you’re all referencing actual cards for the nitty-gritty details of your effects. You’re allowed some leeway for custom blanks, rewarding Innovation there with (usually) full credit, but the main purpose of those categories is to check Adherence. And then there’s Aesthetics, which certainly deals with both, but is predominantly to reward aesthetic Innovation.
At first glance, based on how these supercategories fall into place, they seem pretty well-balanced. 20 points to Innovation (C&O and Aesthetics) and 30 to Adherence (Wording, F&P, and B&P) — that’s the same spread you see in text-based.
But the supercategories have subdivisions, and they are clearly not given equal weight. Balancing, which I would argue is the most important part of adherence to conventions, is only given 5 points!
What can we do about this? We could just weight every category equally — 10 points to each, and that would keep the 20/30 split you see above. But there’s something in me, maybe just an old, outdated judging ethos, that makes me think certain categories (C&O, for one) should be worth more than that.
This also brings me to the other point from the previous discussions — that beginners are disproportionately punished and old-timers disproportionately rewarded. Yeah, there’s some sense to that, seeing as people who have put years of hard work into improving themselves as image-based fakers should naturally get higher scores, but honestly? That’s not what we see.
Take F&P, for example. This one category can make or break a CaC score. If entrants don’t have the right connections to get a cheap-’n’-easy font guide, they have to spend hours making one by hand — or, often, just doing DIY placements for every single card. Who can blame anyone for not wanting to do that, and just resorting to drag-and-drop eyeballing instead?
In contrast, we have “Believability and Playability,” the category that represents the incredibly important skill of accurately balancing a card for a particular era or metagame, being worth only 5 points. Beginners frequently get full credit for this category, even if they get a relatively low score overall due to the other factors, but those 5 points don’t make much of a difference, in the end. On the other hand, old pros sometimes make mistakes with balancing, making a card that actually ends up being quite over- or under-powered, but because they generally nail the wording, fonts, and such, they still get scores in the mid-40s. In other words, as a category, B&P does not matter very much compared to the others.
That’s not to say that there’s no purpose to 5-point categories; I would say that Aesthetics as a 5-point category works very well. It’s kinda a nice bonus for people who do put in the time to do fancy holosheets or pretty custom blanks. But it doesn’t break the score for people who don’t.
Now I’ve got a proposal, so as to rebalance the categories in a way that makes sense. I’ve put them in order from greatest number of points to least, though there’s also an argument to be made for grouping them by “Innovation-heavy” and “Adherence-heavy” categories.
Creativity and Originality: 15 points
Believability and Playability: 15 points
Wording: 10 points
Fonts and Placement: 5 points
Aesthetics: 5 points
Believability and Playability: 15 points
Wording: 10 points
Fonts and Placement: 5 points
Aesthetics: 5 points
Essentially, I’ve just taken 5 points from each of “Wording” and “Fonts and Placements” and assigned them to “Believability and Playability”. This keeps the 20/30 split for Innovation and Adherence, which I think is a good idea to keep, as it has historically worked well for the text-based rubric.
Ultimately, this rubric looks a lot closer to the text-based rubric overall — you could look at it like the text-based rubric, with 5 points taken out of C&O for Aesthetics and 5 points taken out of Wording for F&P. Personally, I think that’s a much better way to do it.
This proposal reflects C&O and B&P being the categories most intrinsically linked to our core ideals of Innovation and Adherence. It reserves 15 points for technical specs, and 5 points for the “aesthetic bonus,” which I think is fair. One could also argue that F&P is more tied to Aesthetics (just a more technical side of it) than it is to Wording, and I think that’s also well reflected in this spread.
How would this new distribution of points be judged?
Well, B&P would become a more nuanced category, for one thing. I judge C&O with a set of “markers” for what each point value means — like, previously seen effects with a fun (often synergistic) twist tend to rate about a 10/15. I have a similar set of markers for B&P, but they’re not nearly as detailed. If the category expanded to 15 points, I would expand my own private criteria to more closely match C&O in scope.
Often, docking more than 5 points from F&P happens because the entrant just eyeballed all their placements. Again, I do not believe this should be punished as much as it has been historically. Under the new system, incorrect placements will cap out at a 5 point deduction. (In general, though, I plan to be more lenient with this and generally encourage people to make their own DIY font guides or seek them out from the community.)
Wording won’t change much; it’s very rare that anyone earns less than 5/15 under the current system anyway.
And that's it! Those are the details on what we're changing, why we're changing it, and how we're going to implement it. If there's anything else I can explain from this section, don't hesitate to ask!
TL;DR: From the September 2020 round of the CaC onward, the image-based rubric will change to reflect a desire to make the contest more fair and accessible to everyone. The intent is to reward good balancing more, and reward font guides and wording-check know-how less. The new rubric is as follows:
Creativity and Originality: 15 points
Believability and Playability: 15 points (+10)
Wording: 10 points (–5)
Fonts and Placement: 5 points (–5)
Aesthetics: 5 points
Believability and Playability: 15 points (+10)
Wording: 10 points (–5)
Fonts and Placement: 5 points (–5)
Aesthetics: 5 points
The text-based rubric will not change, and the old/current image-based rubric will still be used as planned for the July 2020 round's results.
As always, we welcome thoughts and discussion around this! CaC would be nothing without our fantastic faking community, and so your input in terms of how we run it matters a great deal. Let us know your thoughts either in this thread or via PM, and with your help, we hope to make this contest as fair and accessible as it can be.
Jabberwock
