This is a series of loose meditations around the purpose and nature of "rules" in D&D-like RPGs. What I say here could probably be extrapolated out to things like PbTA but that's not my focus here. These tweets by Ben Milton were the catalyst for these thoughts.
I'm starting to see all RPGs as lying somewhere on an axis between "players use rules to overcome rules-based problems" and "players use in-world objects to overcome in-world problems." There's probably a better way to say that.— Questing Beast (@benjamilt) April 21, 2020
I found this to be kind of a surprising statement considering that it leaves out "players use rules to overcome in-world problems", especially given that one of Ben's pet RPG mechanics, item slots (I don't think I've ever seen him miss an opportunity to suggest switching to a slot-based inventory system if the opportunity presents itself) to me typifies exactly that problem solving mode....but the problem-solving mindset between the two modes is very different. However, most RPGs include both things at once, and I find that increasingly frustrating— Questing Beast (@benjamilt) April 21, 2020
Let's unpack these statements though. What Ben is pointing out essentially is the difference between non-diegetic and diegetic design; however, in any medium, no work is wholly diegetic, as every text must include the extradiegetic level of presenting the work to the reader*. In RPG terms, this is simply because the "in-world" space does not actually exist**, and therefore requires a manner of ingress into the fictive world, some extradiegetic device that allows us to access the intradiegetic world: in literature, this is printed text, in film, cinematography, and in RPGs, it is discussion.
This is the first "rule", as in extradiegetic device, of RPGs, then: that by talking about a fictional world, we are allowed to enter it. A rule is a point of ingress into the fictional world.
The problem with RPGs, however, is that they are collaborative, and as such authority over the creation of the intadiegetic world must be distributed and maintained. Thus, we usually come to a second necessary rule to ingress into a fictional world, one that distributes authorial power in some way, such as: one person is the final arbiter of what happens in the intradiegetic space***. Without this, what you're liable to end up with is something like that of children's playtime gone wrong: "I attack you with my laser", "I use my shield", "you don't have a shield", "do too!", "it doesn't work against my laser though, it's an anti-shield laser!", "well my shield is a special anti-laser shield!", etc. The above exchange, as we can see, can not ingress into the intradiegetic space and takes place wholly on an extradiegetic level.
Our hypothetical RPG with only these two rules would occupy the lattermost position on Ben's axis, having only minimal extradiegetic elements. The problem with this type of game, obviously, is that as a game increases in complexity of things that can not be reasonably simulated through simple conversation (say, carrying items, engaging in combat, travelling through several miles of terrain), it becomes increasingly burdensome to resolve conversationally, and the difficulty of ingress into the intradiegetic world increases****. We create specific rules for these scenarios, then, to allow us to ingress into the intradiegetic world for that specific activity or task.
Item slots are a prime example of this. Pure weight-based encumbrance lies closer to the intradiegetic than extradiegetic side, and even farther in would be simply describing what and where your character is carrying everything at every given moment. An item slot system reduces this complexity by introducing a rule that keeps such things abstracted until it becomes salient intradiegetically.
The Supply system in Five Torches Deep is also a great example; instead of buying exacting quantities of every item before an adventure, one simply says what they are taking (torches, food, etc.) as well as specifying the amount of Supply they are bringing, which can be used to replenish depleted consumable items that they have brought.
All of the items in Ben's list representing the "in-world" side of his axis, monsters, spells, items, still represent extradiegetic devices that allow us to more easily ingress into the intradiegetic world for the purposes of imagining that specific object or activity. They are still "rules" as such.
I think what Ben is noting in the first case of "rules interacting with rules", then, is the degree of what Brendan Strejcek terms proceduralism, that is the degree to which a specific rule directs a certain outcome. Brendan's own Hazard Die system we can say is moderately procedural: it directs a certain outcome, yet also makes allowances for Referee's to interpret the results of the outcome as they see fit, up to and including ignoring it (but only so many times). Combat in D&D, however, is rather strictly procedural: you roll to hit, a hit either connects or misses, and then a certain amount of damage is dealt to hit points. We can think of proceduralism then as the degree to which an extradiegetic element ingresses towards another extradiegetic element.
The other factor at play, I think, is the speed at which an extradiegetic element ingresses towards another extradiegetic element. As I think may have become element, extradiegesis and intradiegesis are intertwined, you can not have one without the other. Combat in D&D, for example, still requires the existence of an intradiegetic space for the extradiegetic devices to make sense, but that intradiegetic space need only be touched on in the briefest of manners. This is how we end up with "I roll to hit, 8 damage to the orc." "The orc rolls to hit, 12 damage to you" and so on.
I think rather than Ben's admonition to stick to either an extra or intra-diegetic style doesn't totally make sense in light of this: rather, I think what's important for designers to consider is where do you want your extradiegetic points of ingress to the intradiegetic space to be, how do they achieve that ingress, how procedural are they, and what do you want the breakdown of extra vs intradiegetic time do you want to look like?
These are all very loose thoughts and there's likely to be gaps in my reasoning. Please feel free to point them out and start a conversation!
*I use text and reader to refer to any cultural work and audience member
**It would be interesting to see how LARPs figure in this paradigm
***Obviously there are other rules that could take the place of this one, such as group consensus, but I'm using the Referee model
****I think the main reason this comes up in RPGs so much is, compared to other media, even videogames, the burden of creating and enacting the extradiegetic devices is largely on the reader, whereas in books, films etc the extradiegetic labour of telling/creating has been already done by the author.