Fun game theory discussion with @adba, reproduced here with permission, and maybe mostly as notes for myself for a footnote in the eventual blog post about the Burgle Bug Fairness Principle:
ADBA: Game theory doesn’t imply strictly 2x, as you know. Good email, good classification system.
DREEV: ah, thank you! important thing to point out! the refund should be multiplied by the reciprocal of the probability of the customer noticing – maybe calculated as the fraction of customers that notice, if we want to be frequentist about it?
but of course we’re bayesians and by the principle of indifference we could take 1/2 as the prior and thus double-refunds as a starting point.
or just say double refunds because of focality aka schelling-pointiness and simplicity and not instantly glazing over the eyes of everyone we’re trying to convince about the burgle bug fairness principle
not to mention pareto-dominating the status quo.
but all this will make a great footnote at the very least!
(did i pack in enough game theory concepts there to restore my credibility?)
ADBA: You didn’t lose any credibility in the first place, and I agree with your reasoning for a half being a perfectly sensible default.
I don’t think it’s fundamentally a bayesian vs frequentist question either, and I don’t think “reciprocal of the probability” is the right algorithm. I’d be tempted to cast it as “a refund, plus a reasonable-seeming penalty we impose on ourselves for having let it get that far and as part of restoring warm fuzzies all around”; I’m yet to see an analysis that seems like more than an attempt to use formalisms to slightly sidestep the fuzziness without actually dispelling it.
DREEV: i think you’re right that bayesianism vs frequentism isn’t actually at issue.
quick rationale for “reciprocal of the probability”: suppose we want to fairly redress the burglary in the sense of giving back all the money we burgled, in expectation. if 1 out of n users notice and we refund each of those users n times what we stole then overall, on average, we’ll have refunded 100% of the ill-gotten booty.
of course our intention would be to more-than-refund users who notice as well as exactly-refund everyone who didn’t notice. so that’s supererogatory in one sense but feels important, incentives-wise, i guess because those who notice shouldn’t just believe us that we’re also refunding everyone who didn’t notice.
(this is turning out to be quite valuable to think through with you! thank you again!)
ADBA: I’ll buy that “reciprocal of the probability” has roughly the right shape in some ways (ie, there’s a reasonable case to make that the right ratio isn’t entirely inelastic to the percentage of people who notice), but it falls down badly in several ways:
a) The limits. If people had a 100% chance of noticing, you’d probably still want to give them more than what they lost back. If people had a 0% chance of noticing, you’re not going to give them infinite money (though if you had a way of getting infinite money, crafting a userbase with a 0% chance of noticing to introduce infinite money into the world would be an interesting problem!)
b) Near the limits. Giving 102% or 100.01% back could easily be taken as annoying / insulting / worse than 100% by a fairly high fraction of people, and wouldn’t seem “fair”/reasonable. There’s a huge body of literature about how people will do things like decline to get $1 if the person offering gets $9 of $10 and chose the split and the alternative is that they both get nothing, etc. It’s not an identical scenario, but the notion of “too small being unfair and literally worse than nothing” seems to show up pretty regularly.
I agree that a > 100% incentive on yourselves to get this right is a useful thing to have – both for the same reasons that beeminder is useful to people in the first place, and because it’s the kind of nice-and-rational thing that’s part of why you guys are so awesome.
DREEV: ha, fair, though we don’t have to worry about the 0%/infinity case. if a user does complain then that rules out 0% as the probability! (recall that the burgle bug fairness principle only requires double-refunds if the user notices before we’ve said anything to them. if we catch it before they do, a normal 100% refund is ok.)
i’m on board with the other points though, like not seeming stingy (sting-y yes, stingy no). perception-of-fairness is a big part of all this.