So, is that a lot? Let’s assume a very simple model, with Reddit writers picking words from a fixed distribution, 10 words per reply, and all ‘over 1,000 comments’ threads having exactly 1000 comments.
Then, if the probability of picking ‘fried rice’ is p, the probability of ‘fried rice’ not appearing in that 1,000 comment, 10,000 word thread is
That gives us numbers close to 0.000066, but only for the mid 1940’s. Current level is around 0.000008 or 1 in 125,000. Let’s compute how many words such a long Reddit thread needs to get at that level:
(1-0.000008)^w = 0.22
w log(1-0.000008) = log(0.22)
w = log(0.22)/log(1-0.000008)
That gives me around 190.000 words. That does not seem impossible for me. If long threads on average have 2,500 comments, it is only 36 words per comment. One would have to look at the distribution of thread length and #words/comment to say more about this.
Now, let’s hope there isn’t an embarrassing error in the above computations :-)
I think it would be a better comparison to simply pick 100 other words, perhaps proper nouns like "Communist", "Stalin", "Fascist", "Mussolini", "Khmer Rouge", "Pol Pot", and etc. Do the same analysis with these, and it will tell you if Godwin's Law is telling us something unique about this particular bugbear. (I suspect it is, but haven't run the comparison myself)
Well, all those are simple regimes or dictators etc, whereas Nazi has been relegated to stand in for the ultimate evil, or as a synonym for evil itself.
It seems like a terrible assumption for this case.
Godwin's Law is largely a specific case of conversational drift (unless you start out talking about Nazis). Everyone starts out on topic, then gets more hyperbolic and distracted until you end up at some other idea, in this case Nazis.
So not only are comments in threads not independent samples, we're specifically looking at a proposition about non-independence.
He's formulating a null hypothesis - a (perhaps rejectable) alternative to the unspoken assumption in most of this thread, which is that Reddit, or perhaps comment threads in general, are disproportionately likely to include a mention of Nazis as they grow.
People don't pick words from a fixed distribution in the manner which you wrote. I scanned over 500,000 words I wrote and I never mentioned the word mentioned on the headline of this article. I'm not sure if any kind of meaningful interpretation can be made from the headline, but I think it's an interesting observation nonetheless.
People do not do that, but at least it is a model that one can improve. For example, as other commenters indicate, one can argument that occurrences within threads will be correlated (that will happen in discussions about World War Two, for example) but others indicated that "Godwin's law" occurrences may follow the model better.
In contrast, the OP just presented a bare number, implying that it was high, but without any evidence for that claim.
And yes, there's lots of room for improvement. The model is very simple, the guesses at the Reddit thread sizes are 100% guesses (how long is the average comment? What's the distribution of thread lengths? Etc), I did some serious rounding here and there, trusted Google to do the math right (it is not that hard, but IEEE may be insufficient to get results that are somewhat correct, and I don't know what Google does), etc.
As to your "I never wrote nasi goreng in 500,000 words" argument: let's take my simple/flawed model. It predicts
(1-1/125000)^500000 ~= 0.018
as the probability that 500,000 words do not contain "taboo". So, that simple/rough/flawed model predicts that, out of 1000 people writing 500,000 words, 18 wouldn't mention "Voldemort". You could easily be one of them. It's about as likely as somebody flipping 6 heads in a row. So, I don't think that single data point invalidates the model (no, I haven't done the proper statistics)
The average TV vocabulary is 2,000 words. High school graduates average a 10,000 word vocabulary, college graduates 30,000, and there are over a million words in the English language. Hence having 190,000 distinct words in a reddit thread seems implausible.
Then, if the probability of picking ‘fried rice’ is p, the probability of ‘fried rice’ not appearing in that 1,000 comment, 10,000 word thread is
That apparently, is around 0.22: Solving for p gives us That’s about 0.000066, or 1 in 15,000.Let’s compare that with https://books.google.com/ngrams/graph?content=nazi&year_star....
That gives us numbers close to 0.000066, but only for the mid 1940’s. Current level is around 0.000008 or 1 in 125,000. Let’s compute how many words such a long Reddit thread needs to get at that level:
That gives me around 190.000 words. That does not seem impossible for me. If long threads on average have 2,500 comments, it is only 36 words per comment. One would have to look at the distribution of thread length and #words/comment to say more about this.Now, let’s hope there isn’t an embarrassing error in the above computations :-)