Probabilities of Rare Events

[PatrickPowers]

Some unusual thing happens in your life. How unusual is it?

Statistics are of no use. Statistics depend on collections of data. You have only one datum. Not nearly enough.

How about probability. The problem is, as so often happens, is defining the sample space. Some unusual thing happened. You might make a stab at guessing how unusual it is, but what about all the other unusual events that didn't happen? It might be that the circumstances are such that observing something unusual is not unusual. Dealing with a whole nebulous class of unknown unusual events is not something I'm going to try to do.

What I've come up with is restricting the study to a certain class of unusual events where the problem is tractable and we can get an easy result. That class is that of doubled unusual events. Unusual event happens, then within a short period of time happens again in a way independent of the first. Then we can calculate a conditional probability. It's easy to define a sample space. It's the space in which X happens with the condition that X happened shortly before that.

Let's have a firmer definition of "unusual event." Fix a period of years Y. The first X has happened only once in those Y years. The empirical estimate for the probability of X is that it happens once in Y years. Calculating the conditional probability of the second X is easy as can be. It is a function of how close in time the two events.

An observer can make a data set of such coincidences. The surprise perhaps is that the result doesn't really tell us anything about coincidences but does tell us about the observer. The number of coincidences observed is the product of the number of coincidences occurring in the observer's presence times the sensitivity of the observer. While the two factors are confounded this isn't the end of usefulness. The observer's presence, their personal environment, depends strongly on the observer's habits. If two observer's share the same personal environment then their data sets can be compared to one another to measure their sensitivities. If the sensitivities of observers are known then personal environments can be compared. And so forth.

[quickfur]

As my doctor once put it, there may be a 2% chance of catching some deadly disease or dying of cancer, etc.. But when it happens to you, then for you it might as well be 100%. When you're dying of cancer, does it really matter anymore what number the statisticians care to put to it?

At that point it's a meaningless game of numbers, the harsh reality for you is that you 100% have cancer. You don't get half a cancer just because the probability is 50%, and you don't get 1/50 of a cancer just because the probability is 2%. Once you have it, it's 100% in your reality, regardless of what the statisticians say.

[PatrickPowers]

As my doctor once put it, there may be a 2% chance of catching some deadly disease or dying of cancer, etc.. But when it happens to you, then for you it might as well be 100%. When you're dying of cancer, does it really matter anymore what number the statisticians care to put to it?

At that point it's a meaningless game of numbers, the harsh reality for you is that you 100% have cancer. You don't get half a cancer just because the probability is 50%, and you don't get 1/50 of a cancer just because the probability is 2%. Once you have it, it's 100% in your reality, regardless of what the statisticians say.

After an event has been reliably observed to occur, to such an observer the probability the event occurred is one. The probability that some further event in that class may occur is as yet unknown.

[quickfur]

[...]
After an event has been reliably observed to occur, to such an observer the probability the event occurred is one. The probability that some further event in that class may occur is as yet unknown.

Well see, here's exactly where the catch in statistics is. You can't know what's the real probability of a future event, because it hasn't happened yet! Therefore, it's unobservable, and therefore any attempt to assign a probability to it is an extrapolation. This extrapolation is based on one giant elephant of an assumption: that past events predict future events. Luckily, in general this is true, or at least, true often enough that probability isn't a completely useless concept. But there is absolutely zero guarantee that the future will continue to behave like the past. You may have had a probability of X% winning the lottery for the past 50 years, but next year a Jupiter-sized comet crashes into the earth and wipes out all of humanity, and then your probability of winning the lottery is exactly zero. The X% figure only works as long as the circumstances surrounding past events continue to hold in the future. As soon as it changes, all predictions are out the window.

[PatrickPowers]

Deleted.