EA - An in-progress experiment to test how Laplace’s rule of succession performs in practice. by NunoSempere

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An in-progress experiment to test how Laplace’s rule of succession performs in practice., published by NunoSempere on January 30, 2023 on The Effective Altruism Forum.Note: Of reduced interest to generalist audiences.SummaryI compiled a dataset of 206 mathematical conjectures together with the years in which they were posited. Then in a few years, I intend to check whether the probabilities implied by Laplace’s rule—which only depends on the number of years passed since a conjecture was created—are about right.In a few years, I think this will shed some light on whether Laplace’s rule of succession is useful in practice. For people wanting answers more quickly, I also outline some further work which could be done to obtain results now.The dataset I’m using can be seen here (a).Probability that a conjecture will be resolved by a given year according to Laplace’s law.I estimate the probability that a randomly chosen conjecture will be solved as follows:That is, the probability that the conjecture will first be solved in the year n is the probability given by Laplace conditional on it not having been solved any year before.For reference, a “pseudo-count” corresponds to either changing the numerator to an integer higher than one, or to making n higher. This can be used to capture some of the structure that a problem manifests. E.g., if we don’t think that the prior probability of a theorem being solved in the first year is around 50%, this can be addressed by adding pseudo-counts.Code to do these operations in the programming language R can be found here. A dataset that includes these probabilities can be seen here.Expected distribution of the number of resolved conjectures according to Laplace’s rule of successionUsing the above probabilities, we can, through sampling, estimate the number of conjectures in our database that will be solved in the next 3, 5, or 10 years. The code to do this is in the same R file linked a paragraph ago.For three yearsIf we calculate the 90% and the 98% confidence intervals, these are respectively (6 to 16) and (4 to 18) problems solved in the next three years.For five yearsIf we calculate the 90% and the 98% confidence intervals, these are respectively (11 to 24) and (9 to 27) problems solved in the next five years.For ten yearsIf we calculate the 90% and the 98% confidence intervals, these are respectively (23 to 40) and (20 to 43) problems solved in the next five years.Ideas for further workDo this experiment for other topics besides mathematical theorems and for other methods besides Laplace’s lawAlthough I expect that this experiment restricted to mathematical experiments will already be decently informative, it would also be interesting to look at the performance of Laplace’s law for a range of topics.It might also be worth it to look at other approaches. In particular, I’d be interested in seeing the same experiment but for “semi-informative priors”—there is no particular reasons why that approach only has to apply to super-speculative areas like AI. So an experiment could look at experts trying to come up with semi-informative priors for events that are testable in the next few years, and this might shed some light into the general method.Checking whether the predictions from Laplace’s law come trueIn three, five, and ten years I’ll check the number of conjectures which have been resolved. If that falls outside the 99% confidence interval, I will become more skeptical of using Laplace’s law for arbitrary domains. I’ll then investigate whether Laplace’s law could be rescued in some way, e.g., by using its time-invariant version, by adding some pseudo-counts, or through some other method.With pseudo-counts, the idea would be that there would be a number of unique pseudo-counts which would make Laplace output t...