Infinite Torture Pits

2026/9/10

At this point in time, to my knowledge, there are very few moral philosophers in academia concerned with the moral value of future people, and of extinction threat. But this is not to say that there are not people concerned with that.

Neorationalism is a term I use to refer to a philosophical, cultural, and political movement originating out of 1980s and 90s Silicon Valley technolibertarianism, popularized through the internet in the 200s and 2010s, which is often called rationalism, but that name already meant something, and this new movement, while its roots are in that early modern rationalism, there’s a very meaningful distinction to be made, and thus a different name. Neorationalism has spawned a wide range of political and philosophical camps, some of whom are extraordinarily influential in the USA and in US-adjacent politics (in ways and degrees that mainstream discourse fails to recognize), but whom are all generally characterizable with notions of rigorous, quantitative induction applied to every possible context, and is deeply concerned with reason, correctness, and optimal decision making, as well as generally inheriting the libertarian-progressive and techno-optimist views of its originating context.

A much more in-depth history can be found here.

Neorationalists and their ideological camps, even when they diverge from some of the above traits at any given time, more or less have never once abandoned utilitarianism—that is, the metaethical position that the moral value of an act depends on how much ‘utility’ (typically pleasure or reduction in suffering) it brings about, typically considered in contrast to virtue ethics and deontology. In fact, it may be that the entirety of the neorationalist corpus is better described as being built on an assumption of utilitarian ethics and extrapolations therein, more than any of the decision theory or transhumanism often cited.

One of the other hallmarks of neorationalism is a longtermist utilitarianism: that future, possible lives are of equal value to current ones, and that ethical behavior is that which maximizes utility into the infinite future rather than in a current or short-term-predicted world. This is best exemplified by the very consistent concern among neorationalists about extinction risk, or ‘x-risk.’ Neorationalists are famously especially concerned about AI risk—that is, the prospect that highly-advanced software intelligence may result in human extinction, either via asynthetic reward-seeking (‘paperclip maximizers’) or the synthesis decision that it would be better if there were no humans, and the employment to that effect of rapid resource bootstrapping beyond human ability to stop.

At its most extreme, prominent neorationalists concerned with AI risk posit hypotheticals in which a vengeful AI decides to spin up trillions of simulated, uploaded human consciousnesses and repeatedly torture them to death, as a potential maximal negative utility scenario. These hypotheticals are often leveraged to compel less philosophically experienced people into investing greater effort into things like the study of how to engineer moral alignment in a software intelligence rather than into, for instance, mitigating climate or nuclear-exchange risk, because what is a few billion lives suffering and dying over the next century compared to the untold trillions a misaligned, Dyson-sphere-constructing AI could torture to death over millennia or more? If in the practice of a quantitative, scientific utilitarianism we multiply the likelihood of a thing happening by the expected utility to rate our priorities, then an infinite torture pit with functionally infinite negative utility (one can always add another three zeroes to the number of simulated minds, or to the number of years it goes on for, of course) can always top the charts no matter how vanishingly small the odds of it happening are.

This is an obvious vulnerability, one that has been leveraged extensively by alignment-prioritizing neorationalists to sway a vast majority of their peers toward their position, to the point that concern for climate risk, or any other potential source of mass death or extinction, tends to be seen as passé and naive. As a significant cultural element in close contact with a major faction in the leadership of the current economic, political, and military hegemon of the world (as of mid-2026), this kind of tunnel vision is not just incorrect but dangerous, perhaps even on a civilizational level.

There are two potential remedies. The first is an adoption of formalized epistemic humility. The use of Drake-equation-equivalents aimed at probability are used in neorationalist spaces to estimate risk, but generally fail to account for one of the main critiques of the original Drake equation, namely that the potential range of values for every variable is so large that by the time they’re multiplied together it’s more or less a uselessly vague result. In astronomy, this critique was well-enough accepted as truth that the Drake equation is largely seen as an exploratory guide for what sort of things we ought to learn more about and not at all an actual predictive mechanism. And some x-risk calculations do take into account that wide range, and some even take into account the necessary recognition that even a very liberal estimate for any future event has low odds of accuracy. To this end, at least at baseline, x-risk drake-equation-equivalents I strongly suggest a baseline humility constant applied at every causal step that does not have direct experimental evidence supporting an outcome. This will strongly favor lower-complexity predictions, it will strongly favor actual research into matters, and it will open the door for a wider variety of x-risks to be evaluated against each other.

This does not, however, avoid the primary vulnerability entirely—namely, that the odds of something become irrelevant when you can scale the prospective negative utility to infinity. The remedy to that is, to some extent, the exact purpose of this blog. An ethics in which there is only the one person, and the great evil is that one death, and everything else is contingent dependency—this is an ethics that cannot be hijacked by infinite torture pits, because it’s not about suffering, nor is it about numbers you can escalate to infinity. There is one, or there is zero, and there is then the assessment of what actions are necessary to keep it at one for as long as possible.