Optimization power in versus optimized product out
A more detailed idea of Optimization, especially Progress as optimization. In this thinking there are two ways to continue optimizing:
↑ 4 References
“But if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code, then there is my reason: you are drawing the wrong graph, and it should be optimization power in versus optimized product out, not optimized product versus time.” (Eliezer Yudkowsky, Rationality)
Rethinking optimization over time to optimization over “optimization power in” is interesting. Relates to Progress as optimization and Societal optimization power changes with time. Optimization power in versus optimized product out may be an argument against Inclinism.
The idea that Societal optimization power changes with time implies that it might not be time itself that changes things for the better, but rather Optimization power in versus optimized product out. Eliezer Yudkowsky uses this to argue that we should not carry forward a history of positive change through an AI revolution, since we are not sure how that will impact Optimization power in versus optimized product out
“But if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code, then there is my reason: you are drawing the wrong graph, and it should be optimization power in versus optimized product out, not optimized product versus time.” (Eliezer Yudkowsky, Rationality)
Rethinking optimization over time to optimization over “optimization power in” is interesting. Relates to Progress as optimization and Societal optimization power changes with time. Optimization power in versus optimized product out may be an argument against Inclinism.