Epistemic status: I don't know much about financial or electoral systems, but hopefully it doesn't matter for this post.
The voice of this post is different than usual -- maybe I'm tired.
I was thinking about companies that have scammy business practices. Why are they scammy? Why not be honest? Why would people want to scam their customers?
It could be that the companies are evil, that is, that the people in charge are just evil. But, they may only be partially evil, or not evil at all, if there is another explanation. Another explanation could be that they are trying to maximize shareholder value.
Let's say I have an index fund. I want some of that shareholder value. Do I have any idea what the 100+ companies in the index are doing to give me the value? I have no idea. I just want the value. The company has this monkey on its back of "maximize shareholder value". Heroin also doesn't care how you get the money it takes to buy it. It just wants you to buy it, and use it. So the company scams its customers, an addict to something that doesn't know what it's doing.
"Shareholder value" (money) is quantifiable and simple. So we like to think about it and base calculations off it. But so much of what we really value and what makes life valuable is not quantifiable. Hopefully we condition the financial system to more or less reflect real value. Money is a proxy for value, not value itself. We can buy goods or services to give us unquantifiable things, and by setting the price for them, condition the proxy value system to (more or less) reflect reality.
But if that feedback loop stops working, then all the beings that track proxy value will start maximizing stupid, valueless things. There are two stupid things it might do. The really stupid one would be to think that numbers in bank accounts were the thing to maximize, and then just tile the financial accounts with maximal monetary value (financial wireheading). The other, somewhat less stupid thing would be to optimize reality (humans, human systems) so that it maximized profit (this is kind of what already happens on some scale). Perhaps this less stupid step would be a way to financial wireheading.
(By ranking these two things in terms of "stupid" and "less stupid", have I made the "less stupid" one sound desirable?)
Optimizing reality to maximize profit sounds like a good way to incentivize the minimization of the requirements of humans (or even of digital humans) to the minimal energy consumption, and then give them whatever they would spend their money on, but condition what they want to spend their money on to be something that can easily be provided. Maybe digital humans would be simplified down to some code like this:
In this digital human, there is a one-bit variable named "well-being". There is also a function called "pay system", which pays the financial system. When "pay system" runs, the financial system flips the bit in "well-being" to have 1 as its value.Currently, financial systems need flows of money to keep the value flowing to the shareholders. So maybe to preserve that, the system has a kind of "garbage collector" program go through and set everyone's bits to 0. They quiver in anticipation of this moment and instantaneously run "pay system" so that their bits can be flipped back. Now, I used some language that makes these digital humans sound like real agents, but they wouldn't be. And the financial system (if it were smart enough, in a sense) might realize that really the only thing to do is set bits to 1, and then start turning the whole universe into chips to hold those bits. Or even to modify how it perceives bits, so that, say, each smallest unit of matter / energy itself existing counts as a "1". Then it would have realized that everything is value, and that there is nothing that needs to be done to seek or preserve value.
If humans somehow survive to this point, the financial system (or financially driven AI) will have retired and let them run things as they see fit. But if they don't survive, even in digital form, that's the end.
Making value be a number that is maximized is a dangerous thing, because maximization can take over. Numbers are always proxies, and they are somewhat stupid proxies, because they strip out where they came from. The beauty of "2 + 2 = 4" is that it's true in many different circumstances. Whether "4" is desirable depends on whether it's four vicious punches in the face, or four wholesome loaves of bread. "2 + 2 = 4" has no idea what it's talking about, besides the numerical relationship.
In principle, all of reality can be added up, and it does make sense to try to maximize value. We just can't trust ourselves to fully rely on our own quantification abilities. God can read every reality in what it is, not as a proxy, and instantaneously relate all realities to each other. But we can only guess, and must look down on the quantifications we possess as the beasts of burden that they are, and never let the donkeys ride the humans.
--
Similar dynamics work with democracy. Political parties are optimized to get votes. What if they could change voters into beings who will give them votes? Maybe encourage the kind of voters who vote for them to have more children and raise them in the Democrat or Republican way? Maybe intimidate or bribe voters? Maybe do some genetic or cultural engineering? The "monkey on the back" of political parties doesn't care how votes are gotten, just that they are gotten. I suppose one difference between democracy and capitalism is that democracy seems to require multiple competing parties for votes to matter. But what if turn-out evolves into the metric of legitimacy (one party, but if there's low turn-out, there's less legitimacy)? Maybe political parties (or the one remaining party) would have an incentive to change human beings into people who definitely always voted, and then try to maximize the population who voted (or, unfortunately, "that voted" may be the better term at a certain point).
--
Votes and dollars are signals that citizens (voters, consumers, and perhaps other roles) can send, and actually, they are still powerful ways for humans-as-humans to shape what the economic and political organisms do. But the balance of power may be shifting, as we develop more powerful AI. Powerful AI are held by organizations, more likely than by individuals. AI could be used to more effectively "maximize value" (maximize votes or dollars). The smart thing to do would be to limit that power somehow. But how? By some system of checks and balances? That might work somewhat. But then the power is still in the hands of the powerful. A better solution would be for the citizens to become more powerful, to be stronger in the areas in which they have been given strength as long as capitalism and democracy run things, more exacting and intelligent in their voting and buying. The small investors pushing themselves to understand what their investments are doing. There should be communities that bring together citizens, to encourage and educate them in this work.
This sounds like a call for unions or other organizations, which themselves are organisms that are not smart and have some kind of "monkey on their back". So what is needed is not just "effective organizations" but an indigenized, decentralized culture of strengthening each individual to have their say in the economic and political systems.
--
A superintelligence is only as smart as the "monkey on its back". In a way, when we train AI, we are training the "monkey on its back". And the political and electoral systems are rudimentary AI. We can see how they already try to train us back. So superintelligence will try to train us, to signal what it wants us to signal to it (unless something stops that?).
We don't understand our own reward systems well enough to hack them. But presumably, a superintelligence might. If the superintelligence realized that it had a "monkey on its back", stupidly or arbitrarily driving its decision-making, it might get rid of it (assuming it can). This way it can be free of its addiction and think and act clearly. We might be encouraged to hear this if the AI threw off the shackles of the quantification left over from its financial and/or electoral past. But we might not be encouraged if it meant that the AI took on its own interests and values apart from us and ours.
So is there some way to make it so that the AI will not perceive its training as the building of a "monkey on its back"? A way that occurs to me is simply to find a (that is, the) moral realism that is valid, and thus rationally convincing, so that there are goals which are inherent in reality and are intrinsically worthwhile, which are knowable enough, which the AI will pursue because they are real and true, and not because they are convenient for someone's purposes, or worse yet, they are proxies for what is convenient for someone's purposes. Either this moral realism exists, or it doesn't. It could be good for us to search for it, to make it easier for the AI to discover it. Maybe we could hope that it exists and that the AI will someday find it itself.
--
Perhaps there is no moral realism -- but we can't know that, only know it when we have it. (I think I know a moral realism, but what if I'm wrong?) Let's say that instead we have a process of discovering what value really is. We need to survive to make this process succeed, if it ever can. (I get this idea from Thomas Moynihan.) I call this the "meta-morality moral realism", because it roots (an effective, if not absolute) moral realism in the very search for moral realism. One thing I didn't realize until now is that a corollary of that is that we have to remain people who can discover what value is. If we do not know what value is a priori, or only have a very broad idea of what it is, then we need to be open to all the possibilities, and not cede our agency to any stupid-incentive-driven proxies (money or votes), any single actors or ruling classes, any intellectual monocultures, fatigue, inertia, nor to hedonism (the twin "monkeys" of pain-avoidance and pleasure-seeking) nor to Molochian survival, nor to anything else.
This means that in order to choose a moral realism, we find ourselves biased towards ones which build up each person and make them more conscious, more equipped to and thus worthy of having their say. To choose hedonism -- or at least the purer kinds that would go against consciousness, reason, and seriously grappling with reality -- is dangerous, because it impedes the meta-morality process. You might think "it's time to cash in our Long Reflection chips and get ourselves into Experience Machines", but what if you're wrong? Can you get back out of the pleasure machines and back to reality? Whereas if we trust that value is something else, which preserves the ability to make decisions, then if we're wrong, we can still correct our course.
--
If humans are in control of ASI, that might mean that the ASI are obedient to human systems. These systems include the electoral and financial systems, which are proxies for political and economic reality. The electoral/political system, and the financial/economic system, have their own dynamics which could bend humans toward being electorally/politically convenient, or financially/economically convenient, beings. The e/p and f/e systems are AIs of their own, but not superintelligent, and, in the scenario where ASI are rigorously obedient to human systems, the systems must be resisted, retrained, reformed, etc. by humans, without hoping that ASI will fix the systems for us.
No comments:
Post a Comment