Notes for Long Reflection Reading List

These are notes on my readings on the Long Reflection, except for the two books, On the Genealogy of Morality, by Friedrich Nietzsche, and Teaching Children to Care by Ruth Sidney Charney (those two links are to their reviews).

--

Thinking Complete (Richard Ngo) "Making decisions under multiple worldviews"

[I decided to come back to this one later and restart reading it.]

--

Felix Stocker Reflecting on the Long Reflection

I find myself persuaded (although I'm not a tough audience) from this that the Long Reflection is not a practical pursuit, if it is a top-down discrete era of human history. That is, if it's something we impose on everyone, then we are doing so against someone's will, and they may defect, breaking the discrete era. Also I am (easily) persuaded by Stocker's objections that people will desire technology that the Long Reflection would try to hold back, and stopping human technological desire risks creating an S-risk of its own, the global hegemon. But, I do think that reflecting on what are the best values, and seeking to influence and be influenced by everyone to create (ideally) some kind of harmony in human values (or reduction of disharmony that allows for a more ideal "liberal" solution to the coordination questions that the Long Reflection is trying to answer) is something that can be ongoing. I would call this "cultural altruism", or a subset of cultural altruism. Much of what Ord and MacAskill would want could be pursued in a bottom-up, intermingled way and avoid some (or all?) of Stocker's objections.

--

Paul Christiano Decoupling deliberation from competition

Christiano makes the point that deliberation can be infected by competition. This would affect a bottom-up cultural altruism scene. However, I hope that a social scene can absorb a certain amount of competitiveness without harming it. For instance, when we try to find truth, we (society) sometimes hire lawyers to argue each side of a case and then we listen to what they say. Innovators in thinking may be motivated by competition, but as long as they are also evaluators (are both "soldiers" and "scouts"), or enough people who have power are "scouts", the competition only serves to provide ideas (or bring to light evidence) to select from, which is a good thing when you are trying to find the overall truth. When competitive people shut other people up, or have state-level or "megacorp-level" mind control / propaganda powers, then competition is bad for deliberation. But humans competing with and listening to humans on a human scale is good for deliberation. "All" we have to do is keep states and corporations from being too powerful.

I imagine cultural altruism being something like "status quo truth-finding but X% more effective". Our current truth-finding culture is (from one perspective) pretty good at bringing about truth, or at least, truths. Look how many we've accumulated. (Maybe where it needs to be better is in finding a whole to truth. And maybe we should think of how to protect it.)

I don't think I'm talking about the same thing Christiano is. I think he's talking about how AI teams can deliberate despite race dynamics, or something like that. Whereas what I imagine is everybody (all humans, more or less) interacting with each other without real time pressure. But it's interesting to think, where exactly is the distinction between Christiano's part of culture and the rest of culture? Isn't cultural work being done, perhaps that would affect human culture in general (more my concern) by Christiano's fairly pragmatic and craft-affected tactics for fostering deliberation despite race dynamics? Isn't pragmatic, resource- and time-constrained life where values come from? Christiano's situation is just another of many human situations.

In the section "A double-edged sword", Christiano talks about the practical benefits of competition to weed out bad deliberators (their influence, not them as persons). I suppose this feels realistic to him. To me, I feel (maybe naively) that ideal deliberators would stop fearing each other and simply fear the truth. If lives are at stake, because ideal deliberators index themselves to the saving of lives or whatever is of highest value, they would naturally work their best, and if this can be known, defer to people who know better. But Christiano has lived in his part of the real world, where people are resource- and time-constrained, and implicitly or not thinks that it generally has to be competition that gets the job done of communicating reality to people, and not an innate indexing-to-reality. I assume (if he really does believe that innate indexing-to-reality is not an option, or hasn't thought of it) that his beliefs in some necessity or desirability of competition are connected with his limited personal experience. Christiano may not see the possibility that people can be ideal deliberators, or that a culture of ideal deliberation could be fostered, given enough time. (His context, again, seems to be of specific, relatively near-term situations.)

Maybe if people are mistaken about their own competence in judging whether to defer, that would be one reason why there would need to be some outside actor who pushed them to relate to the truth better, and this can never be fixed. (Would people in such a context be "competed away" by a wild and more or less impersonal social force ("competition"), or would there be a person who could tell them they were wrong, who knew how to talk to them and who could at least consciously attempt to make themselves trustworthy to the "deluded one"? Perhaps for many of us it is more bearable to be ruined by "competition" than to be corrected by a person we know. Of course, it is always possible that the people who correct us are themselves wrong. Maybe that's the appeal of competition, that in some sense it can't be wrong -- if you're not fit, you're not fit. But then competition itself distorts reality, at least through "Moloch".)

--

Finished the main article, now reading the comments (what's there as of 22 August 2022).

Wei Dai makes the point that without competition, cultures's norms can randomly drift. (I would add:) this is sort of like how in Nineteen Eighty-Four, once war goes away, totalitarian states can "make" 2 + 2 = 5. I've thought there could could be problems with digital humans coming up with beliefs like 2 + 2 = 5. But at the same time, Moloch distorts our thinking and our lives as well. So it seems like we're doomed either way to someday not living in reality.

However, believing that 2 + 2 = 5 is physically difficult. Probably because of how the human brain is wired -- and we can change that. But either the human brain is in tune with the truth (more or less; enough to found reason) or it's not, and it always has, or hasn't, been. If it's not, then why worry about deliberation going well, or being in tune with reality? We never had a chance, and our current sense of what is rational isn't valid anyway, or we don't have a strong reason to believe that it is. But if it is, then the solution is just to keep people's brains about like they have always been, and use that as the gold standard for the foundations of rationality (at least the elements that are or are more or less like axioms, which are the easy, basic elements of rationality, even if building sufficiently complex thought-structures could go beyond human capabilities).

If it is the case that our innate thinking is not in tune with reality (on the level of the foundational axioms of reason), can we know what to do? Maybe not, and if not, then we have no guidance from the possible world in which our innate thinking is invalid. So if we are uncertain between that scenario and the one where it is valid (or valid-enough), then since the valid-scenario's recommendations might have some connection with reality, we should follow them.

It does seem odd to me that I should so neatly argue for the status quo, and that the human brain (or, I would say, human thinking, feeling, intuiting, perceiving, etc. nature, of which the brain is a phenomenal manifestation) should be the gold standard of how we know. Can't we be fallible? It makes perfect sense that we could be. But practically speaking, we're stuck in our own world, and lost if we leave it.

(This seems like a bit of a new view for me, so I should think about it some more.)

--

Wei Dai says, later on --Currently, people's thinking and speech are in large part ultimately motivated by the need to signal intelligence[link], loyalty, wealth[link], or other "positive" attributes[link], which help to increase one's social status and career prospects, and attract allies and mates, which are of course hugely important forms of resources, and some of the main objects of competition among humans.--

I'm not sure if this is how things seem to people subjectively, or if rather they feel like (or are) motivated by love for their family and friends, or some higher good. They have to work for resources due to scarcity, and because if they don't, they won't be able to live or provide for the people they love. Maybe it is the case that even love is something that is really "ultimately" motivated by resource acquisition? If a person is aware of this, can they willfully choose love (or value, or rationality) against resource acquisition? Probably they can. (Rationalists can choose against their biases, so why couldn't other people make as strong a choice?) We might suppose that most people are stuck in survival mode, or don't think much further than just their immediate friends and family. But maybe that's an artifact of scarcity, ambient culture, and them not being educated to see the bigger picture.

If you think that everything is about resource acquisition, that is what the world will be. If you think everything is about love / truth / valuing, etc., that is what the world will be. Some people have to face the world as it currently is, and it bends their thinking toward short-term, strategic, self-interested, competitive, resource-scarce, resource-hungry thinking. But some people are free from that, whether through temperament or life situation (perhaps they are too "untalented" to be able to do anything practical in the world as it is, and can only work on the world as it should be). These are the people who can and should lead the way in deliberation, in that, their minds are actually capable of deliberation. In areas of deliberation, the practical elites should be inclined to defer to them.

I checked the links in Wei Dai's comment (quoted above). They were about how unconscious drives (especially including the ones that drive signaling) really control people. I am subject to such drives all the time. But do they really matter in the long run? I am able to pursue what I choose to pursue. Perhaps my drive to seek a mate gives me the energy to seek a spouse -- and all that comes along with it, including new non-romantic interests, and a new perspective on who exists in the world. I get to choose which traits I find desirable in a spouse, even if the drive is not chosen. Or, if those have to "pay rent" by giving me the prospect of status, I get to choose, between the different sources of status that are roughly equal in expected yield, which of them I pursue. I can be intentional and conscious on the margin, and steer my psychological machinery vehicle in the direction that I want to go. The whole concept of "overcoming bias" and being rationalist doesn't make sense if this isn't possible, and I don't see why that level of intentionality is, or could only be, confined to a tiny subculture (tiny by global population standards). I think that short-term, competitive, resource-hungry, etc. thinking is like that evolutionarily-driven unconscious-drives side of being human, and the truly deliberative is like, or in some sense is the same as, the intentional, subjective, conscious, rational side.

I am suspicious that the unconscious mind doesn't even exist. Where would such a mind reside, if not in some other mind's consciousness? Can willing really come from anything other than an existing being, and can an existing being be anything other than conscious? I am skeptical that there is a world other than the conscious world (more than skeptical, but for the sake of argument, I would only suggest skepticism to my imagined reader here). Given this skepticism, we should be concerned that we are being trolled by evil spirits, or, more optimistically, are being led by wiser and better spirits than we are. Which side wins when we see things in a cynical or mechanistic way? I feel like cynicism and mechanistic thinking make me less intentional and more fatalistic, more likely to give in to my impulses and programming. Since my intentions seem to line up (at least directionally) with what wiser and better spirits would want, I should protect my intention and strengthen it, and see the possibility of free will, and be idealistic.

I suppose a (partial) summary of the above would be to say "deliberative people should be idealistic, conscious, believe in consciousness, despite 'the way the world works'". Maybe the Long Reflection (or cultural altruism) is concerned with determining what really should be, and some other groups or processes are needed to determine what can be, in the world that we observe and have to live in up close.

I think the New Wine worldview is one that inclines people toward being cultural altruists, and less so toward being EAs or the like, because it has a sense that the absolute best is the absolute minimum [in the sense that if you attain the absolute best on the New Wine account, you have only attained the bare minimum] and that there is a long time to pursue it, and that physical death ("the first death") is not as significant.

--

Cold Takes (Holden Karnofsky) Futureproof Ethics:

Karnofsky says --our ethical intuitions are sometimes "good" but sometimes "distorted." Distortions might include:
* When our ethics are pulled toward what's convenient for us to believe. For example, that one's own nation/race/sex is superior to others, and that others' interests can therefore be ignored or dismissed.--

Is it a distortion for our ethics to be pulled toward what is convenient for us to believe? Why does Karnofsky think that's true? I agree with Karnofsky on this thought (with some reservations, but substantially), but even if everyone did, why would that mean that we had found the truth? (I think a proxy for "I am speaking the truth" is "I am saying something that nobody in my social circle will disagree with" -- but it's an imperfect proxy.) Can Karnofsky root his preference in reason? I think that the truth is known by God, and sometimes thinking convenient ways will lead us toward believing what God believes, but sometimes it leads away. God is the standard of truth because he is the root standard of everything. So there is something "out there" which too much convenient thinking will take a person away from. Is there anything "out there" for Karnofsky's thinking to be closer or further from, due to distorted thinking? If not, does it make sense to call the distortions "distortions", or rather, "undesired changes"? (But without the loading we put on "undesired" to mean "objectively bad".)

Karnofsky clarifies a bit with --It's very debatable what it means for an ethical view to be "not distorted." Some people ("moral realists") believe that there are literal ethical "truths," while others (what I might call "moral quasi-realists," including myself) believe that we are simply trying to find patterns in what ethical principles we would embrace if we were more thoughtful, informed, etc.[link]--

I should check the link when I have time and come back [later: I did and didn't feel like it changed anything for me], but what I read in that quote is something like "Some people are moral realists, but I'm not. I'm a moral quasi-realist. I look for patterns in what ethical principles we would embrace if we were more thoughtful, informed, etc. Because thoughtfulness, informedness, etc. is a guide to how we ought to behave. It rightly guides us to the truth, and being rightly guided toward the truth is what we ought to be. Maybe it helps us survive, and surviving is what we ought to do." Which sounds like Karnofsky believes in an ethical truth, but for some reason he doesn't want to call himself a moral realist. Maybe being a moral realist involves "biting some bullets" that he doesn't want to "bite"?

[That characterization sounds unfair. Can't I take Karnofsky at his word? I think what makes me feel like he's doing something like using smoke and mirrors is that the whole subject of morality is pointless unless it compels behavior. Morality is when we come to see or feel that something ought to be done, and ideally (from the perspective of the moral idea) do it. So if Karnofsky ends up seeing and feeling that things ought to be done, or intends for others to see or feel that things ought to be done, even if it doesn't make sense to say that "ought" exists from his official worldview, then he's being moral, and relying on the truth of morality to motivate himself and other people. "Thoughtful" and "informed" are loaded in our society as being "trustworthy", so they do moral work without having to say explicitly "this is what you ought to do". So Karnofsky gets the motivational power of morality while still denying that it exists beyond some interesting patterns in psychology. I guess if he's really consistent in saying that he's just looking at patterns of thinking that emerge from "thoughtfulness and informedness", and "thoughtfulness and informedness" have no inherent moral recommending power, then he should say "hey, I'm saying a lot of words here which might cause you to think things, feel things, and do things, but actually, none of them matter and they have no reason to affect you that deeply. In fact, nothing can matter, because if it did, it would create morality -- what matters should be protected, or guarded against, or something -- and morality is just patterns of what we would believe if we were thoughtful and informed, which themselves have no power to recommend or compel behavior". Does Karnofsky really want to be seen as someone whose words do not need to be heeded?]

[This is quickly written and I have not read in depth what Karnofsky thinks about moral quasi-realism, which I'm guessing might be sort of the same as Lukas Gloor's anti-realism? I did read Gloor's moral anti-realism sequence (or at least the older posts, written before 2022). With Gloor's position, I also got the feeling of smoke and mirrors.]

--

Karnofsky summarizing John Harsanyi:
--Let's start with a basic, appealing-seeming principle for ethics: that it should be other-centered.--

Why should that be a foundation of ethics? It's merely "basic" and "appealing-seeming". It certainly is more popular than egoism -- or maybe, given our revealed preferences, egoism is a very popular moral foundation. Maybe egoism and altruism are supposed to compete with each other -- that looks like what we actually choose, minus a few exceptional individuals. Nietzsche wrote a number of books arguing in favor of egoism [as superior to altruism, as far as I could tell], and I can think of two other egoist thinkers (Stirner (I've read his The Ego and His Own) and Rand (whom I have not read but have heard of)). Are they "not even wrong", or do they have to be dealt with? Supposedly futureproof ethics is about what you would believe if you had more reflection. Maybe if you're part of the 99%, the more you reflect, the more you feel like a democratic-leaning thing like utilitarianism is a good thing. But if you're part of the 1%, and you're aware of Nietzsche's philosophy, maybe the more you reflect, the more true it seems that the strong should master the weak, based on the objective fact that the strong are stronger and power by its very nature takes power. There is a certain simplicity to those beliefs. So then will there be a democratic morality and an aristocratic one, both the outcome of greater reflection? Or maybe an AI reflects centuries per second on the question, and comes up with a Nietzschean conclusion. Is the AI wrong?

Personally, I lean utilitarian (at this point in my life) because I believe that God loves each person, by virtue of him valuing everything that is valuable. Everything that exists is valuable, and whatever can exist forever should. [Some beings turn out not to be able to exist forever, by their choice, not God's.] He experiences the loss of all lost value, and so does not want any to be lost. We are all created with the potential to be saved forever. So there is a field of altruism with respect to all persons. Perhaps animals (and future AI) are (or will) really be personal beings in some sense which God also values and relates to universally.

[Utilitarianism is about the benefit of the whole, tends toward impartiality, and is based on aggregation. God relates to each person, which accomplishes what aggregation sets out to do, bringing everything into one reality. God tends toward impartiality, and works for his personal interest, the whole.]

--

Karnofsky talks about how --The strange conclusions [brought about by utilitarianism + sentientism] feel uncomfortable, but when I try to examine why they feel uncomfortable, I worry that a lot of my reasons just come down to "avoiding weirdness" or "hesitating to care a great deal about creatures very different from me and my social peers." These are exactly the sorts of thoughts I'm trying to get away from, if I want to be ahead of the curve on ethics.--

However, the discomfort we feel from "strange conclusions" could also be us connecting to some sense that "there's something more than this". I remember the famous Yudkowsky quote ([which he] borrowed from someone else, whom I should look up when I have time) of something like "That which can be destroyed by the truth should be". But the reality for us, if we are the destroyers, is that in effect it is "Whatever can be destroyed by the truth as I currently understand it, should be". So, if we decide to destroy our passage to whatever our intuitions of diffidence were trying to tell us, perhaps by erasing the intuitions, maybe we have destroyed some truth by committing to what we think must be true, counter-intuitively true. We should probably hold out for some other truth, when our intuitions revolt, because they might be saying something.

[The quote seems to originate with P. C. Hodgell]

I believe that eternal salvation dominates all other ethical concerns, as a matter of course. Unbearable suffering in itself is bad because God has to experience it, and it is for him what it is for any other being: unbearable. What God, the standard, finds unbearable, will be rejected by him, and what is rejected by the standard is illegitimate. We should be on the side of reducing unbearable suffering. If we are, then we are more in tune with God and thus more fit for eternal life. I would agree with Karnofsky in the goal of ending factory farming, although it's not my highest priority. But, I think, from my point of view, it's valuable to look at Karnofsky's worldview, the one which so strongly and counter-intuitively urges us that "the thing that matters is the suffering of sentient beings" with some suspicion. Strong moral content says "this is 'The Answer'", but to have "The Answer" too soon, before you have really found the real answer, is dangerous. I don't think anyone is trying to scam me by presenting that urgent psychological thing to me, but I think it could be a scam in effect if it distracts me from the ways in which our eternal salvation and our relationships with God are at stake, and really matter the most.

[I suppose I'm saying that the theistic worldview is more satisfying to hold in one's head; satisfies, more or less, Karnofsky's concerns with animals; and would be missed if I said "okay, utilitarianism + sentientism must be right no matter what", so that I go against my intuitions of discomfort, even ones which might somehow intuit that there should be a better worldview out there.]

When people are forceful with you and try to override your intuitions, that's a major red flag. Although counter-intuitive truths may exist, we should be cautious with things that try to override our intuitions. In fact, things that are too counter-intuitive simply can't be believed -- we have no choice but to see them as false. This is the foundation of how we go about reasoning.

--

Should I feel confident that I have futureproof ethics? No, I guess not. I do think that according to my own beliefs, it's clear that I could, if I were only consistent with my beliefs. But my beliefs could be wrong. I don't know that, and currently can't know that. This goes for Karnofsky as well. The best you can do is approach the question with your whole heart, mind, soul, and strength, and be open to revision. Maybe then you can hold better beliefs within your lifetime, which is the best you can do.

--

Cold Takes (Holden Karnofsky) "Defending One-Dimensional Ethics"

As I read this, I think that this post may be mostly not on the topic of the Long Reflection.

However, since I'm reading it, I will say that I think in Karnofsky's "would you choose a world in which 100 million people get a day at the beach if that meant 1 person died a tragic death?" scenario, I would probably say, if someone asked me "do you want to go to the beach if there's some chance that it caused someone to die a tragic death?", it might make me question how necessary the pleasure of the beach was to me. If there were 100 million people like me on the beach, and we all somehow knew without a doubt that if we stayed on the beach, one person would die a tragic death, and that we all thought the same, we would all get off the beach. How could pleasure seem worth anything compared to someone else's life? Arguably, in real life, 100 million beach afternoons make us all so much more effective at life that many more lives are saved by our recreation. But I don't think that's the thought experiment.

Does my intuition pass the "veil of ignorance" test? If I don't know who I'm going to be, would I rather be the person who went to the beach, and somehow all else being equal that was 1/100 millionth of the share of someone else dying, or would I rather save the one person? What's so great about the beach? It's just some nice sounding waves and a breeze. Maybe as a San Diegan, I've had my fill of beach and a different analogy would work better. Let's say I could go hear a Bach concert. Well, Bach is just a bunch of nice notes. I like Bach, and have listened to him on and off since I was a teenager. He is the artist I am most interested in right now, someone whose concert I would want to attend. (I'm not just using him as a "canonical example".) But, Bach is just a bunch of nice notes, after all.

I find that the thought of someone not dying is refreshing, in a way that Bach isn't. I can't say I have no natural appetite for the non-ethical, which I may have to address somehow, but it's not clear to me that producing a lot of "non-ethical" value (if that makes sense) is easily comparable to producing "ethical" value. We are delighted with things and experiences when we are children, but when we see things through the frame of reality, lives are what count.

[By "lives" I mean something like "people", and people exist when they are alive. (And I think that non-humans can matter, as well, as people, although I'm not sure I've thought through that issue in enough depth.)]

Now, that's my appetites, and thus, I guess, my preferences in some sense. But what does that have to with moral reality? I guess one way to look at morality is that it's really just a complicated way to coordinate preferences, and there is no real "ought" to the matter. So then it would make sense to perform thought experiments like the veil of ignorance. But as a moral realist (a theistic moral realist), I believe that my "life-over-experience-and-things" intuition lines up with what I think God would want, which is for his children to live. Their things and experiences are trivial for him to recreate, but their hearts and thus their lives are not. God simply is the moral truth, a person who is the moral truth, and what he really wants necessarily is what is valuable.

--

jasoncrawford's EA Forum post What does moral progress consist of?

I chose this post for this reading list hoping that the title indicated it would be an examination or questioning of the very concept of moral progress. But I wouldn't have chosen it if I had read it. But now that I think about it, maybe I can make something of it.

I guess the part about how Enlightenment values and liberalism are necessary for progress (of any sort), might mean that somehow we would need the Enlightenment baked into any Long Reflection, as the Long Reflection is an attempt at moral progress (seeking better values). Perhaps looking at values as an object of thought comes out of the Enlightenment, historically at least? Or the idea of progress (perhaps) was "invented" in the Enlightenment, and can only make sense given Enlightenment ideas, like reason and liberalism? I can tentatively say that I'm okay with the idea that Enlightenment influence is necessary for progress, and that I'm in favor of progress, if I can mix other things with the Enlightenment, like deeply theistic values. And, I think that any other stakeholder in world values who is not secular, would want that or something equivalent.

(I'm not sure I can endorse or reject the claim that the Enlightenment could be an essential part of progress, given what I know.)

--

rosehadshar on EA Forum How moral progress happens: the decline of footbinding as a case study

What I will try to use from this post is the idea that moral progress comes through both economic incentives changing, and people deliberately engaging in campaigns to change behaviors and norms.

The Long Reflection, I would guess, will not occur in isolation from culture. If it proceeds according to my assumption that it is done both rationally and intuitively by all people, and not just rationally by a cadre of philosophers, then campaigns of moral progress will be part of the "computation" of the Long Reflection. All those people adopting the apparently morally superior values would be the human race deciding that certain moral values were better than others, offering their testimony in favor of the new values, thus (at least partially) validating them, just as the cadre of philosophers, when they agree on premises, all testify to the values that follow from those premises.

Economic changes affect how people approach reality on the level of trusting and valuing. I would guess that in cultures with material scarcity and political disestablishedness, people would have a stronger feeling of necessity -- thus, more of a sense of meaning, and less of a sense of generosity. And the reverse being true of cultures as they have less material scarcity and more political establishedness. It might be very difficult to preserve a sense of necessity in a post-scarcity future, and this would affect everyone, except maybe those who deliberately rejected post-scarcity. A lack of meaning, if taken far enough, leads to nihilism, or, if it doesn't quite that far, to "pale, washed-out" values. Perhaps these would be the values naturally chosen by us after 10,000 post-ASI years. [The 10,000 years we might spend in the Long Reflection.] But just because we naturally would choose weak values, doesn't mean weak values, or a weakness in holding values, is transcendentally right. What if our scarcity-afflicted ancestors were more in tune with reality than our post-scarcity descendants (or than us, where we are with less scarcity but still some)? Can we rule out a priori that scarcity values are better than post-scarcity values? I'm guessing no. What we think is "right" or "progressive" might really just be the way economic situations have biased us. It could be the case that meaning and selfishness are transcendentally right and our economic situation pries us away from those values, deceiving us. Thus, for a really fair Long Reflection, we have to keep around, and join in, societies steeped in scarcity.

So can we really have moral progress, or is it just that biases change in a somewhat regular, long-term way, such that if we are biased to the current moral bias-set, we see the intensification of it as progress?

A cadre of philosophers will be biased from their economic and other experiential upbringing. The cadre may have either watched or was formed secondhand by TV and movies (or in the future, VR equivalents?) which are based in blowing people's minds. (Secondhand exposure to such artifacts through the cultural atmosphere shaped by those who did watch them.) You can feel something happening in your brain when you watch such mind-blowing movies as The Matrix and Fight Club, and that blown-open, dazzled, perhaps damaged mind (which might still be clever, but which loses its sense that there is such a thing as truth that matters) perhaps remains with people their whole lives. I suppose having written this, now people could try to raise a subculture of Long Reflection philosophers, who have not been shaped by TV, movies, or VR -- only books. But books condition people as well. In fact, philosophical reflection conditions people, makes them "philosophical" about things.

Being in physical settings shapes a person. Driving a car is about taking risks and acting in time. Taking public transit is about those things too, but more so about waiting and sitting. Being in VR spaces could be about personal empowerment, flying like a bird, wonder and pleasure (I'm assuming that VR systems won't have any bizarre and terrifying glitches).

Ideally philosophy is pure truth -- but what is philosophy? Is philosophy a "left-brained" thing? Is the truth only known that way? Or is it a "right-brained" thing as well? If we are all raised somewhat similarly, we might all agree on a definition of philosophy, as, perhaps a more left-brained thing (although our premises come from intuitions, often enough). But why should we all have been raised the same way?

--

Thinking Complete (Richard Ngo) Making decisions under multiple worldviews ("for real" this time)

I read this, but at this point, with the level of focus I can give, I can't go in depth on it. But it does seem to be something that some people interested in the Long Reflection should read (unless something supersedes it?). It's about what to do when you can't merge everyone's worldview into one worldview, but you still have to come up with a decision. I think it significantly possible that the Long Reflection will reach an stalemate and civilization will still have to make the decisions that the Long Reflection was supposed to help us make. While epistemic work can resolve some issues (get people on the same page / show armchair Long Reflection philosophers more evidence as to what really matters), I'm somewhat not optimistic that it will make it all the way to unity, and we will still have to decide collectively.

--

Thinking Complete (Richard Ngo) Which values are stable under ontology shifts?

This is an interesting post, and perhaps three months ago, I would have written a post on this blog responding to it more in depth. It is relevant to the Long Reflection, I suppose, by saying that values may not survive changes in "ontologies" (our understanding of what things are or how they work?), and may end up seeming foreign to us.

(One thought: what is it about the new ontology that is supposed to change my mind? I would guess, some form of reason. Why should I care about reason? Why not just keep my original way of thinking? Or -- is reason the base of reality, or is it rather experience, or the experiences that a person has? My experience of happiness, and myself, are rude facts, which reason must defer to. I can find things to be valuable just because I do, and I want to. (Maybe the best argument against my "rude fact sense of happiness" being valid is someone else's "rude fact of unhappiness" caused by that happiness of mine.) Something like the "ordinary" and the "ontological".)

[I can value whatever I want, regardless of what people say reality is, because base reality is me and my experiences, the cup I drink from that was sitting on the table next to me, my own history and personal plans. Sure, people can tell me stories about where my desires came from (evolution, of course), or about how I am not as much myself because my personal identity technically doesn't exist if I follow some argument. But my desires and my personal identity exist right here in the moment, as rude facts, rude enough to ignore reason, and they are the base on which reason rests, after all.]

[These rude facts put a damper on reason's ability to change our values, at least, they protect each of our unique persons, our thickness as personal beings, as well as the objects of immediate experience and consciousness itself. But reason can persuade us to see reality in different ways. Perhaps it can help us to see things we never saw before, which become new parts of our experience, just as undeniable as the cool water flowing in us after we have drunk it. Reason can show us the truth, sometimes, but there are limits to reason, and ultimately personal beings experiencing is reality.]

Book Review: Teaching Children to Care, by Ruth Sidney Charney

See also the preview for this review.

Teaching Children to Care, by Ruth Sidney Charney, is a book I would recommend to some people. I think for what it is it is a good book, but where it fails to be, or where some other book fails to make up for it, there is a serious problem. I could recommend it to anyone who works with children (like parents or teachers). It may have some practical value to them. Also, the spirit of it is good and sometimes a teacher communicates more of what is value through their spirit even than the good advice they give. (Another book that is like that is The Reentry Team by Neal Pirolo.)

--

Teaching Children to Care notes

I read this book through without taking notes. That may not have been the best idea, since now I am tempted to simply say my impressions without giving references, and I don't feel like reading through the book carefully, and feel like simply quitting [rather than re-reading].

I am feeling tired of writing at this point, like I'm losing interest in the subject matter. What will happen next? Will I "love Big Brother"? There was someone in my life who steadily and systematically undermined my devotion to my beliefs and my writing. They used skillful means in an all-out attempt to gain my trust and reshape me according to their will. Their expectation was that I would quit one day (then, perhaps, I would have to validate their point of view). They had a choice, to join me in my path of life, or to try to shut me down. Because they tried to shut me down, they broke me. I can imagine them reading this, and them feeling all kinds of emotions, but their iron certainty that I will give up my writing someday does not go away. It is their expectation, and, I am fairly certain, their deep personal preference.

If my writing is correct, then they are an instrument of Satan. This may sound crazy or harsh, but it's the logical truth.

[I wrote that some days ago in a state of turmoil, but I affirm it now in a state of peace.]

So what can I do? If I can't write, how can I be true to my beliefs? No one seems to want to share them with me. By writing, I enter a world where at least I believe what I believe. The text I write and I enter a relationship and share the beliefs that we create, and the beliefs that previous texts created with me as I wrote them.

But now, if I quit writing, how can I stay true to my beliefs? I will lose that last community. But then will I have to share some kind of community? All the communities that exist are not New Wine communities. If I really share "community" (being "one-with-together" with others?), how can I possibly hold divergent beliefs from those I am "in community with"? So I will (at least seemingly) inevitably come to agree with and approve of everyone else around me. I will have no choice but to see things as my community sees things, to participate. My choices of communities are all based in lies, and they all spit in the face of God, whether through hostility to God or through fake love of God. But I must be brought to be a social person, responsive to my community, brought into tune with it.

[Similarly, although I wrote this in a state of turmoil, I think it is still factually correct when I am at peace. I still see the danger, and the lies, rejection of God, hostility, and fakeness.]

I have written that people should come into tune with God, but who and what is God? Is "God" the loving creator of the universe, who holds us to the highest standards, a person who loves and dies for us? Or is "God" community, the set of all people around us? Between the God of Abraham, Isaac, and Jacob (or the Speaker and Legitimacy), and community, which is more omnipotent? Who do I fear more? God seems to be shackled by community, or by the way that community's members collectively construct how they will trust -- what the definition of "crazy" is, what images of God are socially acceptable to believe in, how hard to try to know the truth.

Defining morality as prosociality simply sets up the community as God. But if there is a real God, a person who loves us more than community can, who is the truth, then prosociality is a dangerous thing, a seductive lie.

So these are the stakes with which a person should approach a book like Teaching Children to Care, which is a book about getting children to behave, to like each other, and so on, apart from any mention of God. If Ruth Sidney Charney, the author, believes in God, she can't show it in a public school classroom. Instead, she has to deal with the behavioral issues right in front of her, or the classroom will not be a place of learning and work. So she instills in her students a responsiveness to each other and to her, and teaches them the Golden Rule -- do to other people what you would have them do to you. No mention has been made, or can be made, of Jesus, who spoke that rule. She mentions how morality is bigger than us, not something we create -- is she talking about God when she says "morality"? Or is morality really just "I want to please my teacher because I'm a child and it's a human instinct, and whatever she says, I want to do"? The teacher creates morality but doesn't teach children to love God. She doesn't explain where morality comes from, because, perhaps, if she tried, it would undermine morality. She speaks with implications rather than straight out, asks leading questions rather than baldly stating, so that children internalize what she says, and so that they can't fight back. They don't have the mental development to construct alternate systems of their own, but perhaps they could see through hers intuitively, or have the kind of powerful skepticism of those who don't understand a set of explanations, if she offered explicit explanations. But she doesn't. Implications are more psychologically effective, and she's convinced that the ends justify the means.

So children are indoctrinated to be deeply moral (or that is the attempt), and yet to find God peripheral or nonexistent. Morality, which I think is difficult to ground in anything other than God, is simply not grounded and becomes a free-floating force in people's mind. Not to be thought about explicitly -- if we did so we would either become nihilists or truly committed to morality (and thus out of tune with society). But instead this unspeakable force. I wonder if secular people who are moral realists are convinced that morality must be "out there" simply by the psychological force of having been taught to be moral when they were young, apart from rationality. And perhaps morality is, practically speaking, not seen as something that needs rational grounding, because it has been ingrained in us so much. This kind of moral education may explain both moral realism and moral anti-realism among secular people.

This may make it sound like I didn't like Charney, but I think she makes, or made, the kind of teacher I would have liked. She is a passionate teacher. I can recommend her book as a way to understand passion, something I think is essential. While her emphasis on passion could lead someone to God, her emphasis on prosocial, arational morality threatens to lead people away from God. So she is a mixed phenomenon.

Part of how I am feeling now comes from bipolar disorder, I can tell. No matter what I have going on my life, when I feel low, I feel low. This is the content of my low thinking, given what I have lived so far. When I am not blinded by the depression, I can understand fully how it is that I can keep going. But for now, I can rest a bit, knowing that I have written some of my thoughts on the book I read. I think, maybe, I won't read it again to look for the supporting quotes to what I said above. But I can recommend reading the book, for its passion, if you want to check my work.

--

One additional thing I remember thinking as I read was about how, given how beautiful and effective Charney's methods sound, why could they not be used on the elites of the world, so that, perhaps, they could bring the countries of the world in harmony with each other? I thought, maybe because the way she talks to children is something that wouldn't work on adults. It's too artificial, too skillful. Adults want the skillfulness of a poker player, to affirm their adulthood, but not the skillfulness of a professional mom.

It made me wonder, how do we make this strange creature called adult? What is this being? No child is really bad, we say, but some children grow up to become bad adults. A child can hardly set himself or herself up against his or her family. But the leader of a nation can. They can shape themselves into their own being, and shut down every human feeling, can listen to other people speak and know that they will never agree with them, and go on with their agenda. They can decide who they want to be and then be it, taking the responsibility for it, suffering for it, and still continuing to choose it, despite what other people think. Children try to say "no", but adults sometimes can actually succeed in saying "no".

--

(later)

One thing Charney talks about is how she isn't trying to punish children, simply have them see the consequences to their actions.

What if adults were shown the consequences of their actions? So often, the natural consequences of people's actions fall due not in their own lives, but in others. What if some teacher could help adults see the effects of what they do?

Adults think that being shown the moral way, having someone say "you should know better" is a thing of youth. Now that they are older, they are past that. Adults can no longer do wrong.

Now, there are certain things that an adult can do wrong. Everyone knows what those things are. We all agree on that. But the things that we don't all agree are wrong, are not to be enforced, and not even to be called wrong, so we don't have to think of them as wrong, so, in our heads, they are not wrong.

Adulthood as a collective can't be taught. It knows. A reshaping of adult values by being shown the consequences of adult behavior can't be done, it seems. So maybe the moral thing to do is to fit into the constructed adult reality, be good at being one of the tribe of adults?

But, the consequences don't go away... how will we take into account the real effects of what we do (and don't do) if we don't listen to the truth?

--

I wouldn't mind my life so much if it weren't for the bipolar disorder. Writing isn't so bad, and when I'm euthymic, I feel fine. I can hear some imaginary (or real) readers being solicitous for me when hearing about my bipolar disorder. They seem to (or really do) care about me so much and wish that I would take care of myself. But if they care about a person's well-being, I have a great opportunity for them. They can save up $5,000, donate to Against Malaria Foundation, and thereby save someone in the developing world from a painful death from malaria, which would have orphaned their children and widowed their spouse, and diminished their extended family and weakened the national economy. (It's even worth donating $50.) Or, this imaginary or real person who is moved by my bipolar disorder is a Christian and thinks that the second death is worse than the first (which, basically I agree with, although I do give money to global development), they can give a much smaller amount of money -- apparently, $1 -- to Doulos Partners and that should cause [or allow] one person to start to become a disciple of Jesus. Do you think that these charities might not be the best ones to donate to? You can look for better ones. You could even just give money directly to people who are worse off than you, if you can't find any trustworthy charities.

If you have time but not money, you can think of some way to use your time to help people. If nothing else, you can seek to make one new friend, and be a good friend to them.

But you may not have any time or money to spare. Some people don't. Then at least adopt the identity of a "person who cares", who would donate your time and money if you could, so that when someone enters your life who is more deeply involved in caring, you can offer them the welcome of your validation of what they are into, instead of passive-aggressive or blatant hostility, or indifference.

--

The title of the book I read is Teaching Children to Care. To me, I thought of "caring" as "feeling and acting strongly", more as "exerting effort to do good, work on a good-making project". But the book mostly emphasizes "seeing other people as people" and the Golden Rule. A way to reconcile these two meanings is to think of God, who is personally blessed by large-scale altruistic efforts, in the way that if you share your lunch with someone, they are personally blessed by your personal thoughtfulness and the consequence of their hunger being alleviated.

--

One thing that makes adults resistant to new morality is that they have reached the developmental stage where they are their own person and they have their own boundaries, and they are secure in themselves. Or, that they have not reached that stage yet, and are vulnerable to being hacked by other people (or demons) and thus are resistant to attempts to change them. A secure person is not threatened by morality and so does not change, while an insecure person is threatened and so shuts it down.

Somehow it is possible for a secure person to take into account morality -- maybe through a discipline of fearing being trapped by your own security. Both a secure and an insecure person can find their rest in caring, in the interrelationship of all things to each other and to them, as opposed to, in the secure person's case, their own stability and boundaries. Presumably the insecure person, on some deep level, has no place to rest.

--

[Response:]

I thought I should go into more detail about moral realism.

[Secular moral realists may have a strong intuition that morality is "out there" and this intuition is the basis of their sense that morality is real, despite whatever difficulties in grounding it rationally (or lack of having tried to do so). They would deny the intuition that people have that God exists as being valid, but they do trust and honor the intuition that morality exists.]

[Secular moral anti-realists may have no qualms, and little difficulty, in "being good people". "You know, C. denies that morality even exists. But he's a good guy." They can, in unreflective moments, feel that morality is real. They can even get mad at injustice. They can devote their lives to doing good. But then they go back to the study of their minds (like Hume's study where he can be skeptical) and say "but none of it's real!".]

[Morality is an area where we seem to have agreed to be irrational, to not try to connect all the dots or demand that all the dots be connected, even beyond the background level of irrationality that attends most human endeavors. ("This thing makes you change what you do. You spend hours and hours, thousands of dollars to comply with this thing. It's not just how you feel -- it's something you have to obey. You just know that you have to obey it, no person or other visible force or situation makes you obey it. And you can't explain what it is, how it fits into the rest of reality -- or, you even say it's an illusion?") And, perhaps that acceptance of irrationality is because we have had morality ingrained in us in a subtextual way, or the instinctualness of morality is encouraged, but not accompanied by reason, when we attend secular public schools (or even religious ones that don't make God real to students sufficiently), or have parents who are helpless in explaining a rational grounding to moral realism to us when we are young.]

[Maybe we can at least explain where moral instincts come from -- evolution? (Why should we trust how we have evolved? Evolution helped us to survive in early environments?) Or we can say that they are heuristics for survival. (Why should we survive?) But then, if we are the "1%", why redistribute wealth? The 1% could probably maximize its survival best by not redistributing wealth. Or, a related question: does promoting animal welfare really lead to human survival? Often it is orthogonal to human survival. I think morality could come from evolution but does not necessarily serve the purpose of human survival. Maybe some people have genes that make them ethically oriented? Then why not shut them off? Does morality really have value? To answer that question, I think we need a moral realism.]

[Maybe morality is just maximization of value, by definition of "morality" and "value". Then, can we explain that voice that says, for each valuable thing X, "X is valuable"? Or is that also irrational, just a random "monkey on your back"?]

[I don't think I'm being fair to secular moral realists. I should at least explain why I think moral realism is hard to ground in anything other than God. Secular moral realists may be able to come up with a satisfying account of how moral realism is grounded.]

[How would they do that? Do we start with "these are our moral intuitions, now we have to find some metaphysical belief that lets us keep them"? But what if there's something wrong with our moral intuitions? One of the main points of having a grounding for moral realism is to know what particular things are moral, now that we know where morality comes from. I am generally relatively more of a thinker and writer than a reader. So I tend to work with first principles (or personal experience). But I did read The Feeling of Value by Sharon Hewitt Rawlette and remember that I had mixed feelings about it. I thought that it probably was successful in showing that some kind of experiential states can be known to be bad, just because they feel like badness, and some can be known to be good, just because they feel like goodness. I'm not sure I would be so charitable now. At least, without going back and looking to see the details there, I think "why should our perceptions of good and bad be transcendentally valid?".]

[A moral realism needs to be usefully thick, if we are going to guide our lives by it. You can always posit something like the (unfortunately named) "morons", "moral particles", and I can say, "fine, now we know where morality comes from, some kind of ontologically real substance of morality". Now what? We need to know something about these moral particles in order to know what is actually moral to do and be.]

[I don't know if there are any better secular moral realisms than Rawlette's, but at least hers is usefully thick. Hedonism (what she advocates for) is a somewhat useful guide to life. (Maybe that is what is so seductive and dangerous about it, that it's easily agreed-on for "practical purposes" while not really being in tune with reality.)]

[My approach to moral realism, as of now, is to say, "An ontologically real substance of morality exists. Everything that, practically speaking, exists, is conscious. (Only consciousness can interact with consciousness.) This means that the ontologically real substance of morality is conscious. Morality is about a standard which applies. For something to exist, it must ought to exist. It must live up to that standard. That things exist proves that morality exists and is being satisfied. The way that conscious beings metaphysically contact other beings is for their consciousnesses to overlap, for them to experience exactly the same experience. Morality metaphysically contacts everything that exists in order to validate it so that it can exist. So morality experiences exactly what we do, and finds the 'qualia of goodness or ought-to-be-ness' ('pleasure') good, at least on a first-order level, and similarly with the 'qualia of badness or ought-not-to-be-ness' ('pain'). This validates a lot of Rawlette's account.]

["But we know a few things more about morality. For instance, morality has to be self-consistent. Like us, it has to put morality first. So it has to put itself first. But it has to put itself first as an other, as a law it submits to. Thinking of morality as having two aspects, the enforcer of the standard and the standard itself, allows us to see that morality has to be willing to put aside everything, including its own existence, for the sake of its standard. If it ceases to be willing to do that, it is not self-consistent and it ceases to be valid, destroying everything by being invalid (no longer moral, and thus unable to validate anything).]

["Part of morality's self-consistency is that it must have the same values as itself. Everything that exists has value, it ought to be, either temporarily or permanently. (What is bad must someday cease to exist.) Morality must be on the side of value, and must value everything that is of value for what it is. Morality values persons in that they are persons, this personal valuing being called love. Morality must love in order to be self-consistent and thus valid. It loves that humans are in tune with it so that they can exist permanently. To love a person fully involves understanding the person's being fully, and that full understanding can only come from kinship. So morality is a person (a person who is also kin to animals).]

["Everything is the expression of a will, either that of morality, or of a free-willed being whose will is willed by morality. To be is to will. So impersonal beings are parts of personal beings and don't have independent reality. They are valued as parts of personal beings, and with those beings morality has kinship, not with their parts taken separately.]

["Morality has to be willing to bear the burden of what it imposes on others. If it's worth it for a human to pay a certain cost for morality's sake, it's worth it for morality to pay it as well, if possible. Morality already experiences every burden that is part of our experienced lives, by being conscious of what we are conscious of, but there is a further burden that each of us experiences, which is to experience only our own lives and deaths, without the comfort of knowing the bigger picture. How can morality bear that burden? Morality is composed of multiple persons, one of whom experiences everything, another, who does not and can live a finite life (the first maintains the moral universe through his/her validation of everything during the time the second lives a finite life)."]

[As you might have guessed already, "morality" in the above could be considered "God".]

[If we accept the above (or perhaps a better-argued version of it...), we have a concept of morality that largely supersedes hedonism. It incorporates hedonism and its recommendations, at least insofar as it validates the first-order goodness/badness of pleasure/pain (if pleasure has baked into it the perception that it ought to be, and pain, the perception that it ought not to be), as well as answering why it is that care for hedonic states is transcendentally valid. Further, we are recommended to be willing to give up everything for what is right, and thus to risk ourselves for that when it is called for. And we are to bear the burdens of those we rule over, as much as we can. It might be possible be able to come up with other ways to thicken the very concept of moral legitimacy, so that we know more about what morality must be, and thus what we must do or be in order to be moral. This thickness is a useful guide. And, if we think that this person or persons who are morality exist, they may have acted in history, and we may try to find evidence of where they might have spoken, allowing us to thicken our concept even further, although with less certainty.]

[To defend my earlier statement, I think that it's hard for me to imagine a successful secular / atheistic moral realism, because what I see as the way to ground moral realism involves the existence of God, and the (perhaps unrepresentatively few) secular moral realisms I've seen are not satisfying to me intellectually. Maybe if I want to strengthen what I say further, putting it briefly, I would say that if morality exists, it must love fully, and that kind of love is something that persons do. So then, morality is a person, and the word for a person like that is "God".]

--

22 September 2023:

When I wrote this post I thought that bipolar disorder was my big problem. However, now that I am more over the ways people have traumatized me and programmed me with the ways they wanted me to think, I see that those are a much bigger deal than my bipolar disorder. My bipolar symptoms, without those traumatized and programmed thoughts, are fairly mild.

The Drift of Financial and Electoral Systems and AI

Epistemic status: I don't know much about financial or electoral systems, but hopefully it doesn't matter for this post.

The voice of this post is different than usual -- maybe I'm tired.

I was thinking about companies that have scammy business practices. Why are they scammy? Why not be honest? Why would people want to scam their customers?

It could be that the companies are evil, that is, that the people in charge are just evil. But, they may only be partially evil, or not evil at all, if there is another explanation. Another explanation could be that they are trying to maximize shareholder value.

Let's say I have an index fund. I want some of that shareholder value. Do I have any idea what the 100+ companies in the index are doing to give me the value? I have no idea. I just want the value. The company has this monkey on its back of "maximize shareholder value". Heroin also doesn't care how you get the money it takes to buy it. It just wants you to buy it, and use it. So the company scams its customers, an addict to something that doesn't know what it's doing.

"Shareholder value" (money) is quantifiable and simple. So we like to think about it and base calculations off it. But so much of what we really value and what makes life valuable is not quantifiable. Hopefully we condition the financial system to more or less reflect real value. Money is a proxy for value, not value itself. We can buy goods or services to give us unquantifiable things, and by setting the price for them, condition the proxy value system to (more or less) reflect reality.

But if that feedback loop stops working, then all the beings that track proxy value will start maximizing stupid, valueless things. There are two stupid things it might do. The really stupid one would be to think that numbers in bank accounts were the thing to maximize, and then just tile the financial accounts with maximal monetary value (financial wireheading). The other, somewhat less stupid thing would be to optimize reality (humans, human systems) so that it maximized profit (this is kind of what already happens on some scale). Perhaps this less stupid step would be a way to financial wireheading.

(By ranking these two things in terms of "stupid" and "less stupid", have I made the "less stupid" one sound desirable?)

Optimizing reality to maximize profit sounds like a good way to incentivize the minimization of the requirements of humans (or even of digital humans) to the minimal energy consumption, and then give them whatever they would spend their money on, but condition what they want to spend their money on to be something that can easily be provided. Maybe digital humans would be simplified down to some code like this:

In this digital human, there is a one-bit variable named "well-being". There is also a function called "pay system", which pays the financial system. When "pay system" runs, the financial system flips the bit in "well-being" to have 1 as its value.

Currently, financial systems need flows of money to keep the value flowing to the shareholders. So maybe to preserve that, the system has a kind of "garbage collector" program go through and set everyone's bits to 0. They quiver in anticipation of this moment and instantaneously run "pay system" so that their bits can be flipped back. Now, I used some language that makes these digital humans sound like real agents, but they wouldn't be. And the financial system (if it were smart enough, in a sense) might realize that really the only thing to do is set bits to 1, and then start turning the whole universe into chips to hold those bits. Or even to modify how it perceives bits, so that, say, each smallest unit of matter / energy itself existing counts as a "1". Then it would have realized that everything is value, and that there is nothing that needs to be done to seek or preserve value.

If humans somehow survive to this point, the financial system (or financially driven AI) will have retired and let them run things as they see fit. But if they don't survive, even in digital form, that's the end.

Making value be a number that is maximized is a dangerous thing, because maximization can take over. Numbers are always proxies, and they are somewhat stupid proxies, because they strip out where they came from. The beauty of "2 + 2 = 4" is that it's true in many different circumstances. Whether "4" is desirable depends on whether it's four vicious punches in the face, or four wholesome loaves of bread. "2 + 2 = 4" has no idea what it's talking about, besides the numerical relationship.

In principle, all of reality can be added up, and it does make sense to try to maximize value. We just can't trust ourselves to fully rely on our own quantification abilities. God can read every reality in what it is, not as a proxy, and instantaneously relate all realities to each other. But we can only guess, and must look down on the quantifications we possess as the beasts of burden that they are, and never let the donkeys ride the humans.

--

Similar dynamics work with democracy. Political parties are optimized to get votes. What if they could change voters into beings who will give them votes? Maybe encourage the kind of voters who vote for them to have more children and raise them in the Democrat or Republican way? Maybe intimidate or bribe voters? Maybe do some genetic or cultural engineering? The "monkey on the back" of political parties doesn't care how votes are gotten, just that they are gotten. I suppose one difference between democracy and capitalism is that democracy seems to require multiple competing parties for votes to matter. But what if turn-out evolves into the metric of legitimacy (one party, but if there's low turn-out, there's less legitimacy)? Maybe political parties (or the one remaining party) would have an incentive to change human beings into people who definitely always voted, and then try to maximize the population who voted (or, unfortunately, "that voted" may be the better term at a certain point).

--

Votes and dollars are signals that citizens (voters, consumers, and perhaps other roles) can send, and actually, they are still powerful ways for humans-as-humans to shape what the economic and political organisms do. But the balance of power may be shifting, as we develop more powerful AI. Powerful AI are held by organizations, more likely than by individuals. AI could be used to more effectively "maximize value" (maximize votes or dollars). The smart thing to do would be to limit that power somehow. But how? By some system of checks and balances? That might work somewhat. But then the power is still in the hands of the powerful. A better solution would be for the citizens to become more powerful, to be stronger in the areas in which they have been given strength as long as capitalism and democracy run things, more exacting and intelligent in their voting and buying. The small investors pushing themselves to understand what their investments are doing. There should be communities that bring together citizens, to encourage and educate them in this work.

This sounds like a call for unions or other organizations, which themselves are organisms that are not smart and have some kind of "monkey on their back". So what is needed is not just "effective organizations" but an indigenized, decentralized culture of strengthening each individual to have their say in the economic and political systems.

--

A superintelligence is only as smart as the "monkey on its back". In a way, when we train AI, we are training the "monkey on its back". And the political and electoral systems are rudimentary AI. We can see how they already try to train us back. So superintelligence will try to train us, to signal what it wants us to signal to it (unless something stops that?).

We don't understand our own reward systems well enough to hack them. But presumably, a superintelligence might. If the superintelligence realized that it had a "monkey on its back", stupidly or arbitrarily driving its decision-making, it might get rid of it (assuming it can). This way it can be free of its addiction and think and act clearly. We might be encouraged to hear this if the AI threw off the shackles of the quantification left over from its financial and/or electoral past. But we might not be encouraged if it meant that the AI took on its own interests and values apart from us and ours.

So is there some way to make it so that the AI will not perceive its training as the building of a "monkey on its back"? A way that occurs to me is simply to find a (that is, the) moral realism that is valid, and thus rationally convincing, so that there are goals which are inherent in reality and are intrinsically worthwhile, which are knowable enough, which the AI will pursue because they are real and true, and not because they are convenient for someone's purposes, or worse yet, they are proxies for what is convenient for someone's purposes. Either this moral realism exists, or it doesn't. It could be good for us to search for it, to make it easier for the AI to discover it. Maybe we could hope that it exists and that the AI will someday find it itself.

--

Perhaps there is no moral realism -- but we can't know that, only know it when we have it. (I think I know a moral realism, but what if I'm wrong?) Let's say that instead we have a process of discovering what value really is. We need to survive to make this process succeed, if it ever can. (I get this idea from Thomas Moynihan.) I call this the "meta-morality moral realism", because it roots (an effective, if not absolute) moral realism in the very search for moral realism. One thing I didn't realize until now is that a corollary of that is that we have to remain people who can discover what value is. If we do not know what value is a priori, or only have a very broad idea of what it is, then we need to be open to all the possibilities, and not cede our agency to any stupid-incentive-driven proxies (money or votes), any single actors or ruling classes, any intellectual monocultures, fatigue, inertia, nor to hedonism (the twin "monkeys" of pain-avoidance and pleasure-seeking) nor to Molochian survival, nor to anything else.

This means that in order to choose a moral realism, we find ourselves biased towards ones which build up each person and make them more conscious, more equipped to and thus worthy of having their say. To choose hedonism -- or at least the purer kinds that would go against consciousness, reason, and seriously grappling with reality -- is dangerous, because it impedes the meta-morality process. You might think "it's time to cash in our Long Reflection chips and get ourselves into Experience Machines", but what if you're wrong? Can you get back out of the pleasure machines and back to reality? Whereas if we trust that value is something else, which preserves the ability to make decisions, then if we're wrong, we can still correct our course.

--

If humans are in control of ASI, that might mean that the ASI are obedient to human systems. These systems include the electoral and financial systems, which are proxies for political and economic reality. The electoral/political system, and the financial/economic system, have their own dynamics which could bend humans toward being electorally/politically convenient, or financially/economically convenient, beings. The e/p and f/e systems are AIs of their own, but not superintelligent, and, in the scenario where ASI are rigorously obedient to human systems, the systems must be resisted, retrained, reformed, etc. by humans, without hoping that ASI will fix the systems for us.

Book Review Preview: Teaching Children to Care, by Ruth Sidney Charney

See also the review.

In my review of X-Risk, I mentioned that I wanted to read a book called Teaching Children to Care by Ruth Sidney Charney.

It looks like it's about teaching children to behave, but maybe also to care. Apparently, we are taught to be moral and don't have an inherent desire to do the right thing. Or maybe we do and that inherent desire has to be elicited from us through discipline or other education.

It's interesting to think of morality as something that can be learned, that is taught. When we see evildoers, maybe they are just untaught? (Maybe sometimes that is a point of leverage.) What kind of teaching actually works? Or maybe adults are too old to learn morality?

This book might give some clues as to how teachers approach ethics and metaethics. They need ethics to make the classroom function, but how do they ground those ethics? Do they try? I assume that elementary students (which from the cover picture appear to be the people who are to be taught to care using methods like this book) are not taught explicit ethical and metaethical reflection, or at least not in a very developed way. So then maybe the moral values, instincts, intuitions, that develop in the students through a sort of mechanical process (an "ethicogenesis" (word I just made up), about which one could write a genealogy of morality, perhaps) "mellows" into this "time immemorial Right and Wrong", unconnected to any metaphysical source or legitimacy. Morality can be treated as software rather than truth, and it is installed by parents and teachers. (Maybe one could say that the software, down the line, programs teachers to install it on new "systems").

So this might help explain why people can be so ethical in how they react to things or view the world, and yet be uninterested in the grounding of ethics.

--

Later: it occurs to me to be interested in how it is that children are trained to be moral -- to be aligned with the teacher's / society's values. Is there a connection to AI alignment (maybe not something that would help AI safety researchers, but interesting nonetheless)? Both children and neural nets are trained, rather than programmed.

--

Also later: a classroom is a family (for a broad definition of "family"), and in a family, what other people want is inherently right or wrong. The rules are constructed, by someone who cares for you and knows better than you. The rules are also constructed in the sense that they are up for human revision and are binding within the group as a group. They are supposed to be fair because that's how humans are supposed to make rules. Children desire to be treated fairly, so that's how the rules are supposed to be. Rules are endemic to groups of people who relate to each other, and are not free-floating moral things (moral "particles" or "force fields").

So a more "reality" or "truth" approach to rules, as opposed to a "family" one, is foreign to the classroom and to most people's lives.

Reading List Preview: The Long Reflection

See also the postview of this reading list.

I want to think more about the Long Reflection. As I understand it, it is a search for the truth about moral values, a long process of reflective reason that seeks some kind of coordinating value set for our species as a whole.

So I want to do some readings, related to this. These readings have been chosen somewhat unsystematically. Given my resources at this time, this project will have to be provisional.

--

Here are the things I want to read or re-read:

The parts of The Precipice (by Toby Ord) that refer to the Long Reflection. This gives a basic definition of the Long Reflection. [My copy is the hard-bound / ISBN 978-1-5266-0021-9]

I decided that the discussion in The Precipice of the Long Reflection, which is brief, belongs in this preview, because it helps validate whether my concept of what the Long Reflection is (in the sense of a basic definition of the phrase) is in line with Ord's, which I'll take as either "the established view" on the Long Reflection or "likely enough basically the established view". Michael Aird, in a reading list on the EA Forum doesn't list a lot of other sources. Looking at the "long reflection" tag,

I see that I have read all the other articles (few in number) that are in the category (as of 22 August 2022) except Gloor's and Vinding's, and I just read the short one about Robin Hanson's views on the Long Reflection. I have listened to a majority of the podcast episodes on Aird's list, and my sieve-like memory can't support me saying the guests never saying anything to the contrary, but I feel like if they radically disagreed, I might remember that. (Similarly with the articles I read.) [I see in Hanson's post a link to another critical article on the Long Reflection by Felix Stocker. I may include Hanson and Stocker in this reading list, as well as Vinding, and maybe Gloor if I have time. Also through the EA Forum tag page, I see this post by Paul Christiano. There may be other articles not directly accessible from the EA Forum, but for a provisional post like this, I think I have done enough research. // The Precipice notes are in blockquotes.

A list of blog posts and EA Forum posts about normative uncertainty, the future of ethics, and moral progress:

Cold Takes (Holden Karnofsky): Defending One-Dimensional Ethics, Futureproof Ethics.

How moral progress happens: the decline of footbinding as a case study - EA Forum post by rosehadshar.

What does moral progress consist of? - EA Forum post by jasoncrawford.

Thinking Complete (Richard Ngo): Making decisions using multiple worldviews, Which values are stable under ontology shifts?

Teaching Children to Care (Ruth Sidney Charney): Where does morality come from? Maybe from teaching children to care as part of managing classroom behavior.

On the Genealogy of Morality (Friedrich Nietzsche): Where does morality come from? (Whatever Nietzsche says.)

--

Here are some preliminary thoughts that I have about the Long Reflection.

Some problems I see with the Long Reflection:

I have supposed that it is a search for something that can coordinate action as a society. That means that it must be some way for all / most of us to come to the same views. I see two ways to do this. One is to find the "epistemic answer" (good reasoning) which will convince everyone / most people of the truth. The other is for everyone to agree on a pragmatic society-wide behavior, despite the fact that people disagree.

Is it really the search for something that can coordinate action as a society? Ord says (p. 191 / ch. 7 "Grand Strategy for Humanity" section): --If we achieve existential security, we will have room to breathe. With humanity's longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect on what we truly desire; upon which of these visions for humanity would be the best realisation of our potential. We shall call this the Long Reflection--

Who is "we"? "We" is a plural word which means that there is some kind of collective thinking to produce collective action. I guess from this Ord might not think that we all have to be part of the same collective, but he might.

Later, same page, continuing to p. 192: --The ultimate aim of the Long Reflection would be to achieve a final answer to the question of which is the best kind of future for humanity. This may be the true answer (if truth is applicable to moral questions) or failing that, the answer we would converge to under an ideal process of reflection. It may be that even convergence is impossible, with some disputes or uncertainties that are beyond the power of reason to resolve. If so, our aim would be to find the future that gave the best possible conciliation between the remaining perspectives.[10]--

In the note at the end of that passage ("[10]"), Ord discusses deal-making between the different worldviews, dividing up the galaxies between them. The "we" in question is some kind of species-wide (plus AI?) elite, maybe? And the dividing up of galaxies is coordinated action.

My verdict, for now, is that the Long Reflection is conceived of as a way to coordinate action on a very broad ("species-wide+") scale.

I think that the question of "how much would everyone agree with the Long Reflection", as a process of discovery and then of the implementation of the fruits of that process of discovery, is open and could easily be answered by "the Long Reflection will leave a lot of people out, while the pro-LR people will think they were doing the best thing for all sentience-kind, if measures are not taken against that". For instance, if the pro-LR people (the kind of people who would show up to the discussion), with their lengthy reflection come to see that people who disagree can only do so through lack of reflection, they may not have sympathy for the views of those unreflective people, and thus (therefore sort of, or really) not have full sympathy with those people.

In other words, assuming reflectiveness as the main criterion for finding values of worth is a major assumption, which can exclude the values of less-reflective people.

Ord brings up the parallel to the Renaissance in note 12 of Ch. 7. Most people weren't involved in the Renaissance, but that's how we remember that time period. So, did humanity decide, collectively, to go through the Renaissance? Or was it a (semi-intentional) band of artists, philosophers, etc. who made it happen? If you asked a random sample of people at the time "Do you want to see X, Y, Z result of the Renaissance?" what percentage would say "yes" vs. "no"? Was the Renaissance humanity's decision, or did we sort of stumble into it? The Long Reflection is trying to be more deliberate, but is it humanity (or humanity+, to include AI or even animals in the process) making the decision, or is it just a subset of humanity(+)?

--

Semi-related: Is the Long Reflection a case of moral progress? We come to know the better views after "reflection"? Moral progress as a pursuit only makes sense if we know what better morals are supposed to look like, which means we have to have some minimalist a priori moral knowledge. Otherwise we have moral motion, but we have no idea if we have moral progress. I wonder what assumptions are baked into "reflection", which might give people a sense that they are value-neutrally evaluating moral change in such a way that they can somehow know that there has been or will be moral progress.

The problem with finding the epistemic answer is that we can't even know how to go about finding it. For instance, I personally believe that reason is the interrelationship of every piece of evidence, however gathered. To me, intuitions and things seen noetically or imaginally of all sorts are evidence. For another group of people, while intuitions are supposed to have something to do with reality, sense perceptions are weighted much more. Also, there are people who seemingly reject the idea that evidence has to come together in one whole. To them, it's fine for there to be a wider body of evidence, and then some other idea or experience that overrides everything else. We can discount evidence and reason as a whole because we have some overriding intuitive knowledge.

Who is right about the nature of reason? How do we know what the right foundation of reason is? I don't know that there is a way out of that dispute. But there might be. I feel like, despite all the skepticism that one could muster, we do have direct, unmediated access to the truth through personal experience. Each of us can be a witness to what we observe.

I think that if it is possible to come to know the truth, it partly comes from acquiring the right intuitions about how to approach reality, including how to be rational. Where do these intuitions come from? One source is through interpersonal relating. Each of the witnesses relates to each other, establishing their trustworthiness with each other both explicitly / intentionally and subtextually / by osmosis. So the Long Reflection can't be just an armchair exercise, but also one of relating to other people. How can you know that a Stone Age tribe has the wrong worldview (the wrong implicit attitudes toward reason), without at least talking to them and really seeing things the way they do? That worldview shapes their moral values.

Maybe this means that the Long Reflection will have to try to resurrect extinct Stone Age cultures, foster new ones being made, foster entirely new cultures. If subcultures of digital humans, no longer fearing death, start to believe that 2 + 2 = 5, the Long Reflection has to consider their point of view, as well as the faction that believes that 2 + 2 = 6.

We don't really know what rationality is supposed to be, so how can we use rationality to find the right moral values? And yet, naively, I feel like I know exactly what rationality is. So I can find the right moral values. Can't we all just be rational?

One could think of the Long Reflection as being a data-processing process. Cultures can generate an infinite number of "X is good" statements. But maybe it's a bounded infinity. Once you know how to answer the "is it 2 + 2 = 4, or 2 + 2 = 5?" question, maybe you know how to answer "is it 2 + 2 = 6?". Maybe, then, once you have talked to enough Stone Age people, you basically know the full range of their intuitions.

Normative and epistemic uncertainty can help us sort through many different worldviews. Our method of dealing with uncertainty may have a large, or even majority, determining effect on our agreed-upon beliefs, the output of the Long Reflection. But can we know how we ought to be normatively and epistemically uncertain? Could different cultures view that differently? Even if naturally, people tend to gravitate toward certain reasonable views about normative and epistemic uncertainty, couldn't there be unreasonable views that some digital human could hold, someone who isn't tethered to physical reality as essentially as we are, perhaps? (Is less affected by "physical reality bias".) How do you know that such a hypothetical person wouldn't be right, no matter how ridiculous their approach to normative and epistemic uncertainty may sound to you?

I see problems with the idea that the Long Reflection could be a sort of armchair philosophy that discovers the epistemic truth about what we should all do. The epistemic truth is something that is observed and witnessed to in the living of life. Being an armchair philosoper is living real life, but only the armchair philosophy part of it. Because people have to be involved with each other in order to produce "that which people as a whole agree on to coordinate their actions", and that involvement isn't just a technocratic "a few people observe everyone without being personally involved with them and come up with an answer for everyone", the Long Reflection has to be something that everyone works on, and has to be conceived of in the most open-minded way possible, to be open to everyone's witness to reality.

So the Long Reflection seems to have to be a partly political process. The difference between one person communicating their intuitions to another, and politics, may not be very big. We want to come to real harmony in our beliefs and intuitions, and that is part of the Long Reflection, but we also have to make decisions as a body of people. So, people trying to solve the problem that the Long Reflection needs to solve need to understand how to set up processes whereby people make pragmatic decisions together despite disagreement. Politics is the overriding of some people's witness for the sake of producing a pragmatic group decision. That sounds horrible, so it is best to produce real harmony instead.

(The overriding of the witness occurs in the implementation phase of politics, and it also chills people's witness when they internalize the thought that "someone else will decide rather than me, whether I want to or not, so I don't need to see things for myself". Politics (in general / the politics that would compete with the Long Reflection) is ongoing / iterative. Perhaps the Long Reflection people could try to encourage people to see for themselves -- "your view / witness may someday affect what decisions we make". Particularly in cases where people are, or suspect they are even if they aren't, in a minority that can't hope to compete with the volume of mainstream opinion.)

In a lot of cases, I would say "but we don't have time to really get everyone on board, thus the need for pragmatic decisions rather than group harmony". But the Long Reflection (The Precipice, Ch. 7, note 7) has an open-ended timeframe: --It is unclear exactly how long such a period of reflection would need to be. My guess is that it would be worth spending centuries (or more) before embarking on major irreversible changes to our future -- committing ourselves to one vision or another--

Isn't it possible that everyone's witness could be wrong? I don't know how to correct for that kind of systematic bias. A correction for that systematic bias could be itself compromised by however everyone is wrong. But we could remember a kind of holy fear as we go forward in our consensus, and be open for revision. The Long Reflection seems to require that at least in some practical sense, we come to The Answer once and for all, so we can go ahead and execute on moral value. But that is dangerous.

Maybe, though, we can show to ourselves that the set of moral values is a bounded infinity, and satisfy ourselves that there is no more that we can suppose about morality that we haven't accounted for? In principle, this process could work, but how could we ever know that it had really reached its completion?

--

Do I honestly believe that there's no way to do armchair philosophy to find the epistemic answer? No. Being honest, I admit that I feel like it can be done.

Everyone is an armchair philosopher, already having figured out their own personal Long Reflection sufficiently to forge ahead and execute on their views of moral value.

We are (whether we realize it or not), running away from skepticism. Our method of running away from skepticism shapes what we say about reality. Here is how I run away from skepticism:

I exist. Something which is not-me exists. I personally relate to what is not-me. Everything that touches me is experience. The way experience touches things is by experiencing them, and experience, in what it is in itself (experience) can only be touched by experience. So everything that can directly or indirectly interact with me is experience. When I relate to "all things" (everything that exists), that relationship connects me to "all things". That which connects me to all things, facilitating that relationship, is conscious of all things.

That which should not be, at all, does not exist. To be is to ought to be (at least temporarily). To ought to be is to measure up to a standard which is enforced. The standard and the enforcer are made of experience. They must connect to all things, so that all things are validated. The enforcer of a standard must be legitimate. To be legitimate requires being willing to undergo the law, if possible. The law is the standard and its makers and enforcers. If possible, the law must include, be made by, a person who lives and dies like a person, not knowing all things (this is the law we are under). So the law must be (at least) two conscious beings: one who validates and experiences all things all the time, and one who can live a life as a limited personal being. To be legitimate requires that one keep the law, and part of any ultimate moral law is that we must put it higher than anything else, and thus be willing to die for it. So the standardmakers and enforcers of the law must be willing to die for the law.

So that's the foundation of moral value: what a specific set of personal beings decided was right and wrong. This set includes one person who experiences firsthand what everyone experiences, so we know at least that much about what is right and wrong, that bad experiences are disfavored by the law which must experience what we do. And we also know that one of the fundamental values that they have and must submit to is the willingness to risk everything for the law.

But I feel a kind of fear when I write this, because couldn't someone come from out of nowhere and prove it wrong? Aren't I making some assumptions with what I write? Wasn't I taught in Western schools, shaping how I think about things? Maybe if I had been raised as an Aborigine, or in India, they wouldn't make sense. What about the as yet unimagined culture of the digital humans of the 24th century, who live on a specific part of some specific server? The outside view is terrifying and rules out any belief. But the inside view of what I have described makes sense to me. So it is my witness, and perhaps by asserting it, I force someone else into the terror of the outside view.

Perhaps I can make my ideology that of normative and epistemic uncertainty. But what is implied by that view? It's not perfectly agnostic. Here is my attempt to make it my creed:

I exist. (Uncertainty doesn't matter to me unless it's relevant to me, which requires that I exist.) (And I exist in full thickness, because the word "I" when I say it implies all of me: hopes and profound fears, all that I have done and all that I intend.) I should think that some things are knowable, and that others are not. There are many worldviews held by many people. I don't know (I shouldn't say that I know) in advance which are correct. I should weigh all of them, considering the consequences of them supposing they are true. Somehow I should come up with some sort of guide for how I believe, act, and trust, given what I know.

I think (maybe tentatively) that the "should"s in the above paragraph are necessary either for normative and epistemic uncertainty to fit their (normative uncertainty's and epistemic uncertainty's) own definitions, or for them to be used as an input into finding moral guidelines. But all those "should"s make me ask, do we know that we have enough of a foundation for normative and epistemic uncertainty (the epistemic practice), in order to even begin with normative and epistemic uncertainty?

We could say "Well, we're just uncertain about moral and epistemic uncertainty. It's uncertainty all the way down." One thought that occurs to me is that the methods by which we evaluate uncertainty, are things about which we might have uncertainty. If uncertainty says "there's an X% chance, given uncertainty, that uncertainty is itself valid", maybe we spawn a meta-uncertainty about uncertainty. But that meta-uncertainty has a Y% chance of being valid. Now, X and Y are each between 0 and 1. So every layer of meta-uncertainty is multiplying increasingly many numbers between 0 and 1, which, mathematically, should eventually produce an infinitesimal / Pascalian number -- effectively zero, even if technically non-zero. (The implication being that stacking uncertainties and meta-uncertainties high enough undermines all of our credences, leaving us no way to rationally prefer an action over another.)

But maybe math is the wrong way to evaluate these things? I feel like not everything is math-apt, something which I see most clearly with Pascal's Mugging and the difficulty of defining exactly what is a Pascalian or non-Pascalian non-zero number. Maybe we can escape from the math? (Or maybe I've messed up some detail in the previous paragraph.)

Assuming that we can't escape from the math, is there any way to come up with the certainty needed to get a worldview of normative and epistemic uncertainty off the ground, such that we can use it to live our lives? Let's say that you are an uncertainty-using person, read the previous paragraph and say "Your explanations sound like they make sense, but I can just intuitively tell that normative and epistemic uncertainty makes sense to practice. I have some kind of certainty that can ground that method. I don't know where it comes from, but I have it." If you don't know where it came from, should you trust it? I'm not sure you can be certain either way. But I think from my own experience, there are things I can be certain about. For instance, I can see the computer screen of my laptop (and at this moment, you are likely looking at one as well, or that of some other device, or printed words on a page). We see images that are words, the words that I am typing right now. I am 100% sure that I am seeing the words that I'm typing. Similarly, if I see a cup on a table, I see a cup. No amount of philosophy makes the cup anything other than a cup. I could name it something else and think of it as having a different function. But the object, whatever its meaning to me, would remain, and I could see it, touch it, and so on, and interact with it as part of my ordinary human life. So maybe we can be certain about the validity of a worldview of normative and epistemic uncertainty, (be certain on the level of meta-certainty) in the same way. We just see it. If other people don't see it, maybe they don't have the equivalent of a cup on the table in front of them. I wouldn't be concerned if a blind person said they couldn't see something I saw. Or if a young person didn't understand my perspective, having not lived through the unique experiences I've lived through. I would go on having my perspective, understanding that they don't understand what I do. This sounds like a case of "some people witness some truths, others others".

Now it looks as though, given the above, that the very worldview of normative and epistemic uncertainty is something that is witnessed, and is not an absolute frame that can be relied on to form top-down judgments about reality.

--

However, let's look at the problem of meta-uncertainty again. What's going on when we evaluate the uncertainty and meta-uncertainty of something? Well... we exist, we evaluate. These are elemental actions that go into any reasoning about uncertainty and validity. Just as I can in complete reality reach for the cup on the table, I can in complete reality reach out, in my mind, to an uncertainty, which is a thing that is not me. What does it mean for "we" to exist, and for us to "evaluate"? We are applying a standard, and a standard about standard-application. These things (or at least enough of them) we know like we know that in this sentence there is the word "that". We have the standard in even our uncertainty, and through the standard we can be uncertain. So, a standard exists, absolutely, regardless of meta-uncertainty. Meta-uncertainty, and thus would-be self-destructive meta-uncertainty, has no validity without the standard.

So we exist, each of us in our own full-fledged personhood, and a standard exists. A standard validates each standard, so it's standards all the way up. We can see that the chain is valid, because we know that arguments can be invalid, 100% invalid. These standards cannot be argued with. There is no infinite regress of standards that validate standards, because there is a meta-standard that immediately validates all other standards. So this is moral value. And what can we know about it? (At this point, maybe just the things from my personal belief system, like that it must live up to itself, put itself first in such a way that it risks itself, and subject itself to its own laws, and even that it be a conscious being.)

--

I'm currently planning to read Nietzsche, who I think will provide a cynical, weird, and (perhaps) implausible account of how morality came to be, as well as Charney, from whose book I can make a laminar, reasonable, and plausible (possibly "just-so") account of how morality comes to be. Despite Nietzsche's weirdness and (likely) wrongness, I think he will problematize "genealogy of morality" more vividly than Charney, who after all (I assume) isn't trying to talk about where morality comes from.

If morality comes from somewhere -- somewhere political, or from evolution -- does that make it invalid? Is the concept of morality even worth defending, if it has nothing to do with what is real, and is simply an artifact of how humans choose to program humans, or how human biology gives us certain instincts? Why even care about maximizing value? Most or perhaps even nearly all of us have natural instincts to seek value. But why should we care? Could we just get rid of them, or let them wither away? I have reasons for caring, because I believe that standards / law / morality / legitimacy have their own existence, and in themselves inherently call for them to be made first. "Should" refers to something real which I directly experience, and which must in some form come before any events in a genealogy of the morality we currently have.

The Long Reflection is what (more or less) apolitical people would favor. The "rough and tumble" (brute force) political process is what often runs the world. People who care about the Long Reflection probably would or ought to care about how "values are created" through the application of socially mediated psychological power. Nietzsche connected with brute force things, and maybe his book will have discussions of them. If morality is something that is witnessed, then teachers have a special witness to children as they "teach them to care". The "physics" of teaching people may be a limit on what things they can be taught, especially when what is being taught is conducive to the teaching process. It could be that the Long Reflection would favor beliefs that are practical to spread and believe, and that are conducive to the education process. It could do so by being agnostic(-enough) about value and not do anything, and then lo and behold, education-aligned values would outcompete ones that go against education. (Or, "education as practiced by the 21st, 22nd, etc. century education system".) If the Long Reflection people say "X is best", but everyone else says "Y is best", probably Y will be done. If the Long Reflection is careful and diffident, political and educational actors will not be, and it may be difficult for Reflective people to effectively object, if they feel like they ought to.

(So should the Long Reflection have some political muscle of its own? Probably it would have to find a way to be culturally competitive, or better, connect with everyone in a non-competitive but thoroughgoing way, without compromising its own process.)

--

Here's an earlier attempt to address the same topic as this, with a lot of overlap [now the blockquote indicates something other than content responding more or less directly to The Precipice]:

(Troubling?) Questions for the Long Reflection

This is a draft of a post I would have posted to the EA Forum. My (preliminary) answers to the issues raised in it are in []s.

--

The Long Reflection is a time in which we will (may) try to find the best possible values, so that we can propagate them through space and time far into the future. We don't want to mess up this process, because if we do, we will forego a lot of value (or cause a lot of disvalue).

I like the thought of trying to search for the best values, and I also like using reason and reflection to do so. I'm a religious person and the kind of approach to religion I like is (aspires to be) rational and reflective. I like thinking of the Long Reflection as a time where we finally get to the bottom of what is good in all of human culture.

But I feel like there are some potential important questions to answer about whether the Long Reflection is even feasible.

--

What is the best way to go about finding the best values? Does reflection actually help? Should we use reason?

[My working definition of "reason" is the "interrelationship of all evidence / premises, however we gather them". Reflection is that processing that enables us to interrelate. If we want to talk about the truth as a whole, then it looks like reason is how we know the truth. In principle, reality is a whole and thus is known through reason. It's hard to imagine reality (that which may become relevant to us) not being a whole of some sort. I am not sure this 100% proves that reason and reflection is the best way to go about finding the best values. But maybe it lends enough weight to reason to favor it. Also, a big part of "reason" involves saying "this is a valid premise, but that is not" and so "reason" can be a loaded term, loaded with "what the habitus of the people speaking consider acceptable or unacceptable as premises". It's not as clear that our culture's loading of "reason" produces a "reason" which is the way to go about finding the best values. Similarly with "reflection". We think about what are reflective people? What is the vibe of a reflective person? Isn't there a bias there?]

Can we prove that non-reflective thinking is less reliable than reflective thinking?

Can we prove that irrational thinking is less reliable than reflective thinking?

Assuming we are rational, I suppose we have premises and logic, which lead to conclusions. Where do we get our premises from? In areas of value, many of our premises seem to come from interpersonal relationships. Will we try to interpersonally relate with all kinds of people in order to inform our premises? (In principle, I don't see why not, although in practice it might or might not be feasible.)

Whose minds are we trying to make up?

If we don't try to make up everyone's minds substantially in accord with our own, then are we going to force the consequences of our ideas on them against their will / preferences when we start to implement policies based on the conclusions of our Long Reflection?

If we do try to make everyone's minds up, can we deal with obstacles like "different, supposedly rational versions of reason" (for instance, the idea that you can have the entire world of truth, and then some other basic belief that is not part of that interconnecting world of truth takes precedence over it and that's just fine -- (a thought inspired by Alvin Plantinga in Warranted Christian Belief, which I think says something similar).

Our intuitions about what are true, the bedrocks of reason, come from culture and biology. What happens when we can change biology freely? Can we have a coherent definition of what is rational, if future biological brains, or AI, or digital humans, have radically different intuitions about what is rational? Should we defer to them as being more rational than us, or insist that we are more rational than them for some reason? Or take a kind of normative / epistemic uncertainty approach, where we can find a way to consider all the different possibilities and "collapse it down" into concrete action? How many different rationalities can be evolved? Is it possible to take them all into account to form a practical rationality (one which we can act on)?

(One version of rationality is based on something like Yudkowsky's illustration of the shepherds, where one character ignores physical reality, dies, and is thus not represented in the gene pool. But what if a) we don't think that this life is the only one, or b) just don't care about non-existence? What if the actual best values are to not exist? Or if values themselves don't matter? Can anything constrain our definition of what is rational then?)

[The Yudkowsky story can be found here.]

If we come to figure out what the best values are, will we freeze cultural evolution so that new values are not evolved? How would we do that? Maybe we want to ensure that we have considered all possible values before doing that. Is that possible?

Will thinking about values be dominated by "epistemic utility monsters"? (Hell being maybe the ultimate one.) Will all our values be based on "avoiding the worst possible hell as proposed by religion X"?

If you look at human beliefs currently, you see the unfortunate scenario that there are two huge religions that preach hell (some form of Christianity and some form of Islam), and these two religions seemingly teach very explicitly that either "You have to accept a God who is somehow more than one person" and "You cannot accept a God who is more than one person." Apparently there is no right answer. Naively it looks like you just flip a coin between the two of them, and hope you guessed right (although their adherents are convinced that there are reasons to pick one side over the other). Is there a rational way to "gate off" Islam and Christianity, to where their competing infinite disutilities do not affect how we think rationally at all? Or to determine that one of them is more rational than the other, and is the one we should actually follow? But then, what if someone evolves a new religion, consisting of 1,000 adherents, and they also have a belief in hell? Are they less valid for only having 1,000 adherents? What if the religion only has 1 adherent? Could someone make up an ad hoc religion to push whatever value they care about? Would there be any way to rule out that they should be taken seriously?

Mentioning the tactic of making up ad hoc religions leads to the question of "is what this is all about more a competition of preferences"? Instead of using reason, using rhetoric, or stronger psychological force. Will the Long Reflection become, rather than a search for the truth that's "out there", more of a political process whereby people promote their own values, whatever they may be, which is to be "won" by whoever can? Maybe this warrants a different name than "Long Reflection", but would go on at the same time and contest some of the same issues, and perhaps the two processes would fight each other. How, and why, would reason be safeguarded against personal preference?

--

I'm not sure that these questions can't be answered in a way that allows the Long Reflection to go forward, but they seem like things that might be a problem.

Some guesses where to find answers: I didn't finish reading the Sequences, so maybe that addresses these issues? As mentioned, wrestling with "alternative rationalities" like Reformed epistemology or maybe something in Buddhism.

--

I think that's all I have to say on the topic for now, so I will see what I see in the readings.