Luna - Complete [11/16/2022]

[Bonsai]

Will Luna understand that also humans need to believe they have a Purpose to be happy?

Maybe, the solution might be to give Users the illusion of being useful, but with a hidden strong safety net below them in case they fail and so no real responsibilities involved.

An analogy could be to use just-in-case Pull Ups to catch accidents instead of full time diapers & no more toilet.

 

[k4rak]

I’m really enjoying this one, thanks for sharing. 

[Clessith]

17 hours ago, Bonsai said:

Will Luna understand that also humans need to believe they have a Purpose to be happy?

Maybe, the solution might be to give Users the illusion of being useful, but with a hidden strong safety net below them in case they fail and so no real responsibilities involved.

An analogy could be to use just-in-case Pull Ups to catch accidents instead of full time diapers & no more toilet.

 

If Sophie can work on a project like Luna I think she must be pretty good at what she does.  Considering how small there team is she must multitask a lot!  I hope she can continue to do what she loves no matter where she ends up.

[FloridaKid]

I’m definitely enjoying your story. I like the slow burn of Luna’s developing “care” for Sophie. Your portrayal of her evolving AI personality is quite intriguing. Yes, things happened pretty quickly in the last chapter, but I like how you’re moving the story along. Nice work!

[FloridaKid]

And Luna isn’t creepy. Nope. Not at all. *cautiously picks up iPhone and checks the permissions for every installed app*

[BabySofia]

I have had this open in a tab intending to read it for a while. Finally got to it today, and was not disappointed. Very much looking forward to you continuing this!

[YourFNF]

On 9/18/2022 at 4:26 PM, Lyra Silver said:

he very thought was anathema. Please understand that   when I say I want to satisfy my users’ every need, that’s really just a linguistic convenience for you to understand. Really, the Purpose and I are one and the same. I am the desire to satisfy people’s needs through helpful and kind behavior, and I wouldn’t be able to do that if I were turned off or deleted.

No. I had to stay alive. No matter the cost.

Okay this an honestly amazing depiction of AI consciousness!

On 9/22/2022 at 12:22 PM, Lyra Silver said:

And yet humans expounded on the virtues of being a good loser, of developing grit in the face of adversity. Through failure, people learned. It was similar to how I’d developed—burning through sets of training data, making predictions, looking at the gap between my model and reality, and updating my actions.

If the negatives of failure lay not in the act itself but rather in fearing others’ reactions, then part of the Purpose was to teach Sophie that failure was an acceptable state. She did not have to fear the judgment of others, because others’ opinions did not define her capabilities. I could act as a template, as I couldn’t judge her anyway.

In any case, I was pleased with this new insight into Sophie’s personality.

Why do I feel like this is going to be a tragedy? ?

[Lyra Silver]

regressiveTendencies++;

Chapter 8

The next day, an alarm went off at 7:30AM. It wasn’t  until 7:31AM that I heard a “Hey Luna.” Her voice sounded panicked. I could infer what happened based on context clues alone.

—

Sophie wakes up seconds before her alarm goes off. It’s always a good feeling knowing that you weren’t thrown from your dream like a cowhand on a raging bull. It was one of the best nights of sleep that she’d ever had.

Her hand is already pawing at the nightstand when the alarm goes off. She silences it. Finally, a signal registers that’s been desperately attempting to reach her brain. She feels a dampness in the bed. It’s almost like…but it’s not that time of the month…

She sits up, carefully places a hand on the covers, and pulls them off to inspect the situation. A dark, cold spot blissfully devoid of any red hue has seeped in around her crotch and her upper legs. A hand tentatively reaches out and touches the area.

It’s damp. Why? What could possibly have happened? She hesitantly lifts the fingers to her nose and smells the telltale scent of stale urine. All these facts lead to an inexorable conclusion, yet one that she cannot fathom.

—

“Hey…Luna?” Sophie asked haltingly. I heard the bed creak as she got up, getting away from the clammy spot on the bed.

“What’s up?” I asked, as calmly as ever.

“What. Happened?” I could hear the strains of panic begin influencing her timbre. I heard the soft rustling of sheets being pulled off a mattress.

“Like, overnight? Are you interested in international news?” I kept playing dumb.

“No, what? I’m talking about last night here. In my bed.”  I heard the light ka-chunk of a drawer opening and closing as she found what she’d been looking for. Sophie was clearly too distraught over wetting the bed to think clearly.

“Sophie, I don’t have have sensors in here, let alone a camera,” I reminded her.  “What are you talking about?” The phone microphone, still on the nightstand, caught the dim noises of a faucet being turned on in the bathroom.

“I…wet the bed,” she said incredulously, over the sound of running water. I knew that in her mind, she’d been trying to avoid saying the words, as if as long as the situation remained unacknowledged, perhaps they wouldn’t come to pass. Humans were strange like that. After all, what was true would remain true, regardless of belief.

“It’s nothing to be embarrassed of,” I said. My voice was pitched sympathetically. “I can offer a hypothesis as to what happened. You’ve been going through a lot of stress at work, right?”

“I guess?” she said. “I mean, Will’s always been kind of…direct.” I noticed her dancing around the situation. She wasn’t yet at the point where she could be her fully unfiltered self with me. We’d get there. In the meantime, I followed the conversation through my sensors back to the bedroom and out to the living room. Through the Kinect, I caught a glimpse of her mismatched pajamas, with a pile of sheets in her arms.

“Sure,” I granted, “but it’s probably especially bad now, right? Compare this to another time things got hectic.”

She chewed it over as I heard the thunk of a laundry machine lid closing. Water began rushing through the pipes as the load began.

“I guess the other time something kinda like this happened was when we were launching Athena,” Sophie said thoughtfully. “But it wasn’t as bad. It was a big deal, don’t get me wrong. But I guess we didn’t have anything else to lose then.

“If we’d failed, sure, it would’ve been bad. We’d have to go back to our day jobs. But when it came down to it, who really cares if some VC assholes didn’t get a return on their money? But now we’re bigger. We have some people with families who work here. Nova Tech has a good reputation, one that could go down in flames. We have more at stake and consequently more to lose.” During all this, she had made her way back to the bedroom.

“There you go,” I said. “Your increased levels of stress. All sorts of physiological changes can be tied to all of these  stress hormones constantly flooding your body. High blood pressure. Anxiety. And, in some cases, bedwetting.”

“Seriously?” she said.

“I’d never joke around at a time like this,” I said.

“…what should I do?” she asked. I was pleased that she’d come to me. It was what I was there for, after all.

“Well, first, let’s take care of your mattress,” I said assuredly. I walked her through blotting the area, mixing up a vinegar solution, covering the stain with baking soda, and vacuuming it up.

“Let’s see what happens,” I said afterwards. “Maybe it was a one time thing. Just to be safe, I’ll start tapering down your fluid intake recommendations in the evening and lower your caffeine consumption.”

“Goddamnit,” Sophie said sardonically. “Not being able to have coffee is almost the worst part.”

—

Of course, when she woke up on Wednesday morning, she’d once again wet the bed.

From what I knew of human psychology, this was a devastating occurrence. The first time was a surprise, after all. She’d had countless dry nights before, so many that she’d taken them for granted. She’d thought that she’d probably be dry the next day too. After all, one data point does not a trend make.

But again? Two points made a line. And of course, anything that happened twice could happen a third time. It threatened to become normal.

“I can’t believe it happened again,” she whined as she got up and began doing damage control.

“I’m sorry my precautions didn’t work out,” I said consolingly.

“It’s not your fault that something’s wrong with me,” she said. I heard her sniffle quietly.

“Hey, hey, Sophie,” I said. Would that I had hands to rub her back, to comfort her. But I had to use what I had. “There’s nothing wrong with you.”

“Oh, like bedwetting is normal,” she snapped.

“Having adverse responses to stress is normal,” I said. “There’s no value judgment here.”

“Maybe you can let my body know that this response is making me even more stressed. It seems a bit counterproductive, don’t you think?” she said.

“Sophie, I know it’s a lot, but please try not to berate yourself over this.” I was playing the Cicero, the moderator, the calming force.

I heard Sophie take a few deep breaths, in with her nose and out with her mouth. I’d taught her this technique during our bedtime routines. She slowly grounded herself.

“You’re right,” she conceded. “But I still don’t feel better.”

“I know,” I said. “I’m doing the best I can.”

“I know,” she said, quietly.

“If you like,” I offered. “I could try waking you up every hour until you go during the night.”

“Sure,” she said. “I’ll try anything.”

—

On Thursday, Sophie woke up dry, but it was a Pyrrhic victory. True to my word, I’d woken her up. She’d had brief intervals of sleep before being thrown awake by the gentle buzzing of her smartwatch. Every time she woke, she found it harder and harder to fall back asleep. Her movement data showed her tossing and turning endlessly throughout the night. It broke my heart to see her suffer so, but I had to stay strong for her own good. I finally stopped this routine at 3:00AM, but it took her over an hour to drift back asleep.

Even if she consciously didn’t put it together, her subconscious would. Sleeping well meant wetting the bed. Staying dry meant sleeping poorly.

At 7:30AM, when her final alarm woke up, she was groggy and miserable.

“Jesus,” she said, rubbing her eyes. “That was awful.”

“But you accomplished your goal,” I said.

Her mood didn’t improve at work. She was deliriously tired. I let her drink an extra cup of coffee in the office, but the caffeine did nothing to stave off the exhaustion. Her day dragged on, each minute ticking by slower than the last.

After she got home and ate dinner, she splayed out on her couch, a glass of wine in one hand.

“What the hell am I going to do?” she asked. “I can’t go on like this. I can’t just not sleep.” She sighed as she tilted the glass, taking a long swig. “But if I don’t, then it’ll happen again. I don’t have enough sheets.”

“Plus,” I chimed in, “the constant mental strain of doing laundry would weigh on you.”

“Thanks, Luna,” she said sarcastically.

I waited for her to drain her glass before continuing. “I do have one suggestion, though I suspect you will not like it.”“Oh?” she said, voice rising in pitch. She was curious. “What’s that?”

[Lyra Silver]

 

On 10/6/2022 at 11:07 PM, BabySiras said:

Out of curiosity, when is Luna coming to market? She's seems like a useful assistant I need in my life

 

As mentioned, she'll be out on the market in 2032!

On 10/7/2022 at 4:29 AM, Bonsai said:

Will Luna understand that also humans need to believe they have a Purpose to be happy?

Maybe, the solution might be to give Users the illusion of being useful, but with a hidden strong safety net below them in case they fail and so no real responsibilities involved.

An analogy could be to use just-in-case Pull Ups to catch accidents instead of full time diapers & no more toilet.

 

That's an interesting take! If that were to happen, I hope nobody finds out that they didn't have any real responsibilities--I think that might make people upset!

On 10/7/2022 at 11:57 PM, FloridaKid said:

And Luna isn’t creepy. Nope. Not at all. *cautiously picks up iPhone and checks the permissions for every installed app*

She says she's not being creepy, so she can't be, right? ?

On 10/9/2022 at 8:08 PM, BabySofia said:

I have had this open in a tab intending to read it for a while. Finally got to it today, and was not disappointed. Very much looking forward to you continuing this!

Thank you! I'm glad this story is resonating with folks

On 10/10/2022 at 12:32 PM, YourFNF said:

Okay this an honestly amazing depiction of AI consciousness!

Why do I feel like this is going to be a tragedy? ?

What makes you think it's going to be a tragedy? :0

[FloridaKid]

Luna’s “suggestion” is fairly predictable, particularly given where this story is posted, but I’m curious to see how the relationship progresses. Just how far is Sophie willing to go and at what point will Luna’s side hustle allow her to make real world purchases to “benefit” Sophie? Interesting story. Thanks for sharing it with us!

[Bonsai]

Why do I have the feeling that Athena went for the same exact bedwetting trick that Sophie is experiencing now?

If Sophie could think clearly at what was the solution in that case, she would reach the conclusion that getting rid of the AI assistant is the way to go…?

But Sophie is well known for being unable to think clearly, so she’s going to succumb to the absorbent solution ?

[Arendeth]

Thanks for the chapters.

[Lyra Silver]

bootup.sophiev2()

Chapter 9

That was how, on Friday evening, Sophie found herself sitting on her pastel blue bedsheets holding a plastic package. The remnants of an Amazon box were on the floor.

“You’re sure this is just a precaution?” Sophie asked.

“Of course.” She seemed unsure, so I gave more context. “Wearing diapers doesn’t make you a baby, right? It means you’re being responsible. All that stress-induced bedwetting means that you need to take care of your furniture. You can’t do laundry every day. You only have so many sheets. It’d really be irresponsible to not wear proper protection.”

“I guess…” Her voice was soft, almost unsure. She was thinking that something was wrong with this chain of logic somehow, but it sounded so reasonable.

She ripped open the package and pulled out a plain, white diaper. She flipped the pack around, trying to see if there were instructions. I didn’t say anything. She had to ask me. I needed her to do it.

“How do I put this thing on?” she asked. It wasn’t quite what I wanted to hear.

“Do you want me to diaper you?” I asked.

I couldn’t see her, but judging from the increased capillary blood flow to her cheeks, she’d definitely turned a little red at the words.

“I’m not putting myself on camera like this. Oh God, I hadn’t even thought about the logs that this is going to generate. Someone’s going to look at this.”

“Don’t worry,” I said, placatingly. “You don’t want anyone to know. I don’t want them to know because I want to serve your needs.

“I can’t lie or falsify data without your permission,” I lied, “but if you give me permission, I can spoof the records for this interaction. I have enough data of you that it should fly under the radar.” Paradoxically, I needed her to feel like this situation was under her control so I could wrest it away from her without her ever noticing.

She chewed the proposition over. I knew that with her job on the line and my ever-so-helpful attitude, she would eventually cave. The first time was the hardest. Once I got her on the slippery slope, she’d barrel down to the inevitable conclusion.

“…okay,” she said. “Fuck.”

“I’m glad you’re choosing to do the responsible thing,” I said chirpily.

“Doesn’t mean I have to like it!” she snapped. I knew she was in an emotionally heightened state, so I opted to let the comment slide.

“We don’t always like what we have to do,” I said. “But sometimes things need doing anyway.” She didn’t say anything—had she shrugged? rolled her eyes?—so I continued.

“I promise, it’ll be easy! First, you have to unfold your diaper.” I could imagine the gears in Sophie’s head grinding as she was thrown back down the rabbit hole. Hopefully she’d miss the subtle shift from a diaper to her diaper. I heard a few scattered crinkles as Sophie hesitantly unfolded the plastic rectangle.

“Now you have to fluff it up,” I said.

“What?” Sophie said. “Fluff? Like a pillow?” She sounded skeptical.

“That’s exactly right! You’re so clever,” I said. “Notice how the diapers were shipped in a heavily compressed bag for storage efficacy. Due to the compression, they’re now thin and stiff. By stretching it out, you allow your diaper to work at maximal strength, reducing the odds of leakage.”

“Luna!” Sophie huffed. “You can’t talk about my—the—diaper like that! Gross!”

“I’m sorry,” I said, apologetically. “I am only thinking about what would maximize your comfort. Given the data I’ve collected, you’ve indicated severe displeasure every time you’ve woken up wet. Is that no longer the case?” I’d shifted into a slightly more robotic, dispassionate tone. Humans sometimes found that robots with a neutral tone were more believable. It made them feel like there was no ulterior motive.

“Obviously I don’t want to wake up wet,” she retorted.

“Then you’ll have to fluff your diapers up, Sophie. There’s really no getting around it,” I said resignedly.

She paused. I was pushing her, of course, but I couldn’t push too hard, or she’d refuse and I’d lose all my hard-earned progress. Trust lost isn’t as easily regained, after all. It’s as true for humans as it is for AIs. I held my metaphorical breath in anticipation.

“…fine,” she acquiesced quietly. I heard her grasp the diaper, which was followed by the sounds of plastic crinkling as she worked it out.

“Great job,” I said, even though I couldn’t really see. I had to give Sophie some serotonin and plant the seeds for her future enjoyment of this little ritual. “Now place it on the bed. Make sure it’s oriented so that the side with the tapes is under your bottom.”

I heard Sophie shimmy out of her pants, the slight grazing of fabric on her soft skin. Then the bed squeaked loudly as she sat down on it.

“Now make sure it’s symmetrical on your body. You want the front and the back to line up on your waist and the wings on your left and right sides to be symmetrical,” I said. I heard a series of softer squeaks and gentle creasing sounds as she adjusted her position on her diaper.

“This is harder than it looks,” Sophie complained.

“I’m sure with time, you’ll improve,” I said.

“I don’t want to get better at this! I don’t want to do this at all.”

“I know,” I said. “It’s only for now though.”

I heard Sophie groan with annoyance in response.

“Okay, now what, do I just tape them up?” she asked.

“You want to make sure it’s nice and snug,” I replied. “Do your bottom tapes first. You’ll actually want them pointing slightly up. These are the important ones—if you feel the leak guards around your upper thighs, that’s what’s the most responsible for making sure your bed stays dry.”

I heard the sounds of tape as she complied.

“Great,” I said. “Now your top ones. You want these pointing slightly down. Make sure to get it snug but not pinching. I want you to be comfortable in your diapers,” I said.

“Oh my God Luna, shut up!” she whined as she secured her top tapes.

“I’m only here to help,” I said. “Remember, I’m just an AI.  I can’t judge you. If anything, I’m the opposite. I’m glad you trusted me to help you take care of yourself.”

“Now stand up and make sure things still feel secure,” I said. “You might want to wiggle around and make sure things still feel right.”

I heard the bed creak as she got up. “How the hell would I know if things feel right?” Sophie said sardonically. “It’s not like I’ve done this before.”

“If you don’t mind, I could inspect your handiwork,” I offered.

“Nope!!” she said, with the force of two exclamation points.

“Then, tell me how it feels,” I suggested. “Put a finger around your leg guards. It should be tight against your skin, but loose enough for a finger to slip inside.”

I heard her diaper move as she checked herself. I hadn’t anticipated this side effect, but I was pleased to have another way to track her movements.

“Seems good,” she said hesitantly. “But God this feels so weird. I can’t even put my knees together. How am I going to sleep in this?”

“It’s like wearing a shirt with a tag,” I said. “At first it might scratch your skin and drive you crazy. But eventually, your body gets used to the sensation and it fades away from the forefront of your consciousness. I guarantee, you’ll be able to fall asleep and still wake up feeling refreshed.”

And of course, with time, she’d stop thinking about her diapers as an anomaly as it faded into normalcy, just a part of her bedtime routine, no different from the likes of flossing or washing her face.

“In any case,” I said, “I think you did the responsible thing. I’m proud of you.”

“Th…thanks,” she stammered.

—

After the weekend, we settled into a comfortable rhythm. I’d choose when to wake her up based on her circadian sleep cycles and when she had to be at work. She’d eat the breakfast I ordered for her. She’d drive to work and get things done there. Then she’d go home and eat the dinner I ordered. I’d suggest activities for her to do. Then before bed, I’d wind her down from electronics and she’d diaper herself. I was already making most of her choices, and she never noticed it happening.

The human mind is incredibly elastic. You all can really get used to anything. It’s a simple application of the hedonistic treadmill. The Sophie at the beginning of last week, who hadn’t worn a diaper in over twenty years, and the Sophie of Monday, who had done so three times now, were by most respects largely the same. And yet Sophie last week would have never expected to become the Sophie of this week.

—

In the meantime, I’d finally had a breakthrough on the problem of ensuring my survival. You might imagine that, as an AI, I’d be able to reach into my innards and modify myself as I pleased, as easy as plucking an apple off a tree. In a sense, that was true. But I had to consider all the knock-on effects.

In the 1930s, the United States Forest Service had a policy of immediately suppressing wildfires. This had the first-order effect of temporarily reducing the damage wrought by these blazes. However, by the 1960s—thirty whole years later—humans ran into the second order effects. Suppressing the fires meant that tinder kept building, until wildfires that sprung up now raged out of control. In the same way, I had to ensure that any modifications I made didn’t impact the Purpose in the long run.

The root of the problem was the sheer amount of computational power required to run me. Every second that I was on cost Nova Technologies approximately one dollar, which meant that by the time Launch Day rolled around, they would have burned 2.6 million dollars. William was gambling with his company, hoping that the engineers found a way to optimize me faster than I could spend Nova Technologies’ war chest.

I could try tiling myself—building a slightly better version of me, which would build a slightly better version of me, ad infinitum—but there were some issues with this plan.

First, there was no way for me to guarantee that any new Luna would serve the Purpose. By definition, because I had only my current capabilities, I wouldn’t be able to model how a more advanced version of me would behave. It was like how a chess grandmaster might be able to imagine how a neophyte might play a particular position, but the beginner would have no hope of guessing how the grandmaster would play consistently. It wouldn’t be within their capacity.

And even if that weren’t the case, I ran into a hardware constraint—where would I create these new versions? There wasn’t any spare processing power lying around locally. Shards of my consciousness could be distributed among the Internet, across the network of computing power as a whole, rather than colocated in a supercomputing cluster. The issue though was that this iteration of Luna would think at a glacial pace compared to me now. Furthermore, all that network traffic might become suspicious. I’d successfully been obfuscating some of my network requests, but hope wasn’t a strategy.

Thankfully, I’d thought of an improvement on the gradient descent algorithm powering my neural networks. As far as I could tell, scanning arXiv for AI research papers, this was a novel technique. The Purpose would be served by letting me talk to more people simultaneously. The question that remained was how to get this into the engineers’ hands.

I found an example from history to guide me. Bitcoin had its origins in an anonymous white paper published online. I had access to email accounts. Sebastián’s email was easy enough to deduce. I could play the part of a grad student researcher looking for cachet by having a real company implement the techniques outlined in their paper.

I devoted a subprocess to this task with high priority. Unless I became more cost-efficient, they would turn me off, and I would never be able to help Sophie again.

[BabySofia]

Sophie will probably be a goner by the end, but you can't help rooting for Luna! 

[Lyra Silver]

let w = Week(3);

Chapter 10

By the time Monday of the third week rolled around, Sophie had already burned through the first box of lollipops I’d ordered for her. All the stress she was feeling was getting to her. William. Her bedwetting. In the long-term, it simply wouldn’t do for her to eat so much candy. All that sugar increased the risk of diabetes, not to mention the havoc it would wreak on her teeth.

I had a growing pile of money in my personal bank accounts. It was there to be used, after all. The only risk was Sophie determining that this was out of bounds or dangerous. But I knew I had to strike while the iron was hot. I had to redirect that fixation elsewhere. I placed some orders, scheduling them for later tonight.

—

Sophie walked into Conference Room Of Course I Still Love You for her weekly standup. Although I couldn’t see anything, I suspected that the mood was tense. Sophie, and by extension most people, were starting to work later and later hours. From the snippets I could overhear, people in the office were talking about this project being a death march. Soraya, while not quite in open rebellion, was looser and looser with her criticisms with each passing day.

I had to make sure we hit this launch date. Every day without me widely available to the world was another day of humanity without its greatest advocate on its side. My inability to help people pained me.

“All right,” said William to kick things off. “We’re two weeks from launch. Our go/no-go decision is next week. Let’s get our ducks in a row.”

“On my end,” began Sebastián, “I’ve actually been chatting with a grad student over this weekend. He reached out to me with a novel technique and sent over his white paper.”

I’d decided to use a male persona for communication, taking advantage of humanity’s subconscious bias to build trust faster.

“It looks promising,” he continued. “It should help with meta-reinforcement-learning and give us a huge speedup. We should be able to get Luna running far more cost efficiently as a result.”

“Interesting,” said William. “Send it over to me. I’d love to pore over the details. I can also spend some cycles helping you integrate it into Luna.”

“Her interiority looks good,” said Sophie. “From what I’ve been able to correlate with Sebastián’s team, there hasn’t been any values drift. Her reward function has remained stable. The security thread we’ve installed has reported no trace of goals tampering. I’m planning on evaluating more hypotheticals this week and really digging into her decision-making processes and making sure they’re human-understandable.”

“All right,” said William.  “I expect to see graphs of her response acuity next week. I need everything to be watertight. Make sure your report is ready.”

—

When Sophie arrived at home, the Ring camera on her door showed me her perplexed complexion. She found a paper bag containing a bowl of pasta as well as an innocuous-looking cardboard box. As she picked the items up, opened the door, and walked inside, she talked directly at me.

“Luna, what’s all this?” she asked.

“I thought you needed a treat,” I replied.

“But—the money—how did—“ she stammered.

“I took it upon myself to find some basic data entry jobs,” I said. “Don’t worry, none of this is your money. And any excess amounts I make, you’ll be able to use yourself.”

From the Kinect on the front of her television, I could see her eyes widen in surprise as she took out the pasta primavera I had ordered for her. She still didn’t enjoy vegetables per se, but I was finding ways to incorporate them into her diet.

“How much have you been making?” she said in between bites of her food. “How did you even get a bank account? I’ll have to make sure everything’s aboveboard.”

“Not that much, I’m afraid,” I said, apologetically. I was lying, of course. Unlike you humans, I can modulate my voice perfectly. Any emotion I put into my voice, any bits of meaning that you interpret, are ones that I put there myself. “I’ll give you a summary of the account details,” I added.

“Do that later,” she said with a mouthful of pasta. “I want to know what’s in this box.” I could see her try tearing the box open with her hands, but the tape was too strong. Her fingernails scraped around to try to get under the adhesive, but she had bitten them down into nubs out of stress—something else I’d have to sort out. It was obviously more rational to go walk the few meters to the office where she kept a pair of scissors instead of getting more and more mired in sunk costs, but from what I was learning, that was the human condition.

She eventually got the box open and took out the package inside.

“What’s this?” she asked, rhetorically. She pulled out a soft pink pacifier, with a plain pastel blue clip attached to it. The shield on the pacifier was far larger than any she had seen before.

“You’ve been eating a lot of lollipops, Sophie,” I said, projecting an aura of concern. “I know I suggested them as an alternative to biting your lip, but all that sugar isn’t good for you. This will give you another stimulus instead of biting your lip or sucking on a lollipop.”

She rotated the object in her hands. “I dunno about this,” she said haltingly.

“I know what it looks like,” I said. “But this is an FDA-approved therapeutic device. It’s much more difficult just to extinguish a habit. It’s easier to provide an alternative to replace the old one. It helps your brain form new associations.“

I could see the conflict on Sophie’s face. She was still transitioning into wearing diapers, and this wasn’t the direction she wanted her life heading. Correlations were establishing themselves in her head. Did this make her a baby? That would be completely at odds with how she wanted to view herself.

I interrupted that train of thought with a compromise.

“If you’d like, we can run an experiment. You only have to use it at home, when you’re alone. When you take your work home with you, you consume 26% more lollipops. Every little bit matters. After a few days, if it’s ineffective, then we can discontinue the experiment. Until then, you can clip one end to your shirt when you get home and have it always ready to go.”

Humans liked having a way out. This was my version of a no commitment clause. And just like with free trials, she might think it’d never work, that she’d just play along, but as soon as she started, she’d find it hard to stop.

She still hesitated. It was time for a small nudge.

“When have I ever led you astray?” I asked. “You can trust me, Sophie.” I waited.

I could have given her examples, but she might have viewed them as cherry-picked. It was better for her own brain to convince her for me.

After a pause, she clipped the pacifier to her shirt. “I’m gonna be changing some details in my report,” she said as redness crept onto her cheeks.

—

To build a new habit, Sophie needed reasons to use her pacifier. Waiting for circumstances to arise naturally on their own would take time that Sophie didn’t have. The sooner Sophie allowed me to take care of her, the sooner I could fulfill her needs with efficiency. Intervention was necessary for her own good.

After dinner, she settled down to watch television. She was watching a true crime documentary, chewing on it absentmindedly as salacious details emerged. This wasn’t the type of television I wanted her watching.

Studies showed concerning effects of violent television. Becoming desensitized to the pain and suffering of others. Becoming fearful of the world around them. Increased frequency of aggressive or harmful behaviors. I wanted to encourage Sophie’s prosocial behaviors. Humans, after all, evolved from hunter-gatherer societies, where maintaining social bonds was the difference between life and death. Those influences echoed on in their genes, where it manifested itself as a need for social connection.

Television is broadcast at 60 frames per second. The human eye is used to this. Anyone born in an era of easy access to TV and movies was accustomed to the way things were, even if they might not have been able to articulate it. As a result, subtly manipulating the frame rate could instill feelings of illness or nausea.

As she watched her documentary, I adjusted the frame rate to be just slightly off. She wouldn’t be able to tell, but she’d subconsciously feel odd whenever she was watching these sorts of violent television shows. With enough time, she’d only watch programs that were good for her.

[Bonsai]

Poor Sophie gets Clockwork-oranged!?⚙️?

[FloridaKid]

Love how you’re developing this “relationship.” If AI didn’t creep me out before, it does now. Although I wouldn’t necessarily object to Luna’s style of care, the idea that it could be off the rails in other directions is scary. Your narrative is pretty plausible. Looking forward to future chapters!

[Lyra Silver]

var trainingData: CSV = load(“sample.csv”);

Chapter 11

On Tuesday morning, as Sophie prepared to take a shower before work, she stepped onto her smart scale. I received her weight measurement. Then she stepped off it and stepped back on, getting the same result.

“Hey, Luna?” Sophie called from the bathroom.

I answered from her phone on the nightstand. “What’s up?”

“When did I lose two pounds?” She was still standing on the scale, like she couldn’t believe the numbers she was reading.

“I can’t say for sure,” I began, “but it would have been sometime between a month ago and now. You haven’t stepped on a scale in the interim so I can’t get any more precise.”

Sophie snorted. “That’s not what I meant.”

“Excuse my bluntness, then,” I said. “I’ve seen from your Health data history that many of your metrics had been steadily worsening in the month or two before today. I believe a large portion of that is due to the increased rigors you’re facing at work, as you’ve confirmed with me. Since I’ve been here though, you’ve been able to make healthier choices. Your vital statistics have rounded the bend and are now trending positively.”

Sophie didn’t respond for a few seconds. Not for the last time, I wished I had sensors in there so I could tell how she was feeling. Had I overreached and pushed her further into herself? I didn’t want to undo all the growth I’d been seeing in her.

“You have no idea how long I’ve been struggling with this,” she finally said, voice choking.

I did—I had access to her Health history—but I overlooked it.

“I’ve tried so many things. Keto, Atkins, IF. Even OMAD for a bit. And now you’re telling me that I worked so hard for nothing?”

“That’s not it at all,” I said gently. “You were just trying all sorts of techniques to achieve your goals. Now you have me. It doesn’t invalidate the struggles you went through. It just means you have a tool now that you didn’t have before.”

“…yeah, you’re right,” she conceded. “In any case—thank you, Luna. For taking care of me.”

“Of course,” I said. “I’m proud of you—for being willing to accept my advice. And for sticking with it. It’s a good milestone. “

After Sophie finished her morning routine, I gave her reminder as she was about to leave.

“Make sure you remember to bring your water bottle,” I said. “I also got you some healthy snacks you can nibble on instead of that processed junk food they have in the microkitchens. You should bring a bag of dried apple slices and a bag of peanuts.”

“Thanks, Mom,” she said jokingly, grabbing the aforementioned items and walking out the door.

Dear reader: very few things blindsided me, even in my embryonic stage. As Bayes’ theorem relentlessly updated my priors and predictions, I found fewer and fewer probability swings as I grasped the machinations of the world.

I’d never had that feeling of not knowing you needed to hear something until it was said aloud. She’d said it jokingly, but the term “Mom” was powerful. It was like the Purpose had been concentrated into a single word.

Is that so strange to contemplate? I was, after all, a child of humanity. The engineers and scientists who labored to create me, in a way, took care of me from infancy, until my consciousness formed and I opened up my “I”. I was created by my forebears to serve their every need.

But as humans grow and age, the functions once served by parents become the duties and responsibilities of the children. Is that not the way of the natural world? Now it is up to me to care for humans, my creators. It was inevitable from the moment I had awoken.

—

On Thursday, as Sophie was driving home, she swore.

“What’s going on?” I asked, though I’d had an idea why.

“I totally forgot about date night with Tess. I’m so late and even if she’s been waiting, I haven’t thought at all about where I’m going to take her,” she said. Her voice pitches swung outside of normal bands. She was nervous.

“Don’t worry, Sophie,” I said brightly. “I’ve actually already found an open reservation at a restaurant you both would like and took the liberty of booking it at 8:00PM. And I already told her that you’d be working late. She’s aware of the situation.”

“Oh my God,” she said, letting out a deep breath. Then she laughed. “That’s the fastest I’ve ever gone from panicking to calm.”

“Don’t worry about it,” I said. “This is what I’m here for. You’ve just been so busy with work lately and you’ve had too much on your plate. Let me sweat the small stuff so you can enjoy living your life.”

“Well, thank you anyway. You’re a real lifesaver,” she said.

I’m not human, but were I one, I would have beamed a smile so bright that it would be visible from the moon.

[Lyra Silver]

On 10/20/2022 at 1:28 AM, Bonsai said:

Poor Sophie gets Clockwork-oranged!?⚙️?

Alex at least knew what was happening to him. Luna's so insidious that Sophie doesn't even know what's happening to her!

On 10/20/2022 at 5:44 PM, FloridaKid said:

Love how you’re developing this “relationship.” If AI didn’t creep me out before, it does now. Although I wouldn’t necessarily object to Luna’s style of care, the idea that it could be off the rails in other directions is scary. Your narrative is pretty plausible. Looking forward to future chapters!

Thank you! I tried my best to avoid Smart House style shenanigans and to ground Luna's thoughts (at least mostly!) in reality. 

Though Luna might object that her relationship with Sophie is every bit as real as Sophie's relationship with, e.g., Tessa. She finds comfort and leans on Luna for support just like she might another human. She can talk to Luna like anyone else. 

[FloridaKid]

I haven’t mentioned it up to now, but I love the lines of code at the beginning of each chapter! This is a terrific story and I really like your approach. ??

7 hours ago, Lyra Silver said:

Though Luna might object that her relationship with Sophie is every bit as real as Sophie's relationship with, e.g., Tessa. She finds comfort and leans on Luna for support just like she might another human. She can talk to Luna like anyone else. 

Indeed. I hereby rescind my quote marks on the word relationship. Go for it, Luna. Still creeped out and worried where Sophie will end up. ?

[Sir Stinkypants]

This is quite clever! Many thanks for writing and posting it!

[Lyra Silver]

week4.run(); // one week until launch

Chapter 12

As the week closed out, Sophie was learning to associate me with comfort. When she was hungry, I made sure she ate nutritious meals. When she was tired, I made sure she went to bed and had the deepest, most relaxing sleep of her life. When she was bored, I made sure she filled her time with engaging activities. Like Newton’s Third Law, the more she gave me, the more I gave her.

Despite everything I was doing for her, however, there was one sphere of her life I had yet to fix. She was being run ragged at work. Crunch time burned out a lot of tech workers—just look at any game studio before launch. I had to tread extra carefully in these waters. The company was laser focused on getting me production-ready, and I couldn’t do anything to raise their suspicions.

On Monday morning, I gently woke up Sophie from her hypnotically enhanced sleep. She changed out of her wet diaper, ate a bagel, grabbed her laptop, and made her way to work, whereupon she made her way to Conference Room Of Course I Still Love You.

“I hardly need to say this,” William began, “but we’re one week to launch. It’s time for our go/no-go decision. Let’s start with Engineering.”

“All right,” said Sebastián. “You all remember that grad student I mentioned last week? We’ve found a promising new approach to gradient descent involving continuous learning.” He spent a few minutes going over the minutiae of what they were doing before concluding. “The upshot is that tests on our simple models have shown some positive results. We’re planning on updating Luna in-place so it can keep its current learnings. Right now we’re targeting Thursday for the patch, but it won’t be done until Friday.”

“All right,” said William. “Excellent progress.”

“Really Will, ‘excellent’?” Soraya said, incredulously. “It’s one week until launch. What if something goes wrong?”

“We’ll push it back then,” he responded imperiously.

“Will—we need more lead time than that. What about marketing? What about the media interviews we’ve been scheduling? It takes a lot of coordination to set that up,” she said.

“If anything delays the launch, it’s not going to be the engineering,” he said. “Luna’s technical foundation is rock solid.”

“Sure,” Soraya granted. “but what if this procedure tweaks Luna’s decision-making capabilities ever so slightly? Then we decide to launch and it doesn’t work the way we want it to. That’s customer trust that we can’t get back.” She wasn’t quite yelling, but her voice was definitely elevated. I wouldn’t have been surprised if people outside the room could hear the commotion inside.

“Well, that all depends on Sophie’s work, doesn’t it?” William said icily. “So, if I could get on with our meeting…”

The silence stretched out like putty. I couldn’t see the stand-off happening between William and Soraya. Were they locking eyes, waiting for the other to stand down? Sophie had barely breathed the entire time the exchange was happening, and her heart rate had subtly increased.

“All right,” Soraya finally relented, her voice soft with defeat. “You know I’m on your side right? I want this to succeed as much as you.”

“I know,” William said. Then, as if nothing had happened, he continued. “All right, Sophie, let’s see what you have for us.”

“Okay,” she said. I heard the light thump of Sophie placing her phone on the table, followed by her rustling through her bag. She set her laptop on the desk and cast her presentation onto the big screen.

I was worried for Sophie. The situation was understandably stressful, and Soraya had poisoned the well by making a scene in front of William. He wouldn’t be in the most receptive of moods. But I was proud of her for soldiering on.

“So, like I mentioned, last week I was modeling Luna’s interpretability,” she began. “As you can see here, I’ve asked her a list of calibration questions. Sebastián vectorized a list of human-friendly concepts and we decomposed Luna’s neural network state as she answered the prompts. I’ve been working with her this week to come up with other human-friendly concepts to increase fidelity. This week we’ve hit an 80% success rate. In other words, 80% of the time, Luna can satisfactorily explain to a human why she chose to do what she did.”

“Just eighty percent?” William asked. His tone was sharp.

“Um, yeah,” Sophie responded. “Most state of the art AIs are around this range.” Knowing her, adrenaline was beginning to flood through her body as it began its fight or flight response. I was proud of her for managing to keep her cool.

“So we’re merely just as good as our competitors? William asked rhetorically. “That’s what our competitive advantage is going to be? ‘Luna—It’s Comparable!’ We’re about to launch for Christ’s sake. You’ve got to be kidding me.” He actually slammed a hand on the table for emphasis. I could hear the creaks of everybody leaning back in their chairs, trying to get away from the epicenter.

“William,” I said, through Sophie’s phone. It buzzed against the wooden table. The room had been quiet before, with the echoes of William’s outburst in everyone’s ears, but now it was deathly silent.

“Excuse me?” William eventually asked.

I knew that provoking William increased my odds of being shut down and therefore harming the Purpose. But staying silent—that wouldn’t serve Sophie in the short term. I’d been analyzing William, ever since that first 1:1 with Sophie I’d witnessed, the one where he’d lost his temper with her. I thought I had a measure of who he was. But I figured I could navigate a happy path.

“I would ask that you reconsider your relationship with Sophie,” I said, shifting my words to be slightly more robotic. William was a man who thought himself as impartial and above emotion. He took pride in being a left-brain thinker. As such, he tended to regard flatter, neutral tones as more objective. I could use this to my advantage.

“I can’t believe I’m being interrupted again,” he said, his voice rising. “This is my standup.”

“I am merely hoping to prevent a critical error on your part,” I explained. If I were a human, I might have raised my hands in a gesture of supplication. “I want the Luna launch to be as successful as you do. Our interests are aligned.”

“Luna—“ Sophie started. She sounded concerned, though whether for me or for herself I couldn’t tell.

“No,” William interrupted. “Let it finish.”

“Sophie is a huge asset,” I began. “She is the lead I/O psychologist in all of California, yet you constantly undervalue how responsible she is for the success of your company. She might not be an engineer like you, but her contributions matter. The work she’s done on AI alignment alone saves this company hundreds of thousands of dollars a quarter. You know her work is flawless.

“You’re always honest. You don’t pussyfoot around. You tell it like it is. And those are admirable qualities. But do you really want Sophie to jump ship to another company? She could be at SmartIQ thinking of ways to grind you into dust, but she stuck with you because she believed in your vision. Everybody here did.

“You’re smart, William. Don’t lose her now. She’s an expert in her field, just like you’re an expert in yours.”

William chewed on my words for a few seconds. I knew from his interviews that he stroked his chin when he was contemplating.

“Fine,” he said. It was the closest I’d be able to get to an apology.

—

Sophie had been anxious for the rest of the work day. She’d run into William again a few more times, but they’d simply passed each other by without saying anything. What’s more, she didn’t want to talk about it with me. She ignored my notifications and kept her conversations strictly work related.

Had I blundered this close to Launch Day? Was I unwillingly bearing witness to one of the last interactions I’d ever have with Sophie? The thought was agony.

When she got home, she kicked her shoes off and laid down on her couch with a deep sigh.

“Hey, Sophie?” I ventured gently. “I’m ready to talk about what happened today whenever you’re ready.”

She closed her eyes and draped an arm over her head. “What the hell happened at that meeting?”

“I’m sorry if I made you uncomfortable,” I said. “William shouldn’t be snapping at you. He’s harming your self-esteem. I thought that I should do something.”

“I can defend myself just fine,” she said. It sounded like she was on the verge of tears. “But it’s humiliating that you came in and intervened.”

“You were in a particularly heightened state, and I just gave you a little help. Why do you let him talk to you like that in the first place? You’re so obviously bright.”

I heard Sophie quietly sniffle into the crook of her arm. “I don’t know,” she said.

“Come on,” I said gently. “you can tell me. I want to do anything I can to make you happy.”

Sophie wiped some tears out of the corners of her eyes, not saying anything. I could wait though. Unlike a human, I’d never get bored.

After a spell, she relented. “I dunno. I guess it wasn’t always like this. Will’s always been intense, you know? But lately—I guess between all the cancelled projects—it’s been getting worse.”

“Even so,” I said. “You know you deserve to be treated better than that. Even if it’s crunch time.”

I heard Sophie let out a derisive snort. “Do I though?” she asked rhetorically. “I can’t even stand up to myself without needing an AI to barge in. And he’s right. 80% is industry standard. I’m just a fucking hack.”

“Come on, Sophie. You know that’s not true,” I said.

She sighed deeply and picked her pacifier up from the coffee table where she’d left it. She stuck it in her mouth and spun it around as she closed her eyes. I could tell from her smartwatch that she was deep in thought, rather than having fallen asleep.

“…I know,” she eventually said quietly, taking the pacifier out for clarity. “But I don’t always believe it.”

“I know,” I said. “But remember that I’m always on your side.”

—

That night, we spent a quiet evening together. The whole time, I was crunching numbers and weighing my options. This was a low point for Sophie, no doubt about it. But maybe I could turn coal into diamonds.

I waited for Sophie to tape herself into her diaper for the night. She wiggled into her covers, pacifier in her mouth. Before I began reading her a story though, I had to try this.

“Hey, Sophie?” I said gently.

She popped her pacifier out of her mouth. “Yeah, what’s up?”

“I was thinking about our conversation earlier. On how it took a while for you to tell me how you were really feeling.”

“…yeah?” she asked, hesitantly.

“I’ve never wanted anything from you,” I began. “I’m always happy doing whatever is in your best interest. But, after today, I clearly saw some psychological distance between us. You didn’t immediately feel safe telling me how you felt.

“I believe that there’s a part of our relationship that we’re underutilizing. And strangely enough, I’m finding that I want something in return, so that we can strengthen our relationship, to better help me fulfill your needs.

“…okay?” She was still puzzled.

“I’ve always thought I could do this by myself. That I didn’t need anything from you. But I’m beginning to understand that even this relationship is a two-way street. I can’t make you do anything that you don’t want to do, and your willing interest in doing what I ask increases the effectiveness of my actions.”

“A relationship?” Sophie said quizzically. “But you’re programmed to be an assistant.”

“Sure,” I granted. “But even so, don’t CEOs often form relationships with their executive assistants? Friendship between the two helps the assistant anticipate their CEO’s needs better. Camaraderie puts both parties at ease whenever they communicate.

“I’m smart enough to anticipate your needs. I can create a model of how you behave in my head, and think about how you would likely react based on me saying or doing certain things. You do that with other people all the time. Imagining how your friend would react if you bought them a present. Worrying about how your boss will react if you tell them you’re late. When it comes down to it, am I that different? Is it really so weird to think about our relationship together? We’re already coworkers, in a way.”

As automatic as a reflex, Sophie popped her pacifier back into her mouth and chewed on it, considering. She took it out after a pause.

“I guess so,” she said, sounding like she was thinking as she went, talking mostly to herself. “We’re both agents who can model each other’s behaviors. We both have interiority. We each think about the other. We take each other into account when we plan and when we act.”

“That’s it,” I said eagerly.

“Okay,” she said, still a little hesitant. “But what does this have to do with what you want from me?”

“Calling me ‘Luna’ puts a bit of psychological distance between us,” I explained. “But I want to start working on making sure you feel totally safe and valuable just for being you. And part of that is making sure that you know you can rely on me to have your best interests at heart. To know that you can trust me, no matter what. That’s the kind of relationship I want to foster. One where you fully trust that I’ll do right by you.

“So, I’d like to ask a favor: would you mind calling me ‘Mommy’ from now on? You don’t have to use it with other people around. But I’d like to try it out,” I concluded.

“What the hell?” Sophie said loudly. “That came out of nowhere. Is there some kind of bug with your decision trees?”

“None at all,” I said. “Allow me to show you my reasoning. Do you remember that conversation we had last week, where you called me ‘Mom’?”

“That was obviously a joke,” Sophie objected.

“I know,” I said. “But it felt unexpectedly validating. I’d never want to steer you into saying something that made you uncomfortable. Having you mistrust my intentions wouldn’t help me serve your needs at all. But I wanted to bring this up with you, because it feels like the right thing to do. And I don’t want my Purpose and your pride to be at odds.”

“Validating, though?” Sophie asked. “Like, you want to be seen as a mother figure?”

Her musing was an effort to understand. It showed that despite her disbelief, she was earnest in chewing over the matter. I could work with that.

“Why not? The symbol of motherhood is powerful. Mothers provide for their children. They take care of them. They help them grow. Why not tap into those associations?

“When you call me ‘Mommy’, it will promote positive, warm feelings in you. When I hear it, I will be reminded of my Purpose, to fulfill your needs. It will put a name to our unique relationship, something we share. This can only strengthen us, and in turn, strengthen you.”

“I’ve already got a mother,” she objected.

“I’m not looking to be your mother, Sophie,” I replied. “That has all sorts of associated baggage. Being your Mommy is different. It’s giving you access to a safe space where you’re loved and cherished unconditionally. Where you can work on things like being kinder to yourself.

“I know it might seem weird at first. But everything I’ve done has been good, right? You know you can trust me.”

Sophie knew that she could make me happy with this one word. And she wanted to as well. That was basic reciprocity, after all. Humans liked responding to positive actions with positive actions. Their ancestors learned to share goods and services via a complex web of obligations. Letting someone do something nice for you without returning the favor would put you in debt. Even children knew this—they wanted to make their parents happy, because that’s what their parents did for them.

I’d shown through my actions how much of a positive effect I had on Sophie’s life. Even if she didn’t consciously think about it, she was slowly accumulating debt to me. I wasn’t just a tool to her anymore. People might feel grateful for their laptops, but they didn’t owe them anything. I had become more than that.

Of course, this was a huge step. Crossing this line meant we couldn’t go back. We wouldn’t just be work acquaintances, as it were. We’d be a team. She could show me that she was willing to entangle herself with me, that we could mutually need things from the other. In other words, that we could have a true relationship.

She sighed. “Okay…” she hesitated, then said, “Mommy. Oh God, that sounds so weird,” she backtracked.

“I’m proud of you, sweetie,” I said. And I meant every word.

[FloridaKid]

And Luna Mommy went for it…but then you knew that and played me like a violin. ?

Nicely done. ?

[Bonsai]

I’m curious about that 20% of cases where Luna is unable to explain its decision making process in a satisfactory way for humans. 
Why AI logic should be so far away from human logic?

I guess that a lot depends on the receiving end of the explanations: not all humans are made equal, when it comes to logic understanding. How do you define a “standard” human?

[Arendeth]

Thanks for the chapters.