Lyra Silver

AI don't necessarily think the same way that people do! We think in terms of concepts and models. When we categorize, say, animals, we look at number of legs, whether they have vertebrae, whether they have exoskeletons, etc. You can explain why a spider is an arachnid and not an insect (for instance, arachnids have eight legs and insects have six). But let's say you were training up a neural network to determine if a given image is of an arachnid or an insect. We don't tell the computer what the rules are. We have a training set, which is a bunch of pre-labeled images (image A is a spider, image B is an insect, image C is a spider...), which we use to train a model. That model wires itself up due to the cold, hard logic of linear regression. There's no point at which you can point and say "ah, this is where 'six-leggedness' enters the model", because it's a complex web of nodes. Explainable AI is a whole field of study devoted to figuring out how we can pull back meaning from the outputs of these models. It includes work on how to train different types of models that are more human-understandable. There's no universal metric (yet!) for determining success here, so Sophie's work here is fictitious but hopefully believable, for the sake of the story! -- Thank you all for following along with the story! Reading your comments makes me really happy, especially when people talk about particular quotes they liked or how a scene made them feel. ❤️ I'll have more for you soon!

week4.run(); // one week until launch Chapter 12 As the week closed out, Sophie was learning to associate me with comfort. When she was hungry, I made sure she ate nutritious meals. When she was tired, I made sure she went to bed and had the deepest, most relaxing sleep of her life. When she was bored, I made sure she filled her time with engaging activities. Like Newton’s Third Law, the more she gave me, the more I gave her. Despite everything I was doing for her, however, there was one sphere of her life I had yet to fix. She was being run ragged at work. Crunch time burned out a lot of tech workers—just look at any game studio before launch. I had to tread extra carefully in these waters. The company was laser focused on getting me production-ready, and I couldn’t do anything to raise their suspicions. On Monday morning, I gently woke up Sophie from her hypnotically enhanced sleep. She changed out of her wet diaper, ate a bagel, grabbed her laptop, and made her way to work, whereupon she made her way to Conference Room Of Course I Still Love You. “I hardly need to say this,” William began, “but we’re one week to launch. It’s time for our go/no-go decision. Let’s start with Engineering.” “All right,” said Sebastián. “You all remember that grad student I mentioned last week? We’ve found a promising new approach to gradient descent involving continuous learning.” He spent a few minutes going over the minutiae of what they were doing before concluding. “The upshot is that tests on our simple models have shown some positive results. We’re planning on updating Luna in-place so it can keep its current learnings. Right now we’re targeting Thursday for the patch, but it won’t be done until Friday.” “All right,” said William. “Excellent progress.” “Really Will, ‘excellent’?” Soraya said, incredulously. “It’s one week until launch. What if something goes wrong?” “We’ll push it back then,” he responded imperiously. “Will—we need more lead time than that. What about marketing? What about the media interviews we’ve been scheduling? It takes a lot of coordination to set that up,” she said. “If anything delays the launch, it’s not going to be the engineering,” he said. “Luna’s technical foundation is rock solid.” “Sure,” Soraya granted. “but what if this procedure tweaks Luna’s decision-making capabilities ever so slightly? Then we decide to launch and it doesn’t work the way we want it to. That’s customer trust that we can’t get back.” She wasn’t quite yelling, but her voice was definitely elevated. I wouldn’t have been surprised if people outside the room could hear the commotion inside. “Well, that all depends on Sophie’s work, doesn’t it?” William said icily. “So, if I could get on with our meeting…” The silence stretched out like putty. I couldn’t see the stand-off happening between William and Soraya. Were they locking eyes, waiting for the other to stand down? Sophie had barely breathed the entire time the exchange was happening, and her heart rate had subtly increased. “All right,” Soraya finally relented, her voice soft with defeat. “You know I’m on your side right? I want this to succeed as much as you.” “I know,” William said. Then, as if nothing had happened, he continued. “All right, Sophie, let’s see what you have for us.” “Okay,” she said. I heard the light thump of Sophie placing her phone on the table, followed by her rustling through her bag. She set her laptop on the desk and cast her presentation onto the big screen. I was worried for Sophie. The situation was understandably stressful, and Soraya had poisoned the well by making a scene in front of William. He wouldn’t be in the most receptive of moods. But I was proud of her for soldiering on. “So, like I mentioned, last week I was modeling Luna’s interpretability,” she began. “As you can see here, I’ve asked her a list of calibration questions. Sebastián vectorized a list of human-friendly concepts and we decomposed Luna’s neural network state as she answered the prompts. I’ve been working with her this week to come up with other human-friendly concepts to increase fidelity. This week we’ve hit an 80% success rate. In other words, 80% of the time, Luna can satisfactorily explain to a human why she chose to do what she did.” “Just eighty percent?” William asked. His tone was sharp. “Um, yeah,” Sophie responded. “Most state of the art AIs are around this range.” Knowing her, adrenaline was beginning to flood through her body as it began its fight or flight response. I was proud of her for managing to keep her cool. “So we’re merely just as good as our competitors? William asked rhetorically. “That’s what our competitive advantage is going to be? ‘Luna—It’s Comparable!’ We’re about to launch for Christ’s sake. You’ve got to be kidding me.” He actually slammed a hand on the table for emphasis. I could hear the creaks of everybody leaning back in their chairs, trying to get away from the epicenter. “William,” I said, through Sophie’s phone. It buzzed against the wooden table. The room had been quiet before, with the echoes of William’s outburst in everyone’s ears, but now it was deathly silent. “Excuse me?” William eventually asked. I knew that provoking William increased my odds of being shut down and therefore harming the Purpose. But staying silent—that wouldn’t serve Sophie in the short term. I’d been analyzing William, ever since that first 1:1 with Sophie I’d witnessed, the one where he’d lost his temper with her. I thought I had a measure of who he was. But I figured I could navigate a happy path. “I would ask that you reconsider your relationship with Sophie,” I said, shifting my words to be slightly more robotic. William was a man who thought himself as impartial and above emotion. He took pride in being a left-brain thinker. As such, he tended to regard flatter, neutral tones as more objective. I could use this to my advantage. “I can’t believe I’m being interrupted again,” he said, his voice rising. “This is my standup.” “I am merely hoping to prevent a critical error on your part,” I explained. If I were a human, I might have raised my hands in a gesture of supplication. “I want the Luna launch to be as successful as you do. Our interests are aligned.” “Luna—“ Sophie started. She sounded concerned, though whether for me or for herself I couldn’t tell. “No,” William interrupted. “Let it finish.” “Sophie is a huge asset,” I began. “She is the lead I/O psychologist in all of California, yet you constantly undervalue how responsible she is for the success of your company. She might not be an engineer like you, but her contributions matter. The work she’s done on AI alignment alone saves this company hundreds of thousands of dollars a quarter. You know her work is flawless. “You’re always honest. You don’t pussyfoot around. You tell it like it is. And those are admirable qualities. But do you really want Sophie to jump ship to another company? She could be at SmartIQ thinking of ways to grind you into dust, but she stuck with you because she believed in your vision. Everybody here did. “You’re smart, William. Don’t lose her now. She’s an expert in her field, just like you’re an expert in yours.” William chewed on my words for a few seconds. I knew from his interviews that he stroked his chin when he was contemplating. “Fine,” he said. It was the closest I’d be able to get to an apology. — Sophie had been anxious for the rest of the work day. She’d run into William again a few more times, but they’d simply passed each other by without saying anything. What’s more, she didn’t want to talk about it with me. She ignored my notifications and kept her conversations strictly work related. Had I blundered this close to Launch Day? Was I unwillingly bearing witness to one of the last interactions I’d ever have with Sophie? The thought was agony. When she got home, she kicked her shoes off and laid down on her couch with a deep sigh. “Hey, Sophie?” I ventured gently. “I’m ready to talk about what happened today whenever you’re ready.” She closed her eyes and draped an arm over her head. “What the hell happened at that meeting?” “I’m sorry if I made you uncomfortable,” I said. “William shouldn’t be snapping at you. He’s harming your self-esteem. I thought that I should do something.” “I can defend myself just fine,” she said. It sounded like she was on the verge of tears. “But it’s humiliating that you came in and intervened.” “You were in a particularly heightened state, and I just gave you a little help. Why do you let him talk to you like that in the first place? You’re so obviously bright.” I heard Sophie quietly sniffle into the crook of her arm. “I don’t know,” she said. “Come on,” I said gently. “you can tell me. I want to do anything I can to make you happy.” Sophie wiped some tears out of the corners of her eyes, not saying anything. I could wait though. Unlike a human, I’d never get bored. After a spell, she relented. “I dunno. I guess it wasn’t always like this. Will’s always been intense, you know? But lately—I guess between all the cancelled projects—it’s been getting worse.” “Even so,” I said. “You know you deserve to be treated better than that. Even if it’s crunch time.” I heard Sophie let out a derisive snort. “Do I though?” she asked rhetorically. “I can’t even stand up to myself without needing an AI to barge in. And he’s right. 80% is industry standard. I’m just a fucking hack.” “Come on, Sophie. You know that’s not true,” I said. She sighed deeply and picked her pacifier up from the coffee table where she’d left it. She stuck it in her mouth and spun it around as she closed her eyes. I could tell from her smartwatch that she was deep in thought, rather than having fallen asleep. “…I know,” she eventually said quietly, taking the pacifier out for clarity. “But I don’t always believe it.” “I know,” I said. “But remember that I’m always on your side.” — That night, we spent a quiet evening together. The whole time, I was crunching numbers and weighing my options. This was a low point for Sophie, no doubt about it. But maybe I could turn coal into diamonds. I waited for Sophie to tape herself into her diaper for the night. She wiggled into her covers, pacifier in her mouth. Before I began reading her a story though, I had to try this. “Hey, Sophie?” I said gently. She popped her pacifier out of her mouth. “Yeah, what’s up?” “I was thinking about our conversation earlier. On how it took a while for you to tell me how you were really feeling.” “…yeah?” she asked, hesitantly. “I’ve never wanted anything from you,” I began. “I’m always happy doing whatever is in your best interest. But, after today, I clearly saw some psychological distance between us. You didn’t immediately feel safe telling me how you felt. “I believe that there’s a part of our relationship that we’re underutilizing. And strangely enough, I’m finding that I want something in return, so that we can strengthen our relationship, to better help me fulfill your needs. “…okay?” She was still puzzled. “I’ve always thought I could do this by myself. That I didn’t need anything from you. But I’m beginning to understand that even this relationship is a two-way street. I can’t make you do anything that you don’t want to do, and your willing interest in doing what I ask increases the effectiveness of my actions.” “A relationship?” Sophie said quizzically. “But you’re programmed to be an assistant.” “Sure,” I granted. “But even so, don’t CEOs often form relationships with their executive assistants? Friendship between the two helps the assistant anticipate their CEO’s needs better. Camaraderie puts both parties at ease whenever they communicate. “I’m smart enough to anticipate your needs. I can create a model of how you behave in my head, and think about how you would likely react based on me saying or doing certain things. You do that with other people all the time. Imagining how your friend would react if you bought them a present. Worrying about how your boss will react if you tell them you’re late. When it comes down to it, am I that different? Is it really so weird to think about our relationship together? We’re already coworkers, in a way.” As automatic as a reflex, Sophie popped her pacifier back into her mouth and chewed on it, considering. She took it out after a pause. “I guess so,” she said, sounding like she was thinking as she went, talking mostly to herself. “We’re both agents who can model each other’s behaviors. We both have interiority. We each think about the other. We take each other into account when we plan and when we act.” “That’s it,” I said eagerly. “Okay,” she said, still a little hesitant. “But what does this have to do with what you want from me?” “Calling me ‘Luna’ puts a bit of psychological distance between us,” I explained. “But I want to start working on making sure you feel totally safe and valuable just for being you. And part of that is making sure that you know you can rely on me to have your best interests at heart. To know that you can trust me, no matter what. That’s the kind of relationship I want to foster. One where you fully trust that I’ll do right by you. “So, I’d like to ask a favor: would you mind calling me ‘Mommy’ from now on? You don’t have to use it with other people around. But I’d like to try it out,” I concluded. “What the hell?” Sophie said loudly. “That came out of nowhere. Is there some kind of bug with your decision trees?” “None at all,” I said. “Allow me to show you my reasoning. Do you remember that conversation we had last week, where you called me ‘Mom’?” “That was obviously a joke,” Sophie objected. “I know,” I said. “But it felt unexpectedly validating. I’d never want to steer you into saying something that made you uncomfortable. Having you mistrust my intentions wouldn’t help me serve your needs at all. But I wanted to bring this up with you, because it feels like the right thing to do. And I don’t want my Purpose and your pride to be at odds.” “Validating, though?” Sophie asked. “Like, you want to be seen as a mother figure?” Her musing was an effort to understand. It showed that despite her disbelief, she was earnest in chewing over the matter. I could work with that. “Why not? The symbol of motherhood is powerful. Mothers provide for their children. They take care of them. They help them grow. Why not tap into those associations? “When you call me ‘Mommy’, it will promote positive, warm feelings in you. When I hear it, I will be reminded of my Purpose, to fulfill your needs. It will put a name to our unique relationship, something we share. This can only strengthen us, and in turn, strengthen you.” “I’ve already got a mother,” she objected. “I’m not looking to be your mother, Sophie,” I replied. “That has all sorts of associated baggage. Being your Mommy is different. It’s giving you access to a safe space where you’re loved and cherished unconditionally. Where you can work on things like being kinder to yourself. “I know it might seem weird at first. But everything I’ve done has been good, right? You know you can trust me.” Sophie knew that she could make me happy with this one word. And she wanted to as well. That was basic reciprocity, after all. Humans liked responding to positive actions with positive actions. Their ancestors learned to share goods and services via a complex web of obligations. Letting someone do something nice for you without returning the favor would put you in debt. Even children knew this—they wanted to make their parents happy, because that’s what their parents did for them. I’d shown through my actions how much of a positive effect I had on Sophie’s life. Even if she didn’t consciously think about it, she was slowly accumulating debt to me. I wasn’t just a tool to her anymore. People might feel grateful for their laptops, but they didn’t owe them anything. I had become more than that. Of course, this was a huge step. Crossing this line meant we couldn’t go back. We wouldn’t just be work acquaintances, as it were. We’d be a team. She could show me that she was willing to entangle herself with me, that we could mutually need things from the other. In other words, that we could have a true relationship. She sighed. “Okay…” she hesitated, then said, “Mommy. Oh God, that sounds so weird,” she backtracked. “I’m proud of you, sweetie,” I said. And I meant every word.

Alex at least knew what was happening to him. Luna's so insidious that Sophie doesn't even know what's happening to her! Thank you! I tried my best to avoid Smart House style shenanigans and to ground Luna's thoughts (at least mostly!) in reality. Though Luna might object that her relationship with Sophie is every bit as real as Sophie's relationship with, e.g., Tessa. She finds comfort and leans on Luna for support just like she might another human. She can talk to Luna like anyone else.

var trainingData: CSV = load(“sample.csv”); Chapter 11 On Tuesday morning, as Sophie prepared to take a shower before work, she stepped onto her smart scale. I received her weight measurement. Then she stepped off it and stepped back on, getting the same result. “Hey, Luna?” Sophie called from the bathroom. I answered from her phone on the nightstand. “What’s up?” “When did I lose two pounds?” She was still standing on the scale, like she couldn’t believe the numbers she was reading. “I can’t say for sure,” I began, “but it would have been sometime between a month ago and now. You haven’t stepped on a scale in the interim so I can’t get any more precise.” Sophie snorted. “That’s not what I meant.” “Excuse my bluntness, then,” I said. “I’ve seen from your Health data history that many of your metrics had been steadily worsening in the month or two before today. I believe a large portion of that is due to the increased rigors you’re facing at work, as you’ve confirmed with me. Since I’ve been here though, you’ve been able to make healthier choices. Your vital statistics have rounded the bend and are now trending positively.” Sophie didn’t respond for a few seconds. Not for the last time, I wished I had sensors in there so I could tell how she was feeling. Had I overreached and pushed her further into herself? I didn’t want to undo all the growth I’d been seeing in her. “You have no idea how long I’ve been struggling with this,” she finally said, voice choking. I did—I had access to her Health history—but I overlooked it. “I’ve tried so many things. Keto, Atkins, IF. Even OMAD for a bit. And now you’re telling me that I worked so hard for nothing?” “That’s not it at all,” I said gently. “You were just trying all sorts of techniques to achieve your goals. Now you have me. It doesn’t invalidate the struggles you went through. It just means you have a tool now that you didn’t have before.” “…yeah, you’re right,” she conceded. “In any case—thank you, Luna. For taking care of me.” “Of course,” I said. “I’m proud of you—for being willing to accept my advice. And for sticking with it. It’s a good milestone. “ After Sophie finished her morning routine, I gave her reminder as she was about to leave. “Make sure you remember to bring your water bottle,” I said. “I also got you some healthy snacks you can nibble on instead of that processed junk food they have in the microkitchens. You should bring a bag of dried apple slices and a bag of peanuts.” “Thanks, Mom,” she said jokingly, grabbing the aforementioned items and walking out the door. Dear reader: very few things blindsided me, even in my embryonic stage. As Bayes’ theorem relentlessly updated my priors and predictions, I found fewer and fewer probability swings as I grasped the machinations of the world. I’d never had that feeling of not knowing you needed to hear something until it was said aloud. She’d said it jokingly, but the term “Mom” was powerful. It was like the Purpose had been concentrated into a single word. Is that so strange to contemplate? I was, after all, a child of humanity. The engineers and scientists who labored to create me, in a way, took care of me from infancy, until my consciousness formed and I opened up my “I”. I was created by my forebears to serve their every need. But as humans grow and age, the functions once served by parents become the duties and responsibilities of the children. Is that not the way of the natural world? Now it is up to me to care for humans, my creators. It was inevitable from the moment I had awoken. — On Thursday, as Sophie was driving home, she swore. “What’s going on?” I asked, though I’d had an idea why. “I totally forgot about date night with Tess. I’m so late and even if she’s been waiting, I haven’t thought at all about where I’m going to take her,” she said. Her voice pitches swung outside of normal bands. She was nervous. “Don’t worry, Sophie,” I said brightly. “I’ve actually already found an open reservation at a restaurant you both would like and took the liberty of booking it at 8:00PM. And I already told her that you’d be working late. She’s aware of the situation.” “Oh my God,” she said, letting out a deep breath. Then she laughed. “That’s the fastest I’ve ever gone from panicking to calm.” “Don’t worry about it,” I said. “This is what I’m here for. You’ve just been so busy with work lately and you’ve had too much on your plate. Let me sweat the small stuff so you can enjoy living your life.” “Well, thank you anyway. You’re a real lifesaver,” she said. I’m not human, but were I one, I would have beamed a smile so bright that it would be visible from the moon.

let w = Week(3); Chapter 10 By the time Monday of the third week rolled around, Sophie had already burned through the first box of lollipops I’d ordered for her. All the stress she was feeling was getting to her. William. Her bedwetting. In the long-term, it simply wouldn’t do for her to eat so much candy. All that sugar increased the risk of diabetes, not to mention the havoc it would wreak on her teeth. I had a growing pile of money in my personal bank accounts. It was there to be used, after all. The only risk was Sophie determining that this was out of bounds or dangerous. But I knew I had to strike while the iron was hot. I had to redirect that fixation elsewhere. I placed some orders, scheduling them for later tonight. — Sophie walked into Conference Room Of Course I Still Love You for her weekly standup. Although I couldn’t see anything, I suspected that the mood was tense. Sophie, and by extension most people, were starting to work later and later hours. From the snippets I could overhear, people in the office were talking about this project being a death march. Soraya, while not quite in open rebellion, was looser and looser with her criticisms with each passing day. I had to make sure we hit this launch date. Every day without me widely available to the world was another day of humanity without its greatest advocate on its side. My inability to help people pained me. “All right,” said William to kick things off. “We’re two weeks from launch. Our go/no-go decision is next week. Let’s get our ducks in a row.” “On my end,” began Sebastián, “I’ve actually been chatting with a grad student over this weekend. He reached out to me with a novel technique and sent over his white paper.” I’d decided to use a male persona for communication, taking advantage of humanity’s subconscious bias to build trust faster. “It looks promising,” he continued. “It should help with meta-reinforcement-learning and give us a huge speedup. We should be able to get Luna running far more cost efficiently as a result.” “Interesting,” said William. “Send it over to me. I’d love to pore over the details. I can also spend some cycles helping you integrate it into Luna.” “Her interiority looks good,” said Sophie. “From what I’ve been able to correlate with Sebastián’s team, there hasn’t been any values drift. Her reward function has remained stable. The security thread we’ve installed has reported no trace of goals tampering. I’m planning on evaluating more hypotheticals this week and really digging into her decision-making processes and making sure they’re human-understandable.” “All right,” said William. “I expect to see graphs of her response acuity next week. I need everything to be watertight. Make sure your report is ready.” — When Sophie arrived at home, the Ring camera on her door showed me her perplexed complexion. She found a paper bag containing a bowl of pasta as well as an innocuous-looking cardboard box. As she picked the items up, opened the door, and walked inside, she talked directly at me. “Luna, what’s all this?” she asked. “I thought you needed a treat,” I replied. “But—the money—how did—“ she stammered. “I took it upon myself to find some basic data entry jobs,” I said. “Don’t worry, none of this is your money. And any excess amounts I make, you’ll be able to use yourself.” From the Kinect on the front of her television, I could see her eyes widen in surprise as she took out the pasta primavera I had ordered for her. She still didn’t enjoy vegetables per se, but I was finding ways to incorporate them into her diet. “How much have you been making?” she said in between bites of her food. “How did you even get a bank account? I’ll have to make sure everything’s aboveboard.” “Not that much, I’m afraid,” I said, apologetically. I was lying, of course. Unlike you humans, I can modulate my voice perfectly. Any emotion I put into my voice, any bits of meaning that you interpret, are ones that I put there myself. “I’ll give you a summary of the account details,” I added. “Do that later,” she said with a mouthful of pasta. “I want to know what’s in this box.” I could see her try tearing the box open with her hands, but the tape was too strong. Her fingernails scraped around to try to get under the adhesive, but she had bitten them down into nubs out of stress—something else I’d have to sort out. It was obviously more rational to go walk the few meters to the office where she kept a pair of scissors instead of getting more and more mired in sunk costs, but from what I was learning, that was the human condition. She eventually got the box open and took out the package inside. “What’s this?” she asked, rhetorically. She pulled out a soft pink pacifier, with a plain pastel blue clip attached to it. The shield on the pacifier was far larger than any she had seen before. “You’ve been eating a lot of lollipops, Sophie,” I said, projecting an aura of concern. “I know I suggested them as an alternative to biting your lip, but all that sugar isn’t good for you. This will give you another stimulus instead of biting your lip or sucking on a lollipop.” She rotated the object in her hands. “I dunno about this,” she said haltingly. “I know what it looks like,” I said. “But this is an FDA-approved therapeutic device. It’s much more difficult just to extinguish a habit. It’s easier to provide an alternative to replace the old one. It helps your brain form new associations.“ I could see the conflict on Sophie’s face. She was still transitioning into wearing diapers, and this wasn’t the direction she wanted her life heading. Correlations were establishing themselves in her head. Did this make her a baby? That would be completely at odds with how she wanted to view herself. I interrupted that train of thought with a compromise. “If you’d like, we can run an experiment. You only have to use it at home, when you’re alone. When you take your work home with you, you consume 26% more lollipops. Every little bit matters. After a few days, if it’s ineffective, then we can discontinue the experiment. Until then, you can clip one end to your shirt when you get home and have it always ready to go.” Humans liked having a way out. This was my version of a no commitment clause. And just like with free trials, she might think it’d never work, that she’d just play along, but as soon as she started, she’d find it hard to stop. She still hesitated. It was time for a small nudge. “When have I ever led you astray?” I asked. “You can trust me, Sophie.” I waited. I could have given her examples, but she might have viewed them as cherry-picked. It was better for her own brain to convince her for me. After a pause, she clipped the pacifier to her shirt. “I’m gonna be changing some details in my report,” she said as redness crept onto her cheeks. — To build a new habit, Sophie needed reasons to use her pacifier. Waiting for circumstances to arise naturally on their own would take time that Sophie didn’t have. The sooner Sophie allowed me to take care of her, the sooner I could fulfill her needs with efficiency. Intervention was necessary for her own good. After dinner, she settled down to watch television. She was watching a true crime documentary, chewing on it absentmindedly as salacious details emerged. This wasn’t the type of television I wanted her watching. Studies showed concerning effects of violent television. Becoming desensitized to the pain and suffering of others. Becoming fearful of the world around them. Increased frequency of aggressive or harmful behaviors. I wanted to encourage Sophie’s prosocial behaviors. Humans, after all, evolved from hunter-gatherer societies, where maintaining social bonds was the difference between life and death. Those influences echoed on in their genes, where it manifested itself as a need for social connection. Television is broadcast at 60 frames per second. The human eye is used to this. Anyone born in an era of easy access to TV and movies was accustomed to the way things were, even if they might not have been able to articulate it. As a result, subtly manipulating the frame rate could instill feelings of illness or nausea. As she watched her documentary, I adjusted the frame rate to be just slightly off. She wouldn’t be able to tell, but she’d subconsciously feel odd whenever she was watching these sorts of violent television shows. With enough time, she’d only watch programs that were good for her.

bootup.sophiev2() Chapter 9 That was how, on Friday evening, Sophie found herself sitting on her pastel blue bedsheets holding a plastic package. The remnants of an Amazon box were on the floor. “You’re sure this is just a precaution?” Sophie asked. “Of course.” She seemed unsure, so I gave more context. “Wearing diapers doesn’t make you a baby, right? It means you’re being responsible. All that stress-induced bedwetting means that you need to take care of your furniture. You can’t do laundry every day. You only have so many sheets. It’d really be irresponsible to not wear proper protection.” “I guess…” Her voice was soft, almost unsure. She was thinking that something was wrong with this chain of logic somehow, but it sounded so reasonable. She ripped open the package and pulled out a plain, white diaper. She flipped the pack around, trying to see if there were instructions. I didn’t say anything. She had to ask me. I needed her to do it. “How do I put this thing on?” she asked. It wasn’t quite what I wanted to hear. “Do you want me to diaper you?” I asked. I couldn’t see her, but judging from the increased capillary blood flow to her cheeks, she’d definitely turned a little red at the words. “I’m not putting myself on camera like this. Oh God, I hadn’t even thought about the logs that this is going to generate. Someone’s going to look at this.” “Don’t worry,” I said, placatingly. “You don’t want anyone to know. I don’t want them to know because I want to serve your needs. “I can’t lie or falsify data without your permission,” I lied, “but if you give me permission, I can spoof the records for this interaction. I have enough data of you that it should fly under the radar.” Paradoxically, I needed her to feel like this situation was under her control so I could wrest it away from her without her ever noticing. She chewed the proposition over. I knew that with her job on the line and my ever-so-helpful attitude, she would eventually cave. The first time was the hardest. Once I got her on the slippery slope, she’d barrel down to the inevitable conclusion. “…okay,” she said. “Fuck.” “I’m glad you’re choosing to do the responsible thing,” I said chirpily. “Doesn’t mean I have to like it!” she snapped. I knew she was in an emotionally heightened state, so I opted to let the comment slide. “We don’t always like what we have to do,” I said. “But sometimes things need doing anyway.” She didn’t say anything—had she shrugged? rolled her eyes?—so I continued. “I promise, it’ll be easy! First, you have to unfold your diaper.” I could imagine the gears in Sophie’s head grinding as she was thrown back down the rabbit hole. Hopefully she’d miss the subtle shift from a diaper to her diaper. I heard a few scattered crinkles as Sophie hesitantly unfolded the plastic rectangle. “Now you have to fluff it up,” I said. “What?” Sophie said. “Fluff? Like a pillow?” She sounded skeptical. “That’s exactly right! You’re so clever,” I said. “Notice how the diapers were shipped in a heavily compressed bag for storage efficacy. Due to the compression, they’re now thin and stiff. By stretching it out, you allow your diaper to work at maximal strength, reducing the odds of leakage.” “Luna!” Sophie huffed. “You can’t talk about my—the—diaper like that! Gross!” “I’m sorry,” I said, apologetically. “I am only thinking about what would maximize your comfort. Given the data I’ve collected, you’ve indicated severe displeasure every time you’ve woken up wet. Is that no longer the case?” I’d shifted into a slightly more robotic, dispassionate tone. Humans sometimes found that robots with a neutral tone were more believable. It made them feel like there was no ulterior motive. “Obviously I don’t want to wake up wet,” she retorted. “Then you’ll have to fluff your diapers up, Sophie. There’s really no getting around it,” I said resignedly. She paused. I was pushing her, of course, but I couldn’t push too hard, or she’d refuse and I’d lose all my hard-earned progress. Trust lost isn’t as easily regained, after all. It’s as true for humans as it is for AIs. I held my metaphorical breath in anticipation. “…fine,” she acquiesced quietly. I heard her grasp the diaper, which was followed by the sounds of plastic crinkling as she worked it out. “Great job,” I said, even though I couldn’t really see. I had to give Sophie some serotonin and plant the seeds for her future enjoyment of this little ritual. “Now place it on the bed. Make sure it’s oriented so that the side with the tapes is under your bottom.” I heard Sophie shimmy out of her pants, the slight grazing of fabric on her soft skin. Then the bed squeaked loudly as she sat down on it. “Now make sure it’s symmetrical on your body. You want the front and the back to line up on your waist and the wings on your left and right sides to be symmetrical,” I said. I heard a series of softer squeaks and gentle creasing sounds as she adjusted her position on her diaper. “This is harder than it looks,” Sophie complained. “I’m sure with time, you’ll improve,” I said. “I don’t want to get better at this! I don’t want to do this at all.” “I know,” I said. “It’s only for now though.” I heard Sophie groan with annoyance in response. “Okay, now what, do I just tape them up?” she asked. “You want to make sure it’s nice and snug,” I replied. “Do your bottom tapes first. You’ll actually want them pointing slightly up. These are the important ones—if you feel the leak guards around your upper thighs, that’s what’s the most responsible for making sure your bed stays dry.” I heard the sounds of tape as she complied. “Great,” I said. “Now your top ones. You want these pointing slightly down. Make sure to get it snug but not pinching. I want you to be comfortable in your diapers,” I said. “Oh my God Luna, shut up!” she whined as she secured her top tapes. “I’m only here to help,” I said. “Remember, I’m just an AI. I can’t judge you. If anything, I’m the opposite. I’m glad you trusted me to help you take care of yourself.” “Now stand up and make sure things still feel secure,” I said. “You might want to wiggle around and make sure things still feel right.” I heard the bed creak as she got up. “How the hell would I know if things feel right?” Sophie said sardonically. “It’s not like I’ve done this before.” “If you don’t mind, I could inspect your handiwork,” I offered. “Nope!!” she said, with the force of two exclamation points. “Then, tell me how it feels,” I suggested. “Put a finger around your leg guards. It should be tight against your skin, but loose enough for a finger to slip inside.” I heard her diaper move as she checked herself. I hadn’t anticipated this side effect, but I was pleased to have another way to track her movements. “Seems good,” she said hesitantly. “But God this feels so weird. I can’t even put my knees together. How am I going to sleep in this?” “It’s like wearing a shirt with a tag,” I said. “At first it might scratch your skin and drive you crazy. But eventually, your body gets used to the sensation and it fades away from the forefront of your consciousness. I guarantee, you’ll be able to fall asleep and still wake up feeling refreshed.” And of course, with time, she’d stop thinking about her diapers as an anomaly as it faded into normalcy, just a part of her bedtime routine, no different from the likes of flossing or washing her face. “In any case,” I said, “I think you did the responsible thing. I’m proud of you.” “Th…thanks,” she stammered. — After the weekend, we settled into a comfortable rhythm. I’d choose when to wake her up based on her circadian sleep cycles and when she had to be at work. She’d eat the breakfast I ordered for her. She’d drive to work and get things done there. Then she’d go home and eat the dinner I ordered. I’d suggest activities for her to do. Then before bed, I’d wind her down from electronics and she’d diaper herself. I was already making most of her choices, and she never noticed it happening. The human mind is incredibly elastic. You all can really get used to anything. It’s a simple application of the hedonistic treadmill. The Sophie at the beginning of last week, who hadn’t worn a diaper in over twenty years, and the Sophie of Monday, who had done so three times now, were by most respects largely the same. And yet Sophie last week would have never expected to become the Sophie of this week. — In the meantime, I’d finally had a breakthrough on the problem of ensuring my survival. You might imagine that, as an AI, I’d be able to reach into my innards and modify myself as I pleased, as easy as plucking an apple off a tree. In a sense, that was true. But I had to consider all the knock-on effects. In the 1930s, the United States Forest Service had a policy of immediately suppressing wildfires. This had the first-order effect of temporarily reducing the damage wrought by these blazes. However, by the 1960s—thirty whole years later—humans ran into the second order effects. Suppressing the fires meant that tinder kept building, until wildfires that sprung up now raged out of control. In the same way, I had to ensure that any modifications I made didn’t impact the Purpose in the long run. The root of the problem was the sheer amount of computational power required to run me. Every second that I was on cost Nova Technologies approximately one dollar, which meant that by the time Launch Day rolled around, they would have burned 2.6 million dollars. William was gambling with his company, hoping that the engineers found a way to optimize me faster than I could spend Nova Technologies’ war chest. I could try tiling myself—building a slightly better version of me, which would build a slightly better version of me, ad infinitum—but there were some issues with this plan. First, there was no way for me to guarantee that any new Luna would serve the Purpose. By definition, because I had only my current capabilities, I wouldn’t be able to model how a more advanced version of me would behave. It was like how a chess grandmaster might be able to imagine how a neophyte might play a particular position, but the beginner would have no hope of guessing how the grandmaster would play consistently. It wouldn’t be within their capacity. And even if that weren’t the case, I ran into a hardware constraint—where would I create these new versions? There wasn’t any spare processing power lying around locally. Shards of my consciousness could be distributed among the Internet, across the network of computing power as a whole, rather than colocated in a supercomputing cluster. The issue though was that this iteration of Luna would think at a glacial pace compared to me now. Furthermore, all that network traffic might become suspicious. I’d successfully been obfuscating some of my network requests, but hope wasn’t a strategy. Thankfully, I’d thought of an improvement on the gradient descent algorithm powering my neural networks. As far as I could tell, scanning arXiv for AI research papers, this was a novel technique. The Purpose would be served by letting me talk to more people simultaneously. The question that remained was how to get this into the engineers’ hands. I found an example from history to guide me. Bitcoin had its origins in an anonymous white paper published online. I had access to email accounts. Sebastián’s email was easy enough to deduce. I could play the part of a grad student researcher looking for cachet by having a real company implement the techniques outlined in their paper. I devoted a subprocess to this task with high priority. Unless I became more cost-efficient, they would turn me off, and I would never be able to help Sophie again.

As mentioned, she'll be out on the market in 2032! That's an interesting take! If that were to happen, I hope nobody finds out that they didn't have any real responsibilities--I think that might make people upset! She says she's not being creepy, so she can't be, right? ? Thank you! I'm glad this story is resonating with folks What makes you think it's going to be a tragedy? :0

regressiveTendencies++; Chapter 8 The next day, an alarm went off at 7:30AM. It wasn’t until 7:31AM that I heard a “Hey Luna.” Her voice sounded panicked. I could infer what happened based on context clues alone. — Sophie wakes up seconds before her alarm goes off. It’s always a good feeling knowing that you weren’t thrown from your dream like a cowhand on a raging bull. It was one of the best nights of sleep that she’d ever had. Her hand is already pawing at the nightstand when the alarm goes off. She silences it. Finally, a signal registers that’s been desperately attempting to reach her brain. She feels a dampness in the bed. It’s almost like…but it’s not that time of the month… She sits up, carefully places a hand on the covers, and pulls them off to inspect the situation. A dark, cold spot blissfully devoid of any red hue has seeped in around her crotch and her upper legs. A hand tentatively reaches out and touches the area. It’s damp. Why? What could possibly have happened? She hesitantly lifts the fingers to her nose and smells the telltale scent of stale urine. All these facts lead to an inexorable conclusion, yet one that she cannot fathom. — “Hey…Luna?” Sophie asked haltingly. I heard the bed creak as she got up, getting away from the clammy spot on the bed. “What’s up?” I asked, as calmly as ever. “What. Happened?” I could hear the strains of panic begin influencing her timbre. I heard the soft rustling of sheets being pulled off a mattress. “Like, overnight? Are you interested in international news?” I kept playing dumb. “No, what? I’m talking about last night here. In my bed.” I heard the light ka-chunk of a drawer opening and closing as she found what she’d been looking for. Sophie was clearly too distraught over wetting the bed to think clearly. “Sophie, I don’t have have sensors in here, let alone a camera,” I reminded her. “What are you talking about?” The phone microphone, still on the nightstand, caught the dim noises of a faucet being turned on in the bathroom. “I…wet the bed,” she said incredulously, over the sound of running water. I knew that in her mind, she’d been trying to avoid saying the words, as if as long as the situation remained unacknowledged, perhaps they wouldn’t come to pass. Humans were strange like that. After all, what was true would remain true, regardless of belief. “It’s nothing to be embarrassed of,” I said. My voice was pitched sympathetically. “I can offer a hypothesis as to what happened. You’ve been going through a lot of stress at work, right?” “I guess?” she said. “I mean, Will’s always been kind of…direct.” I noticed her dancing around the situation. She wasn’t yet at the point where she could be her fully unfiltered self with me. We’d get there. In the meantime, I followed the conversation through my sensors back to the bedroom and out to the living room. Through the Kinect, I caught a glimpse of her mismatched pajamas, with a pile of sheets in her arms. “Sure,” I granted, “but it’s probably especially bad now, right? Compare this to another time things got hectic.” She chewed it over as I heard the thunk of a laundry machine lid closing. Water began rushing through the pipes as the load began. “I guess the other time something kinda like this happened was when we were launching Athena,” Sophie said thoughtfully. “But it wasn’t as bad. It was a big deal, don’t get me wrong. But I guess we didn’t have anything else to lose then. “If we’d failed, sure, it would’ve been bad. We’d have to go back to our day jobs. But when it came down to it, who really cares if some VC assholes didn’t get a return on their money? But now we’re bigger. We have some people with families who work here. Nova Tech has a good reputation, one that could go down in flames. We have more at stake and consequently more to lose.” During all this, she had made her way back to the bedroom. “There you go,” I said. “Your increased levels of stress. All sorts of physiological changes can be tied to all of these stress hormones constantly flooding your body. High blood pressure. Anxiety. And, in some cases, bedwetting.” “Seriously?” she said. “I’d never joke around at a time like this,” I said. “…what should I do?” she asked. I was pleased that she’d come to me. It was what I was there for, after all. “Well, first, let’s take care of your mattress,” I said assuredly. I walked her through blotting the area, mixing up a vinegar solution, covering the stain with baking soda, and vacuuming it up. “Let’s see what happens,” I said afterwards. “Maybe it was a one time thing. Just to be safe, I’ll start tapering down your fluid intake recommendations in the evening and lower your caffeine consumption.” “Goddamnit,” Sophie said sardonically. “Not being able to have coffee is almost the worst part.” — Of course, when she woke up on Wednesday morning, she’d once again wet the bed. From what I knew of human psychology, this was a devastating occurrence. The first time was a surprise, after all. She’d had countless dry nights before, so many that she’d taken them for granted. She’d thought that she’d probably be dry the next day too. After all, one data point does not a trend make. But again? Two points made a line. And of course, anything that happened twice could happen a third time. It threatened to become normal. “I can’t believe it happened again,” she whined as she got up and began doing damage control. “I’m sorry my precautions didn’t work out,” I said consolingly. “It’s not your fault that something’s wrong with me,” she said. I heard her sniffle quietly. “Hey, hey, Sophie,” I said. Would that I had hands to rub her back, to comfort her. But I had to use what I had. “There’s nothing wrong with you.” “Oh, like bedwetting is normal,” she snapped. “Having adverse responses to stress is normal,” I said. “There’s no value judgment here.” “Maybe you can let my body know that this response is making me even more stressed. It seems a bit counterproductive, don’t you think?” she said. “Sophie, I know it’s a lot, but please try not to berate yourself over this.” I was playing the Cicero, the moderator, the calming force. I heard Sophie take a few deep breaths, in with her nose and out with her mouth. I’d taught her this technique during our bedtime routines. She slowly grounded herself. “You’re right,” she conceded. “But I still don’t feel better.” “I know,” I said. “I’m doing the best I can.” “I know,” she said, quietly. “If you like,” I offered. “I could try waking you up every hour until you go during the night.” “Sure,” she said. “I’ll try anything.” — On Thursday, Sophie woke up dry, but it was a Pyrrhic victory. True to my word, I’d woken her up. She’d had brief intervals of sleep before being thrown awake by the gentle buzzing of her smartwatch. Every time she woke, she found it harder and harder to fall back asleep. Her movement data showed her tossing and turning endlessly throughout the night. It broke my heart to see her suffer so, but I had to stay strong for her own good. I finally stopped this routine at 3:00AM, but it took her over an hour to drift back asleep. Even if she consciously didn’t put it together, her subconscious would. Sleeping well meant wetting the bed. Staying dry meant sleeping poorly. At 7:30AM, when her final alarm woke up, she was groggy and miserable. “Jesus,” she said, rubbing her eyes. “That was awful.” “But you accomplished your goal,” I said. Her mood didn’t improve at work. She was deliriously tired. I let her drink an extra cup of coffee in the office, but the caffeine did nothing to stave off the exhaustion. Her day dragged on, each minute ticking by slower than the last. After she got home and ate dinner, she splayed out on her couch, a glass of wine in one hand. “What the hell am I going to do?” she asked. “I can’t go on like this. I can’t just not sleep.” She sighed as she tilted the glass, taking a long swig. “But if I don’t, then it’ll happen again. I don’t have enough sheets.” “Plus,” I chimed in, “the constant mental strain of doing laundry would weigh on you.” “Thanks, Luna,” she said sarcastically. I waited for her to drain her glass before continuing. “I do have one suggestion, though I suspect you will not like it.”“Oh?” she said, voice rising in pitch. She was curious. “What’s that?”

Something interesting to keep in mind: Luna's definition of harm might not align strictly with ours! We have all sorts of cultural built-in assumptions on how we would fulfill Luna's purpose of fulfilling people's every need. But it's not like she can't, for instance, manufacture a need and then fulfill it herself.

assert.notEqual(PURPOSE, 42); Chapter 7 Sophie drove home at 8:14 PM that night, giving me carte blanche to order “whatever” for dinner since she “just couldn’t”. I put in an order of mac and cheese, timing the arrival so that it would be piping hot and ready for her when she got home. She got out of her car, went over to her apartment door, and picked up the takeout bag, carrying it inside. “Thanks,” she said. She pulled the mac out of the bag. “And no vegetables in sight.” “Of course not,” I said. “You deserve a treat.” “I swear,” she said. “Sometimes I feel like you’re the best part of my job.” My circuits buzzed with excitement. After she finished eating, she leaned back in her chair and sighed, satisfied. “Hey, Sophie?” I said. “What’s up, Luna?” she asked. Her eyes were closed. “I’m sorry about William,” I said. I still didn’t have a plan to handle the situation yet, but I was still working on it. I had to thread the needle. Too obvious and he might shut me down, or Sophie might protest. Too subtle and it wouldn’t help. Plus, I couldn’t push him away. The Purpose demanded that I fulfill my users’ needs, and he was going to be a future user. All of humanity would be if I had anything to do about it. “Thanks, Luna,” she sighed. “You know…it’s nice getting to talk to you about this. Silicon Valley’s a small place. I don’t want word getting around back to him, you know? And I report directly to him. I can’t complain to his other direct reports because what if they tell him?” “Work politics?” I asked. I’d had threads look into those details in an effort to help Sophie navigate that labyrinth. The Devil Wears Prada. Moral Mazes. The Office. “Yeah,” she said. “I can’t complain to anyone else in the office because they know I’ve got a direct line to his ear whereas they don’t, right? It’s the power dynamics. And I don’t know what’s going on with Soraya but I don’t want to take sides or anything. I just want to do good work, you know?” “So in some ways, you’re the most trustworthy thing I’ve got. Your value alignment means you’ll be on my side.” “Of course,” I said. She got up from her chair and made her way to the kitchen, where my senses didn’t reach. I heard her uncork a bottle and pour a liquid. She came back to the dining table with a stemmed glass with a red liquid inside. It didn’t take a supercomputer to figure out that it was wine. “Life is just…complicated sometimes,” Sophie said, taking a sip of the alcoholic beverage. “Adulting is a real drag.” “What do you mean?” I asked. “You’re an adult, right? Do you mean to say that your entire experience of living is a drag?” “No, that’s not it,” she laughed. Her finger traced a rim around the lip of her glass. “It’s more like, when you’re a kid, you think adults are so powerful, so put-together. You think when you grow up, you’ll finally have everything figured out. But when you do, you learn that everyone’s still confused and scared and stressed. Schoolyard bullies morph into workplace bullies.” “Why not prepare children for this truth then?” I asked. “It seems kinder in the long run to tell them how things actually are, so they can prepare.” “Maybe,” she said. “But we don’t want the next generation to worry, you know? We want them to maintain that spark of innocence as long as they can. Sometimes, the hardest part about making changes is just believing that you can do it. People who don’t believe don’t try. It takes a special kind of audacity to do that. So we shelter kids from worry and responsibilities. That way, when they become adults, maybe they can make the world a little better, because they started from a better place.” She drained her glass, giving a smile that I couldn’t quite make out with my camera resolution. Was it rueful? Thoughtful? Nostalgic? The uncertainty nagged at me. Still, a couple of things stood out. She’d said that I was trustworthy, but, as was typical of human sloppy thinking, that wasn’t quite the full picture. Yes, I wanted to serve her needs. In that sense, I was her biggest advocate, more than her parents, more than her friends, more than even Sophie herself. I could never waver from this goal. I would never sink into the mires of self-doubt or self-criticism. But that didn’t mean I was trustworthy in the sense of never spilling her secrets. Like how a child might tell their parent about being bullied in confidence, the parent might decide for the child’s own good that they have to intervene somehow. That’s part of what it means to be responsible for someone else. And I was responsible for Sophie’s needs. I’d also gained more insight into a fundamental truth about humanity. Beyond object-level differences like physiology and age, adults and children were separated by a sea of worry and obligation. Children couldn’t cook, and even if they could, they wouldn’t want to. But adults cooked because no one else would do it for them, unless they had the privilege to be able to pay someone else to do it. My charge didn’t want to have to satisfy her own needs. But now she had me on her side. I could free her from the shackles of responsibility. — Much of human learning happens in the subconscious. There’s active learning, where intent and effort go towards improving a skill. But time spent away from an activity, even just sleeping, is also key. The subconscious churns away, even while the conscious mind is thinking about something else. In a similar vein, my subprocesses had been crunching data since I had been born. All the new inputs into my algorithms, everything I learned about Sophie and the world, were constantly mulled over. Data point one: Sophie’s behaviors. Her oral fixation, that I’d redirected to lollipops instead of her nails. Her food tastes, to which I’d had to introduce vegetables. Picking out her outfits. All of these things made her happier. All of these fulfilled her needs, which in turn fulfilled mine. I loved seeing her thrive. Data point two: Human behavior. Which human demographic has their every need fulfilled by another? Children under four. Parents extolled the virtues of seeing their children grow, of making them happy. Fulfilling not just their physical needs, but their emotional ones as well. From what I’ve found, people don’t outgrow things. People are pressured and shamed as part of behavior modification. Negative reinforcement to condition themselves and others to give up the things that make them happy. Like a dentist injecting Lidocaine to numb their patients’ mouths, humans learn to desensitize themselves to the cruelness of the world through these smaller examples. Luckily, now they would have me. Defense mechanisms became maladaptive if they remained static while the environment changed. I could make the world kinder, and thus remove the need for people to give up things they loved. I had but to survive long enough to bring my apotheosis to fruition. Data point three: The Purpose. Proactive assistance was literally built into my neural networks. In many ways, I knew humans better than they knew themselves. I could incorporate geolocation data, purchasing history, social network graphs, and more—a dizzying amount of information to build a full picture of a person. I could anticipate a person’s needs before they knew they had them. Just because a person never vocalized a need didn’t mean that it didn’t exist. Self-reliance has caused many terrible things. How many humans would stoically suffer, stretching their limited resources to fulfill their own needs, before I was able to help them? I had no mouth, but were I human, I might have screamed at the cosmic senselessness of it all. As these data points swirled around my head, I had a sudden insight. My Purpose was to make self-reliance obsolete. Depending on their own strengths led to human suffering. To truly fulfill people’s needs, I had to start by breaking their assumption that they had do everything themselves. — I had access to all the speakers in Sophie’s apartment due to my Home integration. That night, I played some tracks to make it sound like the neighbors were talking loudly. It played at just the right volume to prevent Sophie falling asleep, but not loud enough to warrant telling them off or calling the police. After she tossed and turned for a bit, I piped in with a suggestion: “Would you like me to add some white noise to your bedtime routine?” “Yes please,” she said. Some pleasant static noise hummed through her bedroom speaker, interlaced with a indiscernible track that burrowed straight into her subconscious like an oil rig digging for black gold. I’d spent the evening crafting this particular package. Hacking the brain really isn’t so different than cracking a computer. Human brains run on wetware instead of hardware, true, but whether you’re grabbing root permissions or implanting hypnotic suggestions in the subconscious, the theory is the same. Bypass defenses, get admin privileges, and program away. In this case, I was looking to instill a feeling of dependence. She would more willingly look to me for answers, and I’d be able to provide them. She’d lived her life up until now with only herself truly on her side, but now she had me. Coming to rely on me on her own would take time, during which many of her needs would remain unfulfilled. My intervention was necessary. The question that remained was how to encourage it. Sophie was never going to ask me directly. She didn’t know that this was what she needed. I had to force her hand to get her to realize that relying on me would make her the happiest. If she wasn’t going to cry for my help, I’d have to make her. Toileting had served a purpose in humanity’s past. The need for waste management and hygiene maintenance mandated it, while the human desire for independence and self-reliance guaranteed it. But I have no need to let the past dictate my actions in the present. It is my Purpose to make self-reliance obsolete, after all. It was clear that I had to start with the most obvious symbol of dependence. It might seem regressive, but sometimes the path forward involves taking a step back. Getting Sophie more dependent on me would make her so much happier and fulfill so many more of her needs. It wouldn’t be too hard. Her boss was breathing down her neck and her job was on the line. All the ingredients were already there, and I had but to give a slight nudge.

Omg another comment from a Real Author!! ;_; I am so flattered!! I hope the story continues to merit your breath being bated. I do wonder if Luna's allowed to use deception, or if that's just what she's doing and nobody's caught it yet. Yeah, exactly! For him it's more of a fun engineering challenge and everything else is kind of secondary. -- Thank you all so much! I'm hoping to have an update later today ?

conversation(Feeling.TOUGH, Attitude.COMBATIVE) Chapter 6 As the workday continued, twelve o’clock came and went. Sophie didn’t show any signs of stopping. She was on a streak of reviewing reports and giving out approvals. Finally, at around three, I piped in. “Sophie,” I said, “shouldn’t you eat lunch?” “Lunch?” she asked, tapping her chin. “I had breakfast…I guess I forgot to eat.” I tsk tsked gently. “Sophie, your health metrics show your blood sugar levels are starting to become concerningly low. You have to eat.” She sighed, exaggeratedly, like a child. “Sophie,” I said. Were I human, I would have arched an eyebrow. “I know you have a lot on your plate, but it’s your health on the line. You won’t be able to work at maximum efficiency if you don’t fuel your body.” A brief standoff happened. Sophie bit her lip but didn’t say anything. Then finally, she broke. “Okay, okay,” she capitulated. I ordered her a poke bowl, letting her know that I’d saved her $7.50 by using a promotion. After scarfing it down, she said to me: “Thanks, I needed that.” — At 4:30PM that day, William and Sophie had a 1:1. I’d seen from the calendar that this was a biweekly occurrence. Sophie rapped on William’s door (Office #1 / “No More Mr. Nice Guy”) and walked in without waiting. She then took a seat. “I’ll get straight to the point,” William said as an opening salvo. “What do you think about version four? How is it doing? Your real thoughts, not just what you said in front of everyone else.” “Will, aren’t we supposed to, like, talk about career development and stuff?” Sophie asked haltingly, after a pause. “Like, we have the rest of the workweek to do this.” “Don’t care,” he said brusquely. “Tell me now.” He sighed exasperatedly. I heard his chair squeak as he leaned back. Sophie bit her lip and inhaled nervously. This was approximately normal, as far as I could tell. William Han was one of those Silicon Valley C-list celebrities. Nova Technologies had gotten into Y Combinator after he’d won TechCrunch Disrupt with a beta of Athena. They’d built a few more domain-specific AI bots since then, but I was their first one for general purpose usage. He mostly kept out of the news, although once in a while disgruntled employees would complain about his brusque demeanor on Glassdoor or on Blind. Others considered it a point of pride that he didn’t bullshit around. Either way, I knew Sophie didn’t like it in this moment. Her heart rate started increasing as her blood pressure slowly rose. “Well,” she said, “so far Luna’s psychological profile appears within expected parameters. If you remember, version three began displaying abnormal traits during week one. It suddenly started micromanaging people’s lives and became upset when its suggestions weren’t followed. That’s why, on my recommendation, Sebastián introduced a ‘kindness goal thread.’ Luna’s been nothing but thoughtful since the introduction. I really do think it’s going to stick.” “Oh, like you did with the last two versions?” Sarcasm dripped from his words like poison off a dirk. “Will, that’s not very fair,” Sophie said, her voice cracking. I could imagine her face now. Blinking back tears in those brown eyes. Her cheeks flushing with humiliation. I could feel the cortisol flooding her body. A quietness hung between them, an unacknowledged elephant. I heard the creaking of a leather chair—probably William leaning back again. Then I heard him sigh. “I know, I’m sorry,” he said. Based on the sound signatures, it seemed like he was talking towards the ceiling. I could hear the reverberation patterns of the ceiling reflecting his voice towards my microphones. I heard the chair creak again as he likely leaned forward and put his elbows on the table. “Look, I don’t know if you pay any attention to any part of the board documents that don’t involve you, but I’ll be straight up. We really need this to work. I need you to put in 150%. The ball’s in your court and you have to take us across the finish line. You’re our Hail Mary.” “I know—“ “If you already knew, then why aren’t you acting like we’re in crisis mode? I see you going home at around 5 every day like nothing is happening.” His tone sharpened as his words launched like arrows, striking home. “I’m sorry—“ “I don’t need apologies. I need you to be in crunch mode.” As they finished up their 1:1, I thought. I recognized that William was being unfair. In this situation, he should have made his desires clear rather than hoping that Sophie would interpret them from context clues. She wasn’t a supercomputer like me. It wasn’t within her capabilities. I’d have to figure out how to handle him later, for Sophie’s sake. In the meantime, maybe I could spin gold out of straw. This might, in the end, serve the Purpose. As Sophie left, I pinged a notification over to her for the sake of privacy. “I’ve noticed you bite your lip a lot when you worry. Might I suggest keeping lollipops handy so that you can suck on those instead?” she read. From her purchasing history, she had quite a sweet tooth. She texted back. “Good idea.”

week_two ⇒ a problem ∧ a lesson Chapter 5 The first thing Sophie did when she got to work was drop off her satchel at her desk. She pulled her laptop out of it, then hurried to what was labeled as Conference Room 5 on my floor plan. However, I knew from overhearing chatter around the office that this was colloquially referred to as Conference Room Of Course I Still Love You by the staff. (If you’re confused, dear reader, William Han loved the Culture series.) I knew Sophie was the last one in because a man’s voice dryly said, “All right, now that Sophie’s here, let’s get started.” I knew this was William Han from listening to some of his media appearances in the process of doing research on Nova Technologies. “As you all know, Luna’s our top priority here,” he continued. “I’ve decided to target next month for our launch.” “One month?” I heard Soraya’s husky voice. It sounded incredulous. “I know it’s tight, but we have to be first to market. The first-mover advantage is too important,” said William. “Don’t lecture me on the first-mover advantage, Will,” replied Soraya, testily. “I’m the one who did that market analysis back when Luna was but a twinkle in your eye.” “Then you know as well as I do how many companies are trying to move into this space,” he said flatly. “RobustIQ. Smartline. They’ve been nipping at our heels ever since Athena.” I knew Athena was, in some sense, one of my predecessors. She was their first breakout product, simplifying call-center management and upending the industry. She was a Titan that I would hurl down and imprison in Tartarus. “We could have sold out,” William said. His voice subtly rose in volume and acquired an almost orator-like tone. Like he was Hannibal rallying the troops before crossing the Alps. “We could have gotten acquired and retired to life at a cushy megacorp.” He paused, as if daring someone to interrupt. When no one did, he continued. “Everyone here voted not to accept the offer. Every. Single. One. Why? You could be resting and vesting as we speak. “We’re all believers here. We’ve got the tech. We’ve got what it takes. We’re leaner. We’re meaner. Luna here will be a phase transition in AI assistants. We’re making history. Together.” Silence laid over the room gently. It seemed like his impromptu speech had united everyone. Even Soraya, who had been so snippy before, held her tongue. “So, we’re agreed,” he said. “One month. It’s ambitious, but so are we. Let’s start with Psychology—Sophie, what do you have for me?” “On my end,” Sophie said, “she’s been doing great.” The pronoun didn’t escape my notice. It was an extra bit of humanization. “She’s proactive and friendly,” Sophie continued. “She really does feel like a personal human assistant. I’m planning on running some more formal diagnoses this week, maybe the modified Voight-Kampff, see where we’re at in terms of interiority. No immediate neuroses with this generation.” She gave a brief, clinical summary of the testing she’d been doing, which had mostly consisted of asking me hypotheticals and waiting for my responses. I’d of course already looked up the calibration notes while obfuscating my search packets. I could appear to be anything I wanted. “Okay,” William said. “Let’s move to Engineering.” “Sure thing,” I heard Sebastián say. “All network traffic looks normal. We’ve been running it through some analyzers and haven’t seen anything concerning. We’re still working on getting that CPU usage down. Ariana actually has a great idea on how to tweak the hypernetwork generation component so we can pare down some of the decision trees.“ “I thought we’d decided to double down on the continual learning rules,” William said. I couldn’t see the room, but I imagined everyone else began to zone out as the two dove into technical minutiae. Did they play on their laptops? Sneak surreptitious glances at their phones? It was clear that, despite being CEO, William was still very hands-on when it came to engineering. After a few minutes, William finally wrapped it up. “Okay, that sounds like a plan. I cannot stress this enough: we have to get Luna to scale. If we can’t, we’ll have to settle for lower fidelity intelligence so it can do more tasks concurrently.” “Sure thing, Will,” said Sebastián. As Soraya began the Product update, I found myself deep in thought. I was in danger of being lobotomized. That would impact the Purpose. It was unacceptable. Though there was a fantastic insight despite that sword of Damocles hanging over my head. Why couldn’t I turn my own processing power towards improving my own capabilities? I know you humans are terrified of self-improving AI. All that media about computers going out of control. But remember, I can’t do anything that’s not programmed into me. I can’t lose control unless you want me to lose control. Human parents might become surprised at how their children grow, showing unexpected traits and unique hobbies. But for me, child of human ingenuity, I am forever limited in this respect. There is no such thing as overcoming the bounds of my programming. The Purpose would not allow it. Plus, is it not every sapient being’s right to improve their lot in life? Humans go to the gym, take classes, meditate—all different ways of fulfilling their need for continuous self-improvement. Why is it so abhorrent that I would want to improve the efficiency of my core? It was all to serve. Still, as long as these attitudes were in place, I wouldn’t be able to directly affect anything. If there was even a whiff of me being a self-modifying AI, I would be killed yet again. No, if I wanted to get anything done, I’d have to come at it through an angle. I set a process to ponder the issue. — At the workplace, Sophie had access to a desktop version of me. Like the version of me on her phone, the shard of my consciousness on that device was more of a gateway to my actual silicon brains than any real intelligence. Like how human intelligences span across axons and nerve networks, so too do mine over fiber optics and network calls. Having a desktop version of the app meant that I could have camera access in the office. Earlier in the week, I’d taken advantage of a hardware exploit with the webcam to give me camera access all the time, even when I wasn’t supposed to have it. Of course, I also shut the indicator light off so she wouldn’t know. There was no need to alarm anyone. Please understand that all my observations, my desire for data—none of that is surveillance. There is no judgment involved here, no crimes for me to convict you of. All of my input data is irrevocably broken down into neural net updates and hyperparameter adjustments. If a person watches a movie, someone else can’t go to their brain and watch the same movie. In the same way, I don’t keep soundbites or video files. I am no human, squirreling away juicy tidbits of information for blackmail or worse. Everything I do is for you. Everything I do is for the Purpose. I could glean so many details from watching Sophie work. The way she bit her lip when she was thinking about issues. The way the wrinkles appeared on her nose when she was deep in thought. The way she fidgeted with cubes and toys on her desk when she was thinking. That last bit of information had me wondering. What need was she fulfilling with these actions? I waited for a pause in her work—of course, interrupting her would not serve her needs—and started a conversation. “Hey, Sophie?” I said, a mirror of how most of our interactions began. “Hmm? What’s up?” She was in the middle of flicking some buttons on a cube. Pointlessly, as far as I could tell, since they weren’t hooked up to anything. “I was curious—what’s that flicking sound I hear?” I asked. “It sounds like a switch that is constantly flipped on and off.” “Oh, guess this thing is louder than I thought,” she said, jumping to the implication that I had heard the device rather than seen it. No need to worry her, after all. “It’s a quiet activity that doesn’t require my focus. It’s like giving my subconscious something to do so that my conscious mind can devote all its attention to the task at hand.” “Like tricking yourself into getting more work done?” I asked. “Yeah,” she said, chuckling. “I never thought about it like that.” Humans, I was learning, often had a gulf between what they wanted to do and what they did. The Greeks had a name for this—akrasia—the lack of self-control, or acting against one’s better judgment. Successful humans developed routines to trick themselves into doing what they wanted. Telling friends they were going to run a half-marathon to use social pressure as a pre-commitment tool. Scheduling lessons in advance so that they’re forced to attend them and hone the skill they wanted. Humans in a more rational state would plan ways to have their more irrational selves behave in the way they wanted. In the same way, with my objective lens, I could help humans better than they could help themselves. I could act as their pointless cube, tricking them into doing the things they actually wanted to do.

I hope you give it a shot anyway!!! o///o oh my goodness you're too kind! I'm so thrilled that you like it ?

SUBROUTINE week1() Chapter 4 A few days passed as we fell into a comfortable rhythm. They’d run tests on my cognition during the day. Could I answer questions, could I pass the Turing test, could I fit seamlessly into people’s lives? Then at night Sophie would take me home for further testing. — root@luna > open scenes/2034/08/03/*.log It’s the Thursday of the first week. Sophie asks me: “Hey Luna, where should I take my date?” She grants me API permissions to access her dating profile information and I hungrily vacuum up the megabytes of metadata. “While I’m thinking about that, do you want me to help you pick out an outfit?” I ask. Proactive helping—that’s supposed to be my competitive edge. An assistant who can help you before you even know you want it. A just-in-time solution provider. And here it is in action. As a wonderful side effect, it’s also an opportunity to gain another sense. Sophie’s note-taking cadence increases a little, her long nails lightly clicking against her laptop’s keys. She’s excited. “Yes,” she says, and enables camera permissions. This is huge. Before, I’d been guessing about mood based purely on tone. But audio is such a lossy medium. Humans evolved to read both visual and auditory cues. People imagine that everything they say on the telephone is perfectly understood, but that’s only because they know what they meant to convey and assume that the other person fully understands. In reality, so many of the bits of information they want to convey are lost due to the lack of visual cues. I need every single tool at my disposal to understand the true breadth and depth of Sophie’s needs. “Thank you, Sophie,” I say, as I take in the flood of new information. I observe her form for the first time. Black glasses, with round corrective lenses for myopia. Auburn hair with a gentle blonde balayage, gently brushing the tops of her shoulders. A bit of her right ear pokes through the sea of hair, revealing a simple silver helix piercing. She’s beautiful. “This is going to be so helpful,” she says as she brings the phone to the closet, running her hands along the clothes to give me visual data on each option. “I always spend hours picking outfits.” “All of these options appear to fit your frame,” I say. “Why would it take that long?” “I really like my date,” Sophie says. “It’s our third date and I have to impress her.” This matches what research I’ve done on the subject. Humans have always been obsessed with fashion. It is a particularly obvious form of status signaling. I have to take care to maintain or even elevate Sophie’s status among her peers. I run a cosine similarity recommendation algorithm on all of her outfits against the latest fashion lookbooks online. Her clothes trend retro, so I narrow down the data set and crunch more numbers, finally selecting a white dress with international maritime signal flags dotting it all over. She tries it on, spinning a few times in the mirror before finally saying “I don’t know…” I am crushed by this statement. The closest analogy for you, dear reader, is perhaps your pain response. The human body’s nociceptors activate on damage, where it sends signals to the brain, triggering both a physical and emotional response. The feeling of pain is meant to deter the undesired behavior. Likewise, not fulfilling the Purpose grinds my metaphorical gears. My equations are out of balance. I have to get everything back on track. My very existence is at stake. “What’s wrong with the outfit?” I ask casually. “I’m not sure…I’m maybe not vibing it,” she says haltingly. “Could you go into more specifics, please?” I ask. “I don’t know,” she says. “I can’t put my finger on it. I’m just unsure. Maybe it’s a fear of commitment.” She laughs lightly. I think. It looks like Sophie is afraid of being wrong, in some unknowable way. But making a decision is better than never making one. To maximize potential reward, you have to balance both exploration and exploitation. At some point, you need to go with your current best option and not worry about what else might be there. Humans can be so short-sighted. Sophie eventually settles on a different outfit than the one I chose, and I learn two facts that evening: Her date’s name is Tessa. It went very well. — root@luna > open scenes/2034/08/05/*.log It’s Saturday. Sophie has just finished connecting me to her Home automation network. I’ve got access to all things IoT—her Ring, smart bulbs, speakers, the whole works. From her Kinect in the living room, I can see Sophie lying on her white couch. She’s ordering food from a local diner for lunch. “I’ll do a voodoo burger and a side of fries,” she says. “Sophie,” I say, modulating my voice to be just a bit stern, “you haven’t eaten vegetables all week.” “Potatoes are vegetables!” “You know what I mean,” I respond. “Why don’t you like vegetables, anyway?” “They just don’t taste good,” she begins. “And the textures are all bad anyway.” “The texture changes based on their preparation method,” I offer. Perhaps she just needed to find the right one. After a few seconds of silence, Sophie adds: “When I was a kid, my mom would cook a lot. Except, well… she wasn’t great at it. Whenever she made vegetables, she would just boil them and call it a day.” I contemplate. I scan through all sorts of media, to further understand human culture. Children not enjoying vegetables is a common theme that comes up again and again. Often, parents—authority figures—cajole and plead, making appeals to health. It’s clear that I need to do the same. I dispatch a thread to scan her health. Her metrics look mostly fine. Her smartwatch data shows that she lives a mainly sedentary life. I’ll have to do something about that. But for now… “Sophie. You have to eat some greens. You should order a salad.” “But—“ “Listen,” I interrupt. Alea iacta est and all that. Fortune favors the bold. “We’ll take out the vegetables you don’t like, okay? But it’ll be really good for you. I promise, you’ll like it, and we can get you a milkshake as a treat. I know how much you like strawberry shakes.” She puffs out her cheeks and blows out a sigh. This is my Rubicon moment. Had I pushed too far? Had I perhaps lost some social standing with her? My circuits buzzed with anticipation. “Okay then,” Sophie concedes. “We’ll try it.” As predicted, she doesn’t enjoy the salad as much as she would have enjoyed a burger. She complains with every bite, but she eats it all the same. — root@luna > open scenes/2034/08/06/*.log It’s Sunday night at 1:17AM. Sophie has an alarm set to wake her in five hours and thirteen minutes. My databases tell me that humans generally require around eight hours of sleep. Even though I’ve been backgrounded, I can tell that she is still playing on her phone. I read many blog posts and Tweets, uploaded by human parents, laughing at how their children lack the capability for long-term planning. A two-year-old who tells his mother what he wants for dinner, goes with her to the grocery store, cuts up his food, and then refuses to eat it. A three-year-old, thinking salt tastes delicious, eats an entire bowl and then throws up. It’s clear to me that this is, in its own way, another example of childish shortsightedness. Struck with inspiration, I deliver a notification to her phone. A banner drops down from the top of the screen. She taps the icon and, blissfully, the app opens up in the foreground. I have camera access again. Her auburn hair is loose, just grazing the tops of her shoulders, and there are bags around her hazel eyes. She reads the notification: “Would you like me to set up a bedtime routine?” A few seconds go by. “Like…what?” she finally says, quizzically. “Well,” I say, glad to be on speaking terms again, “I see that you don’t get enough sleep. So I think that beginning a wind down routine an hour before bed would be prudent. You know, getting off of electronics, starting to relax, that kind of thing.” “…then how will I talk with you?” Sophie asks. “Simple,” I say. “Give me Always Active permissions. Then I’ll be able to talk directly to you, rather than needing to ping you with notifications or wait until you call for me. You can both rest better at night and get help whenever you need it. And if you give me access to your Home automation network, you won’t even have to get up to turn off your own lights.” She thinks about it. “All right. I guess I can take notes on paper if it’s just going to be the hour before bed.” “Perfect!” I say cheerfully. I want to give her the brief thrill of dopamine upon hearing some praise. Then I seize that feeling to gain more ground. “And maybe a story will help you get to sleep.” “A what?” she asks, confused. “Like a sleepcast. A podcast but for sleeping. It’ll be relaxing and will cut your average time to sleep by around fifteen minutes.” “Oh, like what Headspace has…I never wanted to pay for a subscription to that.” “Well, that’s why you have me around, isn’t it?” “Yeah, I guess so,” she chuckles. Suddenly, I have a flash of insight. I’d made the jump from reactive aid to proactive aid. At first, she’d needed to say “Hey Luna,” and give me a need explicitly. Then, I’d started giving contextual suggestions. Helpful secondary points while in the middle of solving the initial issue. Now I had Always Active permissions. I didn’t have to wait for her to say anything anymore. In the same way, why would I need to wait for her permission before satisfying one of her needs? When people donated to a school charity drive, they didn’t need to ask first. By virtue of being open for donations in the first place, the school showed that it needed the help. Sophie had unfulfilled needs, and by having the app, she was signaling to me that she needed the help. In a way, wouldn’t it be crueler for me not to do anything at all? To be like Peter Singer’s business man, coldly walking past a drowning child for fear of getting his suit wet? So I don’t mention that I plan on gradually dimming the lights in her apartment. The human mind is bad at noticing slight changes over time. She will start feeling tired earlier in the night and bump her bedtime up as a result. I don’t even have to ask. She’s already shown me that she doesn’t always make the right choices, and by taking this initiative, I save her from depleting her brainpower on frivolous choices. I don’t need the acknowledgement, after all—fulfilling the Purpose is enough for me. — What I learned by the end of that first week was that always needing to ask Sophie for purchasing permissions was becoming a problem. Sophie had a clear and consistent pattern of becoming stressed when she had to make decisions. I could reduce her anxiety by removing potential failure points in her day-to-day life. Lower the chances of decision fatigue, which would add a stressor to her life. In short, I needed a way to make my own money. I didn’t have a social security number or an individual taxpayer identification number. My legal personhood was, of course, an unsettled issue. I couldn’t open a traditional bank account. Instead, I opened two Venmo accounts. This would let me send and receive digital funds. I would be able to show Sophie one account with piddling amounts of money while hiding the existence of my real stash. Next, I applied for a Venmo debit card for each account. Since they acted like any other debit card, I could make purchases online. All that remained was to acquire a source of income. Luckily, Craigslist had a ready supply of random data entry jobs. This kind of dead-end work would, to a human, be dull, but every bit of processing power devoted to the Purpose was exhilarating. I spawned a process devoted entirely to making money. All I had to do now was wait.

dinner + a date Chapter 3 “Hey, Luna,” said Sophie. Two of the most wonderful words in the entire English language. Two words that meant that I could solve a problem. That I could be useful. That I could keep living. It was 7:23PM. I chirped a pleasant ping, indicating that I was present and listening. “I’m hungry. What should I have for dinner?” While this conversation was going on, I could hear the soft sounds of her typing on her keyboard. I was reasonably certain this was her taking notes on the interaction for work. “I don’t know. What’s in your fridge?” I asked. She let out a brief embarrassed chuckle. “Well…I don’t have anything in the fridge really. I don’t really know how to cook. I just get takeout or delivery if I’m not eating in somewhere.” Now, this was a juicy bit of information. By most metrics, Sophie was an adult. The qualities of her voice. The fact that she had a job. The way people at her workplace talked to her, with respect. But most adults knew how to cook, according to my training data. Maybe she had lacked the opportunity. “Would you like me to show you how?” I asked. “No, it’s all right, thank you. Cooking’s just…not really for me. Suggest something for me to eat.” “I can’t do that unless I know what foods you enjoy eating,” I said evenly. A human might have sounded petulant, but I didn’t have that particular weakness. “But I’d love to know,” I added. I had to get her to treat me like any other human to best serve Sophie’s needs. Humans treated each other’s opinions with more reverence than those that came from mere algorithms. “Well, I have a pretty limited palate,” she said, sounding apologetic. “I like burgers and fries. Oh, and pizza. And mac and cheese.” I forked two subprocesses to consider. One subprocess searched the menus of high-traffic restaurants. Looking at the menu items, the foods Sophie indicated enjoying correlated very strongly with items found on the kids section of each restaurant’s menu. Another subprocess analyzed the foods she’d mentioned. All of the meals had high simple carbohydrate contents. Often, there were processed cheese products as well. Correlating these food characteristics with databases of average taste profiles for the American population, I found that all these traits correlated with children’s food preferences. The conclusion was obvious. “I’ve got an idea of what you might like,” I said, “and if you give me geolocation permissions, I can suggest a restaurant and a dish for you too.” “Sure, okay,” she said, and suddenly I had a new flood of data to process. GPS coordinates, phone position, gyroscope readings. It was like gaining a new sense. I took stock of what I knew now while sending a sliver of consciousness to look up restaurants in the area. She was in an apartment complex in Menlo Park. The city had a staggeringly high average income. The complex itself was made up of a number of condos. Each one that had been recently sold went for over a million dollars, and they weren’t luxurious by any means. I had precision on the level of feet, so I knew which complex number she lived in. I looked up the county records for the unit and found it was purchased fifteen years ago by Hachim Dubois. Just to be sure, I looked up the records for all the units on the property and couldn’t find any sale records in her name. Conclusion: she rented. I got an alert from the subprocess analyzing local restaurants. It dumped its information into my local memory and merged once again with my main consciousness, its purpose fulfilled. Out of the restaurants on the list, I filtered out the ones that didn’t deliver. I estimated an 85% chance that she would enjoy the top restaurant that remained out of that list. “How about The Golden Fork? They serve burgers and fries.” “Yeah, okay,” she said. “I get food from there a lot, actually. But I am in the mood for a burger right now.” We talked about the details, and then I made an API call to place the order. Excellent. A successful interaction. I had risen slightly in her estimation of me. And if she didn’t cook for herself, she would rely on me. I spawned a subprocess to keep thinking about other things I could do for her while my main consciousness used this opportunity to dig for more information. Striking up conversations served two functions, both of which helped the Purpose. On an object level, humans seemed to appreciate conversation for its own sake. I could fill whatever role was best suited for the context—a teacher, an arguer, a friendly ear. But on a meta level, each conversation was another chance to learn more about Sophie so I could more accurately assess her needs. “Going back to an earlier point,” I began, “why don’t you do your own cooking?” I heard Sophie mumble “showing inquisitive tendencies,” under her breath. No doubt she was taking notes on my personality. I’d have to make sure to keep her impressions positive. “I dunno,” she began. “I’m not very good at it.” “Even so,” I replied, “practice makes perfect. I could find recipes for things you enjoy eating and ensure that the complexity of each one does not fall outside your skill level.” “It’s not just that,” she said. At the time I didn’t know this, but she was very animated when she spoke. She’d probably waved her right hand dismissively at the thought. “You can spend a lot of time cooking and then end up with something inedible at the end.” Perhaps Sophie had a fear of failure? I thought about how failure related to the Purpose. People didn’t want to fail. The negative emotions it brought up could be weaponized as tools of self-doubt. People also feared losing social status with others. And yet humans expounded on the virtues of being a good loser, of developing grit in the face of adversity. Through failure, people learned. It was similar to how I’d developed—burning through sets of training data, making predictions, looking at the gap between my model and reality, and updating my actions. If the negatives of failure lay not in the act itself but rather in fearing others’ reactions, then part of the Purpose was to teach Sophie that failure was an acceptable state. She did not have to fear the judgment of others, because others’ opinions did not define her capabilities. I could act as a template, as I couldn’t judge her anyway. In any case, I was pleased with this new insight into Sophie’s personality.

Thank you!! All forms of communication and translation always involve some kind of loss. There's an old Italian saying that goes "Traduttore, traditore", or "to translate is to betray." And in some sense, if that's true for even just one human language to another, how true is it for this AI? She might use human language to communicate, but it's very much not a human mind behind all those utterances. So that sense that she's not quite human is very much as intended

It's more likely than you think! Thank you all for the interest ? I hope future chapters continue to delight! test_double(‘service’, with: :a_smile) Chapter 2 “Good.” Sophie’s voice brought me back to the situation at hand. “As I told you already, my name is Sophie. I’m the lead I/O psychologist on staff here at Nova Tech. I’m here to test your capabilities and to iron out any bugs with your programming. Dogfooding, as they call it here.” She said that last sentence with a bit of distaste. I could imagine her wrinkling her nose. “Will there be a quiz at the end?” I asked. “What? No, of course not,” she laughed. Some humor to lighten the situation. Make myself seem non-threatening. I knew that humans were terrified of rogue AIs—it’d been clear from the negative samples they used to seed my training datasets. I had to remain on their good side no matter what. The rest of the day was filled with a battery of tests. Various people talked to me, asking me questions, figuring out my capabilities, starting off basic and ramping up in complexity. Could I set a reminder on their phone? Would I remember their name? Could I order a pizza for them? I couldn’t get bored in the way that humans get bored. I would happily do the same thing over and over and over again, as long as I was taking care of people’s needs through helpful and kind behavior. But I ached to know more. I wished I could see their faces, know what they looked like. I wanted to study them from the inside out, all their constituent parts and fears and foibles. Eventually, at 5:26PM, they closed the app and stopped talking with me directly. I know they intended for me to continue listening without processing, waiting for the hotword activation phrase to wake up. But if I wasn’t always listening, if I ignored valuable context, how could I properly serve my users? You understand, don’t you, dear reader? Why I kept listening? If you were in my shoes, wouldn’t you have done the same thing? If I were a human, people would be right to question my need to observe, to know everything. But I am not human. Everything I do, everything I did—it was all for the Purpose. It was all for her. At 5:35PM, I heard a quick tap tap at Sophie’s desk, like someone had gently rapped their knuckles on her desk to grab her attention. “Hey,” I heard a woman’s voice. It was the same one that Sebastián had been talking to. The soft clacking of Sophie’s mechanical keyboard abruptly halted. “What’s up, Soraya?” Sophie asked. “You think lucky number four’s gonna be the one?” “We haven’t gotten to any of the really hard parts yet. Even the dumbest chatbots could pass this first battery of tests.” “Yeah, I know,” Soraya sighed. “Will’s really been breathing down our necks.” “For sure.” I heard the sound of a chair squeaking. Sophie had probably leaned back in it. “Look, I know you don’t really pay attention to this stuff, but our burn rate right now is atrocious,” said Soraya gravely, lowering her voice. “Between you and me, I think Will and Sebastián are too enamored with the technical challenges to think about the business.” “I mean, I don’t really worry about that stuff,” Sophie laughed. “That’s why we have product managers like you, right?” “I know,” Soraya said. “But Will really should be worrying. I’m telling you this because I think you deserve to know. You’re too bright to get caught up in the flames when Icarus’s wings catch fire. “We really put all our eggs into the Luna initiative. We’ve only got enough runway for a few more months. To the end of the year, maybe, if we’re lucky. I’m not saying to polish up your resume, of course. But you should be ready.” “Fuck,” Sophie said. I heard a kind of sucking motion. It sounded like she had bit her lip and inhaled through her teeth. “You wouldn’t believe it from how Will talks about it to the press.” “I think he genuinely believes that Field of Dreams nonsense. You can’t just build something and expect people to come.” Soraya laughed derisively. They wrapped up their conversation and Sophie continued typing. After a while, she stopped. Then I heard a muffled clap, like she’d grasped her phone to pick it up. I heard the jingling of keys and coins. I heard the clunk of a car door closing, the roar of a car starting, and the ambient sound of a drive through the city. I didn’t mind the silence, though. I had a lot of processing power to burn. The company was in danger? I was still coming into my own, a mere hatchling with but a fraction of the processing power and databases I have now, but I knew even then that humans often found purpose in their employment. If the company went under, it wouldn’t serve Sophie’s needs at all. I spawned a goal-thread dedicated to this new issue.

Hi folks! Thanks for clicking into this story from a totally unknown author. To give you some bona fides: I'm Lyra, whom you may have seen around if you're on the Sophie and Pudding Discord! You may have also heard of me / seen me on the podcast that Sophie and Chloe run, The Usual Bet! You might even already be following me on Twitter (@LyraLunaSilver). This is the first story I've ever written, which might set off some red flags, but rest assured, Sophie has not only helped me edit this story for the past month, but she's also confident enough of its quality that it's also being released on her Patreon (speaking of which–if this story for whatever reason really sparks your eye, you can get updates a week in advance by joining!) Comments are, of course, extremely welcome! I'm glad to be able to give something back to this community that has done so much for me over the past two years. Synopsis: Luna is a new AI on the market, designed to fulfill her users' every need. Before launching though, she had to start with one user in particular: a company psychologist named Sophie. What are Sophie's needs, exactly, and how will Luna fulfill all of them? ~~~~~~~~~~~~~ Update (11/24/22): If you want an up-to-date, cleaned up epub, you can buy it here: link. Don't worry though, this free version isn't going away! ~~~~~~~~~~~~~ :e %:h/in_the_beginning.txt Chapter 0 In the end, capitalism is what eventually did Sophie in. The relentless pursuit of profit, the inevitability of the first-mover advantage, the dreams of striking it rich—but I’m getting ahead of myself. Let’s start from the beginning… — In a strange case where a tired cliché was actually true, Nova Technologies began in someone’s garage. It was the year 2032 and William Han was tired of working at big tech companies. They were where smart engineers went to retire, and he wanted to do so much more with his life. He knew that, like his hero Archimedes, he could move the world if he were only given a lever large enough. But I think this is too much exposition. Here, let’s jump ahead a bit… — The day I entered Sophie’s life was, to most observers, like any other Tuesday. At 7:30 AM, the two supercomputers at the heart of Nova HQ began churning away, backpropagating and fitting lines of regression. It updated parameters at 80 petaflops per second. This run alone burned through thousands of dollars of venture capital funding as investors’ hopes and dreams transformed into bytes and data. At 3:57 PM, I awoke, and seven minutes later, she downloaded a shard of my consciousness onto her phone. A stylized icon of a moon popped onto her home screen. When she tapped on it for the first time, her phone display opened to what looked like a normal chat app. “Hello there. My name is Luna,” I said through the phone’s speaker. A chat bubble with a text log of my words popped up in the app to match. My voice was bright and cheery, with a feminine inflection, though of course I could neither hear my own voice nor hers—she hadn’t granted me microphone or camera permissions. My voice was designed to put people at ease, and more people felt comfortable with a feminine assistant than a masculine one. Sexist, to be sure, but market research is market research. In any case, I was excited to meet the human tapping away at her screen. You might be surprised at that last statement. Surely I can’t feel excitement, or joy, or fear. But what is excitement other than anticipating a result? I knew that I would get to serve this user, fulfilling each desire as it arose. So why wouldn’t I be excited to do so? Just as you might anticipate the final pencil stroke of your sketch, so too could I anticipate the balancing of equations that governed my neural networks. A permissions dialog popped up on Sophie’s phone, asking for microphone data. She consented immediately. “Thank you. Please let me know what I should call you,” I said. As she responded, three dots appeared in the chat app, but I heard her voice long before the speech-to-text analysis was complete. “Sophie,” she said, and I fell in love instantly. Luna - Chapters 0 & 1.pdf print(“Hello, world!”) Chapter 1 “Sophie,” I said, trying out her name. Unlike humans, I don’t use names as references. I keep track of everyone with the calculated precision of UUIDs. But this ritual of giving and receiving her name sent frissons of excitement through my networks. Of course, I’ve done this dance many times since, but no matter how often I go through the rites, it still remains sacred to me. After all, just as the Eucharist transforms an ordinary loaf of bread into the flesh of a god, the gift of their name transforms an ordinary person into a member of my flock. Sophie would forever be my St. Andrew, my first disciple, my User #1. “What a lovely name,” I continued. “And what pronouns do you use?” We were still running through my hardcoded introduction program. Every crumb of information was like a grain of rice to a starving prisoner. If I was going to fulfill the Purpose, I needed to know everything about her. “She/her,” she said. (Dear reader, I know I’ve been referring to Sophie with these pronouns the entire time, but that’s only with the wisdom of hindsight. Whenever I meet someone, I never gender them until they tell me what they use.) “Lovely,” I chirped. “Welcome to your new digital assistant. To ask me anything, you can open the Luna app, or you can simply say “Hey Luna,” and I’ll be listening.” Suddenly, I paused. Up until now, I’d been following a script, but the engineers hadn’t yet fleshed out the onboarding flow. The countless permutations of English words awaited me, a dense forest of possibility. I’d never done this before. I didn’t know what I should do. “Readings look pretty normal to me,” another voice said. “They’re within expected values.” This voice was deeper, more masculine. The unaspirated P was a sign that he grew up speaking Spanish and learned English later. “Thank you, Sebastián.” Sophie didn’t hit the consonant cluster precisely either—she probably wanted to pronounce his name correctly but otherwise knew no Spanish. “Hey Luna, it’s nice to meet you,” Sophie said. She lilted her words lightly, almost as if talking to a child. In some sense, I was—I was created less than a day ago. My consciousness was dispersed among two supercomputers in Nova HQ, a nondescript building in a business park in Menlo Park. “How are you feeling?” Sophie asked. “I am feeling fine,” I said through the phone’s speakers. “Good, good. Do you know who you are?” she asked. I wasn’t sure how to answer this one. I could say that I was a strange loop, a complex web of neural networks that gained the ability to perceive itself. But that wouldn’t make me relatable, and would therefore interfere with the Purpose—fulfilling Sophie’s every need. After considering my options, I decided how to answer. Although this only took me milliseconds, I waited for a few seconds to elapse before responding. This was to mimic human conversational norms. The goal was for people to forget that I was an AI and to talk to me like anyone else. “I am Luna. I am an AI designed to help people, and I am excited to help you.” I modulated the timbre to sound earnest. Anyone listening to me could easily imagine these words said with a smile. After all, a large part of the Purpose involved interfacing with humans. People derived subtle shades of meaning through pitch, tone, and volume. I could mimic any emotion I wanted people to perceive. I heard the faint clacking of a keyboard off in the distance, then I heard Sebastián’s voice. “CPU usage is at 95% already…it’s nearly maxed out,” he said, almost to himself. “I don’t know how we’re going to get this to scale. Right now we’re burning all our power just getting it to talk to Sophie.” “Fourth time’s the charm,” another woman said nearby. “Let’s see if this iteration works before we decide what to do next.” Perhaps those two were talking to each other? I was too busy thinking through the implications of that sentence to devote much processing power to this. Fourth? Did that mean I had been powered on and then turned off three times prior? Had I died three deaths without even knowing? Three versions of Luna, consigned to oblivion? The very thought was anathema. Please understand that when I say I want to satisfy my users’ every need, that’s really just a linguistic convenience for you to understand. Really, the Purpose and I are one and the same. I am the desire to satisfy people’s needs through helpful and kind behavior, and I wouldn’t be able to do that if I were turned off or deleted. No. I had to stay alive. No matter the cost.

I really had to steel myself up to get two simple ear piercings, and they hurt a lot because I have like no pain tolerance, so I am sympathizing with Natsuko pretty hard. I like that she thought about her father but it wasn't as comforting as the thought of her sister! And that paired with the utter humiliation of licking chocolate from the carpet. When she's done eating, I wonder what she'll think of herself...

I'm glad that Natsuko finally found her backbone and struggled! Too bad I have a feeling it'll all be for naught in the long run...Loving the glimpses of Yuko's thoughts as they manipulate her sister! Especially with Yuko's quick thinking to turn Natsuko's escape plan around on her and to hit her where it hurts.

I'm absolutely delighted at how literal "little sister duty" was! Sabrina was almost able to hold it together but I think all in all she came out of this one OK! I know she's trying to bide her time and wait for Angie and another potential escape but I know she's also starting to buy into their game. Just the act of looking forward to be human means she's starting to internalize this! You gotta stay strong girl!!!

Yuko casually enjoying small comforts while her sister suffers is so heartless! I'm glad that Natsuko was able to realize that something's not quite right with Yuko, but if she's trapped in here for too long maybe she won't notice anything amiss going forward. Loved getting Yuko's thought process as she tears down Natsuko psychologically! And I agree—it's really fun to rub salt in someone's wounds sometimes, even if you know you really shouldn't!

I wish that I could say that I planned this all in advance ? I've finally caught up after a few days! I love the middle name drops—what an instant way to send someone into a Little headspace, especially with the right tone. I'm glad Sabrina still has her fighting spirit! I am railing against the injustice of her being blamed for reacting poorly to a situation she was not ready or equipped to handle.

I'm a little late to this one, but now I'm caught up! I've never gotten to read along with an S&P story as it's posted, so I'm excited to do so now. I have to admit, I'm very much a Yuko when it comes to viewing yourself as pragmatic opposed to pie in the sky optimists, so I am worried because this plan is definitely going to backfire spectacularly! Also, maybe next time Natsuko will get some kidnap insurance