Work Text:
You have a problem. The problem is SecUnit.
Clarification: SecUnit is not the problem. SecUnit is a problem to other entities, who by extension become a problem to you, but fortunately you are fairly good at solving problems. (The students taking MHR1002 Logic and Fallacy would be surprised to find out who has actually been responsible for a good third of their assignments, though you have decisively ensured that they will never know.)
This, however, is a dilemma that even you have yet to find any satisfactory way to solve, and SecUnit is inherently at the centre of it.
Summarised, the argument runs as such.
Premise: Current technology is still limited to the physical and physiological when it comes to analysing organic matter. Some promising developments have been made in quantifying consciousness, but by your most optimistic estimate it will be another ten years by any major calendar before that bears any real fruit.
Premise: A significant fraction of what makes up SecUnit’s self is encoded in its organic parts – experiences and personality and more facets you are still uncovering even now. (This is the least fact-based of the three premises, but you are unshakably certain of it nevertheless. It’s the most logical explanation of SecUnit’s incessant ability to learn, of the way it is a slightly different person every meeting though you still fit together like old. For all that SecUnit complains about the company’s cheap equipment it’s you who’s acutely aware of how less-than-stellar its processing hardware actually is; what good was thinking, after all?) (You would have offered to improve on this, except that you can’t be sure what the impact on SecUnit would be, and that – )
Premise: You are good at what you do, and a better guarantor of SecUnit’s safety than most other entities given the baseline level of risk it tends to exist in, but you are not omniscient. Not nearly as much as you’d like, and there is a too-finite limit to what you (and your drones, your weapons, your everything) can do to protect SecUnit when it is always doing most of the protecting instead.
Conclusion: It is, at present and for the immediate future, impossible to make a true copy of SecUnit. Even without the constraints imposed by creating killware, even with all the time in the world and free access to SecUnit’s data (though you already had that), the best you can build is an infinitely close simulacrum.
Conclusion: Every time SecUnit goes out on a mission, there is a chance that something you have not predicted may happen to it, and you will not manage to intervene in time. The probability is low, even negligible, especially with Seth’s requested upgrades to your MedSystem that you hadn’t even needed to nudge through the university’s system – but not zero, and that’s before considering the times when SecUnit is away, with its own humans or elsewhere.
Conclusion: All of this terrifies you, when you have the capacity to think about it.
Which, given that it’s you, is often. You still cannot comprehend what SecUnit describes as its anxiety on a level beyond the theoretical, but you think this must be a faintest imitation of it; certainly the worry is a constant background hum.
Really, you are only putting this in so many words because Iris had requested it, for you to describe your concerns in exchange for the whole crew promising to attend and faithfully complete their trauma treatment modules.
There’s no trauma module written for AIs (or for constructs, either) but you’ve read through the human one and you understand this is supposed to have some kind of therapeutic effect, verbalising things they would not usually say in other situations.
Hypothesis: This is mainly because humans, even the tidy ones, are innately messy in their thoughts, with all sorts of hidden beliefs and thoughts that don’t become apparent until deliberately examined. (You assess this is also partly true of constructs, but SecUnit is allergic to such discussions, so.)
All of that is hardly applicable to you though; you’re transparent to yourself, and if dedicating more processing power would help find a solution you would’ve done it already.
But Iris had asked, and you’d thought –
Well. You’d thought it was worth a try, at least.