Hauser Final Project

Entitled ‘Neosentience: Sensitivity, Persistence, and Insight’

SeamanFinalPaper  < — PDF, 1.1MB


Mirror as Neosentient

Click here to see large PDF: _alexandriaJarvis-mirror-as-Neosentient

As technology becomes an extension of daily life the human body develops a new phenomenology through which it senses and interprets the world. Each technologically mediated experience enables the human an altered or enhanced set of rules and functions.
As a technology and it’s scenario of use is imparted or discovered within a community, a new social order develops around it as it becomes accessible and acceptable. In the new social order, this new way of being begins to govern or even subsume daily habits and interactions. The shift ushers in a reliance on technology in which community members experience a fragmented reality. Ironically, this fragmentation occurs both while these individuals are united to and separated from this relationship.

This new phenomenology, created by playing off of our old is thought to be the key to recreating a kind of intelligence which will exceed our own. Whether it is artificial, ambient, near or far one can only imagine it would develop it’s own phenomenology. How will we relate to this new sentient being? How will it “see” and understand the world? What will it do with what it “see’s” and senses? Lastly, when this neo-sentient is no longer just a tool, offering the human a new phenomenology, what would happen when we realize it has it’s own phenomenology?


In a short exploration “Neosentient as a Mirror” I chose the most overt visualizations and alterations I could relate to the notion of an altered sense of the world and self: alterations of the representative body. Also, I was interested in the experience and affordances of communication through technology which led me to consider issues of self awareness through technology. After thinking about our relationship to ourselves, technology, and others I chose my agent: a mirror, to house the neosentient-human relationship. The mirror is a simple technology which we have built up both an expectation and mythical mystery about. The scenario I filmed is an exploration of the moment in which a human stumbles upon a neosentient which has recently discovered self-awareness. It’s alterations to the human form are minimal and subtle at best. However, beyond this scenario one could imagine it’s interventions being both expansive, bold, information rich, and applied to a specific context of use. The motivation of the neosentient in this scene is undetermined, but could be explored in further iterations. The schema of interaction could also be explored but in this case the mirror lends us to consideration of the analogy of the neosentient as mirror. Also, the communication between the neosentient and the human is at best strained in this scenario, but one could you imagine a willing and curious observer? All in all, this study has lead me to come into a critical awareness and imagination of my relationship to technology. And from this exploration, I find I am once again delighted by my “old” phenomenology and perhaps less miffed (or more thankful) of the new one I have gained!

Neosentience as Sensitive Persistence

Link to Live Presentation

can’t get the embed to work: sorry

Spatial Temporal Biological Boundary Gradation and Senescence

Many advances in the realm of artificial intelligence are now occurring. A new and related branch of investigation building upon these studies is the robotic paradigm of neosentience, or an intelligent, embodied, multimodal sensing and computational robotic system (Seaman and Rossler 2008). With the ability to learn, intelligently navigate, interact through natural language, generate simulations of behavior, create, display mirror competence, and gain contextual knowledge through multimodal sensing, the neosentient exhibits the mechanistic triumphs of the human body. The human, however, is rapidly shifting in how it interacts with the external environment. Organic, compartmentalized, and discrete interactions are being replaced with inorganic, nebulous, and gradation-based relationships. With rapid changes occurring in the way the humans relate to their environment, the neosentient, and its potential for multimodal sensing and intelligent interaction with other entities, may be defined both spatially and temporally.

There have been numerous models proposed to define an entity as living or non-living. An integral portion of many of these theories is defining the boundary between the external environment and the actual living organism. The traditional and biological view of living things takes root in compartmentalization. A discrete boundary exists between the organism of interest and the environment. Amongst program, improvisation, energy, regeneration, adaptability, and seclusion, Koshland has classified compartmentalization as a key component of a living entity (Koshland 2002). This requirement of sequestering functional parts has been defined as crucial for organisms on the terrestrial (McKay 2004) as well as extraterrestrial level (Ricardo 2009). Additionally, life has been strongly classified as being based in organic materials. The atoms of oxygen, nitrogen, hydrogen, and carbon have been identified as the fundamental building blocks for life. With the exception of a few metal ions, organic molecules have predominantly been associated with living systems (Shapiro 2007).

With recent advances in the fields of biomedical engineering, neuroscience, materials science, and computer science, these traditional definitions of life are being challenged. In particular, there has been a significant progress made expanding the interface of the external environment and the organism. New theories of what defines a part of a living organism are trending away from absolute boundaries and compartmentalization. In the realm of human health, numerous advances in prosthetics for damaged body parts are already being implemented. Cochlear implants, mechanical devices implanted within the inner ear, are being utilized to augment hearing and provide sensation back to those who have damaged their sensory organs. Artificial limbs and prosthetics, far more functional than ever before, are being given to patients for amputated appendages. These devices often restore not only the ability to engage in normal daily activities, but also provide them with the skill and prestige to participate in sports—displays of heightened and refined physical ability. Additionally, transplants for organs from donors of not singularly human origin, are being performed daily. The completion of the Human Biome Project has closely linked the microflora of the intestinal tract to human health. It has now become clear that there is an intricate and intimate linkage between the human genome and intestinal microbiome. Collectively, microorganisms and the human make up the human metagenome, revolutionizing medical intervention and care (Hattori and Taylor 2009). The boundary between the living and the environment is becoming increasingly nebulous.

Furthermore, researchers are looking into the use of inorganic and other “new” macromolecules. These new collections of atoms could be used to synthesize life and recreate the formation of the first cells. Silicon in particular has showed promise. The element has the same valence electron pattern as carbon, the “backbone” of most life on earth and has the potential to show similar bonding properties. Many physicians are looking towards utilizing artificial parts for transplantation. Widespread use of xenogenic transplantation utilization, fetal brain cell transplantation, and transplantation of isolated cells proved wrong. Research for the twenty first century mostly likely will consist of hybrid and entirely artificial, implantable devices (Rowinski 2007).

From these changing dynamics, I will argue that we must view the body in both space and time. This new view will have important implications subsequently for the neosentient. In reflecting upon the importance of spatial location for the neosentient, I will draw upon the embryological model of sequential induction. To further emphasize that the environment is only not influential at an infinitely large distance away, I will draw upon the physics theories of gravitational and electrical potential energy. In examining the role of time and the environment in relation to the neosentient, I will examine aging and degradation of mechanical and chemical elements.

I will next look at the implications of this new viewing of the human and neosentient form for the established definition of the neosentient. Sensing and application potentials, interactions with other neosentients and humans, senescence, and senility for the neosentient will all be explored. Subsequently, I will reflect upon how our own definitions of sentience and senility may change. Senescence and aging may trend towards having an inorganic rather than organic basis. Products of material science research are being implemented for sports health and medicine. Health and the process of aging may be limited more by materials than by biological elements.

From this analysis, I will reflect upon the important questions this new definition of boundary will address. In particular, the value of aging and death for the neosentient is worth exploring as the boundary between living and non-living is blurred.

Neosentient: detecting a relaying a visual representation of self and others

Technology as an extension…
Technology becomes an extension by which we experience the world. A plane allows a pilot to experience flight, though he himself is not flying. A telescope gives a sailor a magnified vision of the shore, something which is imperceptible at our current vantage point. A camera allows a person to “see” a loved one through video chat and suddenly they become telepresent. We experience the world through our body, but more and more we rely on technology which transforms it into an extension of our body. In the example of the video chat, the camera physically exists within the space in the place of our eyes. The camera becomes our eyes. The technology detects that experience in some ways but does not have the same level of sentience as we do. The camera technology is detecting particular variables of that reality, such as image and audio, and relaying a representation to us. I am interested in creating an experience in which people come into a critical awareness of this phenomenon of extended experience through technology with by altering the representations which they expect to have.

I am interested in studying each element within this relationship
and each transfer of information which happens (arrows)…

In relation to Neo-sentience….
When thinking about technology as extensions Bill Seaman asked the question, if that technological object had a phenomenology in the way that we do, what would that be like? The neo-sentient is that the technology might have not only aspects of the sentience we have but also new aspects of perception and understanding about the environment.
It is interesting to think about this idea of the phenomenology of a neosentient in reference to the diagram above. The neosentient would not just be detecting and relaying a representation to us but also is aware of their own representation and perception of reality. It could detect one thing and relay something else. (We often feel this way with humans.) It could relay a representation that was endowed with additional pertinent information.

In review of all these things here are some questions:
What if the technology could be sentient of the experience it is relaying to us? If it could have an enhanced way of experiencing that moment more similar to the way we experience how could that change our relationship with both technology and the represented reality? What could be the phenomenology of that camera, telescope, or plane which is the intermediary actually creating the vision, clarity, or flight.

Representations of self // Developing the final project….
A mirror is a simple kind of technology which yields a representation of ourselves and has a high degree of expectation from previous interactions. It lends nicely to a comparison of the way in which we create digital representations of ourselves. It is also an object which represents self-consciousness and self-image.

I would like to create a mirror which replaces or alters the reflection you expect to see.
I am using the mirror as a way to think about issues of how we create and rely on digital representations of ourselves and others through digital technology and the internet.

Below: Simplified diagram of the transfer between the self and the digital representation of self and others. (eg: Could be thought of as a diagram of social media)

Below: We expect a mirror to supply a representation (reflection) of ourselves.

However what if this reflection/representation was partly altered or replaced.

  • watching a mirror means watching yourself, looking at yourself
  • moments of lag of in reflection
  • long periods of lag
  • a mirror gives an image of what is in front of it, but what if it wasn’t in front of it
  • a mirror which gives a reflection of the back of your body not the front
  • glitches in image (like pixels)
  • glitches in image: broken image (mirror)
  • image of someone else
  • replaced body part (technology as extensions of body)
  • enhanced body part (fun house)
  • What is the memory of the mirror? What data does it have?
  • What is the phenomenology of the mirror?
  • Could a mirror also be looking at you? Not just you looking at it
  • parallels self-consciousness (watching self) and self-image
  • People don’t look at mirror but past the mirror right? to see themselves (maybe a mirror as its own object has power in that way, if it had it’s own will, because it could provide a false sense of reality)
  • a mirror which matches your emotion
  • a mirror which presents a version of the current you and the previous digital representations of yourself
  • the mirror as a surface: a mirror which has images just below the surface
  • a mirror which alters the reflection of you and another person, switches them in some way?
  • Think about times that you see a mirror in which there is an illusion which doesn’t exist such as in a store where it looks like the room s extended.
  • what if you were looking in the mirror and whatever you fixated on would morph
  • perhaps the image of yourself shows up somewhere else instead of in front of you, like on the ceiling
  • think about how we rely on the representation in the mirror, ie- seeing what we look like to understand who we are/how our physical appearance is

Stories of process….
Previously I was interested in looking at tangible interactions and embodied experience as a way to contrast with the cognitive and disembodied experiences of technology. I was still interested in experience and digital versions of ourselves, making sense of the world through extensions of technology. I was a little confused as to how to think about this relationship and was having problems approaching it through this way of focusing on the differences between the embodied experience I have in the physical world and disembodied experience I have when I am engaging with technology, such as the computer.

Ranulf Glanville suggested I reflect on situations in which I was having an extended experience through technology but instead of feeling disembodied perhaps I it felt as if it was embodied or as if it was transcendent of embodiment. I identified and actually noticed two particular experiences which happened during skype chats with a loved one. This lead me to think about technology as an extension of the body as well as technology relaying representations of ourselves and another person to us.

Below are personal notations of these experiences…

  1. experience 1: I had a video chat in which I experienced a sense of helplessness in the telepresent moment however I realized it was exactly similar to that which would be happening if I was actually present. In this moment of helplessness all I had was the opportunity to be aware of my feelings and empathetically extending them to my loved one whom I was video chatting with. I caught myself looking at this person and it created a self-awareness of this action. It was an odd experience.
  2. experience 2 video chat: extension of body, I had an experience where I was video chatting with a loved one and they left the room for an extended time. I was busy on the phone while they left the video chat and was not aware, upon returning to the chat screen, that 1.the camera was still on and 2. that the person had truly left. This was an odd experience as I realized there was a hyper awareness in which it felt as if I was in fact in the room. My imagination ran wild and I imagined a situation in which I saw something such as a ghost or an intruder in the apartment. This thought was interesting because it operated off of my expectation to see something in the other side of the frame. Also, it made me aware that in the case of the intruder I would experience vulnerability, though I wouldn’t be physically vulnerable. The technology was an extension by which because of my expectation there was a hyper awareness of the room.
  3. accidental pocket dials This previous experience made me think of when someone pocket dials you and for the first 30 seconds of the message you are listening very closely because you are confused and expecting a voice but get background noise or lots of voices instead. I liked the idea of thinking of the phone as translating voice messages as representations of people/places/times. I also liked how in listening to a pocket dial, the way you listen to the information you are receiving changes as you realize it is not a typical message.
  4. surveillance cameras (connection to 2) I also thought about surveillance cameras and looked at an artists project to surveil her love life. This was through the Exposed exhibit at Tate Modern. She noted that when she went on dates the awareness of the camera created a hyper attention in the beginning but wore off as time went on. I also thought of an experience I had in which I was watching a surveillance output at a grocery store elevator. The elevators were in front of me and I was watching people on other floors.
  5. expectations and representations I thought about a bed which has a form of a person from the blankets and we think it is a person, or a tube which we mistake as a snake. I also like the combination of surveillance and representative physical form in this work I saw by Ryan Roe, called hang in:

A History of Computer Chess

Understanding the history of computer chess as a field for measuring the intelligence of machines (and humans) is crucial to developing Neosentient Games. At first, citing the Mechanical Turk as a starting point for this history seems like a misnomer. The Mechanical Turk exhibits no autonomy and, in fact, isn’t strictly a computer. However, since the Neosentient will be invented through anthromorphic introspection, perhaps the idea of an augmented human (or augmented computer) is a good place to start developing Neosentient Games. After loosing to Big Blue in 1997, Gary Kasparov invented a chess variant for augmented play. Many chess masters create variants based on their intimate knowledge of the sport, adding or omitting pieces and extending the board, but Kasparov’s version of chess incorporated focused on augmenting the original game of chess by giving each human player a mechanical team mate. In Kasparov’s Augmented Chess, each human player could query their computer assistant for information. Is this the beginning of Neosentient Games? How can we develop this field towards the notion of the Neosentient?

Works for other classes

KO Final Project
Essentially, for Greenberg’s class, I’ll be examining the production of assertoric knowledge in the fields of academia, economics, and law (or possibly medicine.  three fields in total).  I’ll examine each to see how humans make and justify statements, and how others then evaluate and react to these statements.  I’ll aim to arrive at a Minimal Common Use Case from this work suitable for development (i.e. following UML standards).  This MCUC may be developed in time to meaningfully use it in the NLP project.
NLP Final Project
For Haas’ class, I’d like to complete a research proposal to develop a Requirements Specification for a data model that will natively handle assertoric statements in a user-centric way.  The basic research question is, “How can we represent assertoric knowledge in a structured, reliable, machine-parseable way?”
The Need
Assertions are a large part of human language, but are difficult to describe in a formal way.   Assertions are statements that assign an element of truth to a statement.  And beyond the complexity of assertions, there are myriad ways humans respond to encountered assertions:
  • Doubt the assertion
  • Hold it certain
  • Accept it provisionally
  • Distrust the asserter
  • Evaluate how the assertion might be tested
  • Inquire what else the asserter has asserted
  • Inquire what others have asserted about the same topic.
  • and so on
These reactions themselves are often expressed as assertions.
For instance, let’s say a stock analyst predicts that GDP will grow 1.2% this quarter (this is an easy case since verification is well-defined).  We’d plausibly want to know what the consensus estimate was, and what this asserter’s track record with this type of estimates is.  We’d also want to make sure all agree at what’s being measured, the level of statistical or human error involved, etc.  Most of these things are ignored for simplicity’s sake, but occasionally an analyst will insinuate, for instance, that jobs numbers were inflated for political gain.
A secondary need related to assertion parsing is a categorization of the uncertainties and validation methods attendant to an assertion’s components.  For instance, the statements “It’s sunny outside today” and “it was sunny outside yesterday” have insidiously different attendant uncertainties and methods of validation.
Current Systems
Current systems may allow these types of functionalities in a round-about way.  But, as the verification criteria for assertions get more vague (e.g. “George W. Bush’s childhood in Texas had a large impact upon his presidency…”), the native support for this linguistic act diminishes.
A functioning system would allow for enhanced machine processing of natural language by allowing a parsing of text more closely aligning with the way language is actually used.  It will, of course, offer manual markup capabilities far before this.  In this way, it can be viewed as an extension of markup techniques like part-of-speech tagging: initially, tagging will be manual, but the existence of the framework will enable machine learning and, eventually, processing of assertions.
I’ve developed a prototype framework (attached) for assertoric content.  This framework, combined with a taxonomy of uncertainties and methods of validations, will be central to an eventual model.  Additionally, the model will need an inherently perspectival architecture. That is, the beliefs formed upon encountering an assertion will need to be relative to a specific agent.  This will replicate the perspectival nature of human experience, allow the modeling of the assertions and beliefs described above, and allow for machines to operate as assertoric agents.
Broad Impact
The ultimate realization of this technology combined with advances in AI would allow systems to believe, disbelieve, and conditionally accept assertive statements by others, due to its support for perspectival representations of facts.  Breazeal and others have talked about the benefits of social-emotional intelligence in robotic systems; endowing such systems with assertoric agency is another highly important step in the anthropomorphization of computational systems.