AR via Hannah Schraffenberger and Edwin van der Heide

This blog pos surveys two papers, From coexistence to interaction: influences between the virtual and the real in augmented reality (2013) and Everything Augmented: On the Real in Augmented Reality (2014), written by media researchers Hanna Schraffenberger and Edwin van der Heide.

Hanna Schraffenberger is a PhD student and researcher at the Media Technology Research Group at Leiden University. Schraffenberger research interests focus on interactivity, interactive art and Augmented Reality and its applications in the arts. Edwin van der Heide is an artist and researcher of sound, space and interaction. Heide is also is a part-time Assistant Professor at Leiden University (Media Technology MSc programme) and heading the Spatial Interaction Lab at the ArtScience Interfaculty of the Royal Conservatoire and Royal Academy of Art in the Hague. Heide is interested in the terms "composition" and "musical language" in reference to the spatial, the interactive and the interdisciplinary . Additonaly, Heide's artistic practice focuses on installations, performances and environments, with particular interest in audience-work explorations, interactions and relationships.

From coexistence to interaction: influences between the virtual and the real in augmented reality, 2013, https://ses.library.usyd.edu.au/handle/2123/9643

Media technology researchers Hannah Schraffenberger and Edwin van der Heide are interested the concept of possible symbiotic relationship between what they call “real” and “virtual” objects. The pair utilises augmented reality (AR) which allows virtual or digital content to be added “into” our environment. One to which we accept as being real. To them, the connection of the virtual and real not only creates a sense of co-existence, but influence and interactivity. Specifically, how the real affects the behaviour of the virtual and the virtual affects the behaviour of the real. Schraffenberger and van der Heide use imaginary scenarios such as a virtual ball interacting with real dominos, allowing the audience to experience influence between objects that cannot exist in a purely physical world. A ball can exist within the “real” physical world, but a virtual ball cannot.

To Schraffenberger and Van der Heid, the virtual is synonymous with being free from physical laws allowing new forms of interaction to be possible. And the virtual cannot directly apply forces to real objects, meaning physical world interaction might not be possible within the virtual. These two theories result in questions such as ...what types of interaction between the virtual and the real are possible? When/how can the real affect the virtual? When/how can the virtual affect the real? What makes the interaction believable? Which problems do arise? And most importantly to my research… what possibilities emerge? To which they explore in their research with the a bouncing virtual ball and dominos.

In this simulation, Schraffenberger and Van der Heid use a virtually created ball which bounces on a surface. Utilising max/Msp/ jitter software and a webcam, the ball is integrated in a view of a real world environment. This real world environment has also been rebuilt with appropriate physical qualities such a mass and restitution in the virtual to allow the ball to react with the real environments counterparts. The outcome of this test illustrates that it is possible to simulate certain existing real-world interactions in AR, but also only shows that the real can influence the virtual, as the virtual ball has no effect on the real table. In the second stage of their inquiry, Schraffenberger and Van der Heid simulated the virtual ball to collide with real dominos. Wherein the ball hits the first domino and deflects back to where it came from: The virtual reacts to the real. Yet, the dominos do not fall upon impact. What is apparent between the two examples is that virtual environments can invisibly overlayed to create simulated object collisions between “real” and “virtual” objects to affect the virtual. But if the “real” object should be affect - in this case, by physics, the dominos should fall. The “real” dominos cannot fall in the real, and their simulated fall in the virtual is irrelevant as the “real” remains standing.

Hybrid objects, which have a virtual and physical presence, are one way the virtual can affect the real. Continuing with the ball and domino scenario, this means it completed in pure physicality first. A real ball heads towards real dominos and hits the first starting the expected chain reaction fall. The virtual sphere, now has a physical counterpart to which it follows. And when viewing via the screen the virtual replaces the physical. Essentially, a virtual object needs a physical object to copy to simulate a the scenario of a real object being physically affected by a virtual. Alternatively, objects that have their own mechanics or electronics can be simulated to appear to be controlled or manipulated by virtual objects without the need for hybrid objects.

It is important to note that interactivity between the virtual and the real isn’t reserved to visual forms. Another indication of physical interaction or collision is sound. And so, simulated noises are created fro the impact of the virtual ball bouncing on the real table, derived from the magnitue of real on real collision, the virtual noise noise is scaled accordingly.The preceding exploration has shown that the virtual and the real can influence each other and that real-world interactions can be simulated in AR.

As mentioned earlier, the virtual and the real have different constraints they are expected to abide to. Namely the real is expected to obey physical laws, while the virtual is not. Schraffenberger and Van der Heid believe the virtual object is then not tied to realistic behaviours but is still tied to “believability” or reserved for being physical counterparts. This is because object believability isn’t tied to the notion of reality as we accept it, more so the reality the object is derived from - its own epoch. In fact, imagined objects with their own behaviours are accepted in AR as they are in literature and cinema: expectation of reality and disbelief are suspended temporarily and are superseded by imagination and speculation . “Presumably, if the virtual represents a real object and behaves like a real object, we expect it to affect the real environment just like a real object. However, we assume that if the virtual does not appear and behave like a real world object, we do not have such expectations. “

------

"Everything Augmented: On the Real in Augmented Reality" in Journal of Science and Technology of the Arts, December 2014,https://www.researchgate.net/publication/280573296_Everything_Augmented_On_the_Real_in_Augmented_Reality

"AR can be understood as an environment in which virtual and real elements appear to coexist."

A concept that has been attributed to "virtuality continuum" derived from Paul Milgram and Fumio Kishino’s A Taxonomy of Mixed Reality Visual Displays (1994). Milgram and Kishino’s continuum ranges from purely virtual environments to entirely real environments.

"AR is placed within this continuum and describes an otherwise real environment that is augmented by virtual objects.Likewise, Azuma’s (1997, p. 356) widespread survey of Augmented Reality summarizes AR as a field that “allows the user to see the real world, with virtual objects superimposed upon or composited with the real world.”While researches tend to agree that in AR, virtual content appears to exist in a physical environment (or the so-called ‘real world’), there is surprisingly little consensus on what is actually augmented by this virtual content. In this paper, we address and explore the conceptual characteristics and possibilities of AR."

" WHAT IS AUGMENTED IN AR? The term itself – Augmented Reality – indicates that reality is augmented. However, Hugues, Fuchs and Nannipieri (2011, p. 2) argue that this is not the case: “If reality is by definition everything that exists, then strictly speaking reality cannot be augmented since it is already everything. So what is augmented?” In existing AR literature, we can find different views on the matter. Many argue that it is not reality but the perception of reality that is augmented. For example, Normand et al. (2012, p. 1) point out: “Reality can not be increased but its perceptions can. We will however keep the term ‘Augmented Reality’ even if we understand it as an ‘increased perception of reality’.” Similarly, Ross (2005, p. 32) refers to AR as that “what should be called augmented perception of time and space.” Also the widespread survey of AR by Azuma (1997) claims that AR enhances a user’s perception of and interaction with the real world. Hugues et al. explicitly address the question as part of their AR taxonomy and distinguish between AR environments that augment the perception of reality and environments that aim at immersing users in an artificial environment.

The concept of being real or not is something I often struggle with but I think the most easy way to combat this is the idea of perception.... things we accept as being real.. things we accept as being figments of the real ... imagination, idea, concepts... they are not fake because they are immaterial.. by the experiential quality of "realness" or physical perceptions of realness... or with things we accept as being real help aid things we perceive as being unreal....or atleast open the possibility of them being real

Furthermore, there is the notion that in AR, our real physical environment is augmented. This has for example been stated by Milgram and Kishino (1994, p. 1322): “As an operational definition of Augmented Reality, we take the term to refer to any case in which an otherwise real environment is ‘augmented’ by means of virtual (computer graphic) objects [...]”. (Unfortunately, the authors are not completely consistent and also refer to the augmentation of the display of an otherwise real environment.) "

This notion of augmenting an environment/space via the inclusion of virtual objects is something i'm interested in, in reference to my studio practice. How augmenting an environment not question the the perception of reality of it, but subsequently the objects too.

"Besides the idea of an augmented environment, we also find the notion of augmented space. The media theorist Manovich (2006, p.219) introduces this more general concept and describes it as “physical space overlaid with dynamically changing information, multimedia in form and localized for each user”. Manovich lists AR as one of the technologies that already create such augmented spaces."

I think this idea of localisation is really important. That the virtual feels close to the perception of the real - as being real. It also insinuates this idea of focus

"In addition, there is the conception of an augmented physical world. This idea is expressed by Craig (2013, p. 16) in his book “Understanding Augmented Reality”. Here, he places AR in the context of the human desire to alter and make adornments to the physical world. He lists several key aspects of Augmented Reality, among which the view that “[t]he physical world is augmented by digital information superimposed on a view of the physical world” (italics in original)."

I'm also interested in this idea of human desire to adorn, to add, to alter. How virtual additions can increase the multiplicity and variety of objects in our world. How their presence can make comment or reflection, or more simply offer something different, without physical impact.

"Looking at Wikipedia’s current definition of AR (“Augmented reality”), we again find a different opinion on what is augmented in AR. As of October 7, 2014, AR is described as “a live, copy, view of a physical, real-world environment whose elements are augmented [...]” (italics added by the authors).Mackay (1996) approaches the topic in yet another way. The author considers the carrier of the physical equipment as augmented (e.g., the user is augmented when he/she carries a helmet and an object is augmented when sensors are embedded in it) and consequently distinguishes between an augmentation of the user, an augmentation of the physical object and an augmentation of the environment surrounding the user/object."

The idea that augmentation its three-fold. It is not just that we experience augmentation through the screen/device.. but that we actively augment ourselves in the process, and that the technology itself is augmented. This kind of inter-related augmentation - that for the corporeal references idea of the cyborg. The idea of extension for all participants in the encounter - human or non-human.

" AUGMENTATION As the reviewed literature illustrates, there is little consensus on what is actually augmented in AR. One potential reason for this is the different underlying understandings of what it means to augment something. Does augmentation refer to the mere addition of virtual content to our (view of the) physical world, does it imply an improvement or enhancement (for example of the world or of our senses) or is it something else entirely? (...) "

"TWO TAKES ON AUGMENTATION One problem with the use and meaning of the term ‘augmentation’ is that it can refer to two different processes. Firstly, augmentation can refer to the addition or integration of virtual content (in)to physical space. In this context, augmentation refers to the creation of Augmented Reality, or, in other words, to the external process that turns something ‘unaugmented’ into something augmented. Secondly, it can refer to what the virtual does to the real. Here, augmentation is used in the sense that the virtual augments the real (and, as we will argue, vice versa). This time, augmentation is an internal process. It refers to what happens in AR and concerns thephenomenon Augmented Reality. Both uses of the term are equally legitimate and both sides of augmentation are crucial to AR. However, if we look at existing AR research, much focus is placed on augmentation in the former sense and addressing how to create AR or how to add the virtual to the real. For example, there is plenty of research into technologies and techniques that enable or support the integration of virtual objects in our view of the real physical world, such as tracking or calibration techniques (cf. Zhou, Duh, and Billinghurst, 2008). Notably, this former understanding of augmentation does not match our experience of AR. When experiencing AR, we ideally do not experience the addition or integration of virtual content but its result: the apparent presence of this virtual content in the real space, and, as we will show later, the spatial and content-based relationships between this virtual content and its real surroundings. (Of course, in many existing AR implementations we are made aware of the fact that the virtual is added to our view because of technical imperfections such as timing delays or spatial alignment problems. However, in an ideal scenario, the virtual would simply appear to be there.) In other words, we experience a hybrid virtual-real environment. (...) "

The idea of perfect or seamless integration is not something im particularly interested in. I'm not trying to convince the audience that the objects are real - im more interested in the idea thst they could be. That there is a possibility. Playing with glitches, scale and spatial alignments are all elements that help not only with ambiguity and the idea of speculation but also a rejection of, or playing with, the ideals or accepted notion of reality and object qualities. That just because these things are not as we are use to, doesn't mean that they are fake or unreal.

" UNAUGMENTED VERSUS AUGMENTED REALITY AR requires virtual content that can be perceived alongside with the real environment. However, this is not sufficient for AR. Let us consider a text message, website or advertisement that is part of our view of the world (as, for example, possible with a head-mounted display). Even though virtual content is presented together with (or on top of) the real world, we would not consider this AR. Similarly, if we turn on the radio, we might hear a newsreader speaking, who is not really present in our physical environment (Schraffenberger and van der Heide, 2013a). Just like in typical AR scenarios, we perceive something that is not really there. Nonetheless, we do not call this AR. Why not? The answer becomes clear if we slightly modify the examples. Imagine that the newsreader appears to be sitting at your table or that his voice seems to originate from right behind you. This certainly can be considered AR. What has changed? This time, there is a spatial relationship between the newsreader and the environment: he is part of the environment; he appears present in the space. Spatial relationships are thus decisive for AR. Another possible AR scenario is that the newsreader says something that relates to your environment. He might, for example, comment on your breakfast, refer to the color of your shirt or otherwise relate to you, your environment or something in your surroundings. This time, the newsreader relates to the environment content-wise. (And as a consequence, the newsreader might again feel present to some degree). In this way, AR can be a result of content-based relationships as well. The same principals apply in the case of visually overlaid information. If we replace the random text- message, website or advertisement in our view by a message that informs us about the historic background of the building in front of us, we are dealing with an AR scenario where the virtual relates to the real content-wise. Spatial relationships are likewise possible: advertisement figures might appear as if they were part of the real environment and walk on the real street (most probably they lead the way to their associated shops).

Can the object relate with or interact with the real environment and not conform to it?

" WHAT THE VIRTUAL AUGMENTS Relationships between the virtual and the real distinguish AR scenarios from other cases where virtual content coexists on an independent layer. Possible are spatial relationships between the virtual and the real as well as content-based relationships. As discussed in depth in our previous research (Schraffenberger and van der Heide, 2013a), we believe that augmentation is the result of such relationships between the virtual and the real. [2] On the basis of this, we can present a preliminary answer to the question of what is augmented in AR: The virtual augments that, to which it relates. However, this is only one half of the story. The virtual not only relates to the real, the real also relates to the virtual. The spatial and/or content-based relationships are between them. Rather than claiming that something is augmented in AR, it would thus be appropriate to say that there is a real component and a virtual component to AR. Their relationship constitutes the augmentation. Unfortunately, this view is conflicting with the language associated with AR. Even the term “Augmented Reality” implies that something (reality) is augmented rather than a relationship between two components. (...)"

" AUGMENTED ENVIRONMENTS In an augmented environment, there is a relationship between virtual content and its real surroundings. As pointed out, this relationship can be spatial and/or content-based. A spatial relationship is common in cases where virtual visual objects are integrated in a real 3D space. When, for example, a virtual chair is added to a real desk (cf. Azuma 1997) there is a spatial relationship between the real environment and the virtual chair: the chair is part of/integrated in the real environment. (...) Besides visual augmentations, we can also find various examples of sound-based augmented environments: Cilia Erens’ sound walks are designed for a certain walking route and mainly use unprocessed binaural recordings of everyday-sounds (Erens; cf. Schraffenberger and van der Heide, 2013a) [1]. When the participant navigates the environment and listens to the composition on headphones, the recorded sounds merge with the sounds present in the existing environment and invite the participant to make connections between the added sound and his or her surroundings. (..) " "Augmented Environments generally offer the participant or observer to walk around, navigate and explore the environment. A change in the observer’s position usually results in a change of what is/can be perceived. In the case of content-based augmentations, the presented content adapts to what is present in current environment and, for example, allows the participant to learn more about his or her current surroundings. If one moves around in spatially augmented environments, virtual objects might be seen from different perspectives or soft sounds might get louder if one walks into the direction of their origin."

"4.2 AUGMENTED OBJECTS The fact that virtual elements (appear to) exist in a real environment does not necessarily mean that the virtual content also augments this environment. There are cases where the virtual relates to, or becomes part of, a particular physical object rather than the general environment. One example of an augmented object is the augmented zebrafish by Gómez-Maureira and Teunisse (Gómez-Maureira, Teunisse, Schraffenberger, and Verbeek, 2014). In this project, the zebrafish’s skin is projected on a physical bigger- than-life zebrafish (see Figure 2). The audience can look inside the fish and reveal additional information (for instance, an X-ray visualization and a basic anatomical schematic) by stepping in front of the projector and moving their shadow over the fish’s surface. This is realized with a kinect sensor and a secondary projector. The kinect detects the shadows and the secondary projector fills in the shadows with the additional content. Here the virtual content primarily relates to (and becomes part of) the fish, rather than to the general surrounding space. Both components, the virtual and the real, are designed in a way that deliberately leaves out certain characteristics. These ‘missing’ aspects are filled in by the other component, resulting in one hybrid virtual- real model (cf. Schraffenberger and van der Heide, 2013a).

Mixed virtual-real objects are also used in the field of Augmented Prototyping. Here, too, digital images are projected on physical models, resulting in a tangible prototype to which, for example, different material and lighting conditions can be applied (Verlinden, De Smit, Peeters, and van Gelderen, 2003; cf. Schraffenberger and van der Heide, 2013a). As the augmented objects in turn are part of their environment, the virtual also – to some degree – relates to this environment. Due to this, a clear distinction between augmented objects and augmented environments is not always possible. This shows for example in Pablo Valbuena’s site-specific intervention N 520437 E 041900 [the hague city hall] (Valbuena, 2008). In this work, the The Hague city hall serves as a canvas for projections that appear to dynamically transform the architecture of the building. However, as this building is an integral part of its surrounding space, the immediate environment is also affected by the intervention. The distinction between augmented environments and augmented objects becomes clearer when we consider the ‘user’ or the participant/audience. Whereas viewers are part of the environment, they are usually not part of an augmented object. Where environments usually invite the audience to navigate them, augmented objects often facilitate interaction with the objects. AUGMENTED HUMANS In the same way that the virtual can relate to real objects, it can also relate to humans (who essentially can be seen as a special kind of object).

The animate object

(...) From a technological perspective, augmented humans do not differ much from augmented objects. However, this category differs from the perspective of the participant. There are two potential roles the participant can fulfill. As usually, he or she can be an observer of the augmentation. However, additionally, the participant can also be the ‘object’ who isaugmented. Given this fundamentally different role of the participant, it makes sense to treat this as a separate conceptual category.

(...)

AUGMENTED PERCEPTION? It has been argued that AR is in fact an augmentation of our perception (e.g., Normand et al., 2012; Ross, 2005). In our understanding of AR, this is not the case because the virtual does not relate to our perception but to something that is perceived. Nonetheless, we can find examples where AR is used to extend our perception as well as examples of sensory extensions that show interesting similarities with AR. In the following we will explore the possibility of “Augmented Perception” and show how it relates to the field of Augmented Reality.

Perceiving more There are many things that we cannot perceive as humans due to the way our senses work. To name just a few examples, we cannot hear ultrasound, we cannot see in the dark and we are insensitive to magnetism. However, there are devices that help us to overcome some of those sensory limitations and that allow us to perceive things about the environment we normally cannot perceive. A well-known example of such a device is a hand- held Geiger counter. The device produces audible clicks that correspond to the amount of radiation that is present at the current location (cf. Schraffenberger and van der Heide, 2013a). We will use the term Augmented Perception when we refer to the perception of additional information that becomes part of how we perceive the space. An emerging field of research in this context is sensory augmentation. Sensory augmentation systems translate information that we normally cannot perceive into stimuli we can perceive and thereby allow humans to perceive more and ‘new’ aspects of the environment. An example of a sensory augmentation device is the vibrotactile magnetic compass belt (Nagel, Carl, Kringe, Märtin, and König, 2005). The belt is worn on the waist and indicates the direction of magnetic north with vibrations. Unfortunately, research regarding the effects of wearing the belt did not yet yield conclusive results. None of the participants in Nagel et al.’s study experienced a local magnetic field. However, two participants, after wearing the belt for a longer period of time, reportedly experienced the input from the belt as a property of the environment rather than as mere tactile stimulation. Despite the fact that this study requires follow-up research, it addresses many interesting questions, such as whether new senses can be developed and learned. With respect to AR, it is particularly interesting to distinguish between information that simply informs participants about the environment and information that is perceived as part of or as a property of the environment. [3] Information on the amount of radiation) relates to and informs us about the environment/space in a similar way as virtual content can inform us about the surrounding space in AR. Augmented Perception and AR also relate to one another in the sense that AR concepts and technology can be used to translate what is hidden from our senses into something we can perceive. In other words, AR can be used as a method for Augmented Perception. For example, in the context of visual AR, there is research into applications that make it possible to see hidden or occluded objects (e.g., Sandor, Cunningham, Dey, and Mattila, 2010). However, certainly not all AR applications aim at making the imperceptible perceivable. Essentially, both Augmented Reality and Augmented Perception allow us to perceive more. A key difference between them is that Augmented Perception aims at making something perceivable that is already there – a real but imperceptible information inherent in the environment is translated into something we can perceive. In contrast, AR applications add new additional content to the environment. Simply put, Augmented Perception allows us to perceive more. In AR, there is more to perceive.

(...)

5.1 MULTIMODAL AR If we look at common AR scenarios such as augmented environments or augmented objects, it stands out that the real often has a visual appearance. However, at the same time, the real usually is more than just ‘something visual’. For example, in a simple AR scenario, a virtual bird might appear to sit on a real tree in a real garden. This garden is something we can see. However, the garden is not merely a visual thing. It can be touched, it has a smell and if we listen to our surroundings, we might hear real birds singing. Even if the virtual bird is visually integrated in our view of the garden, it nonetheless relates to and becomes part of the garden as such – it is not just experienced as part of what we see. This means that although AR is often focused on vision, it is – just like reality – first and foremost a multimodal phenomenon. [4] "

How does Ar objects relate to they physical - sound, touch, smell - the senses. A rounded experience - modality.

"The fact that multimodality in AR is the norm rather than the exception is especially interesting in the light of existing research into multimodal AR. Here, the idea of multimodality is mainly used with respect to multimodal user-interaction. This means that a user or participant can interact with the AR scenario in a multimodal way; either by giving multimodal input such as gestures and speech, or by receiving multimodal output from an AR system such as a combination of visuals, tactile feedback and sound. These forms of multimodal AR are certainly interesting. However, if we restrict multimodal AR to those scenarios where the virtual addresses more than one sense, and cases where participants can interact with virtual content in a multimodal way, we seem to overlook the most fundamental way in which AR can be and often is multimodal – the fact that the real component in AR often stimulates more than one sensory modality. Future AR can acknowledge this and actively work with the fact that our environment engages all our senses. This can be done by relating the virtual content not just to what is seen but also to the other aspects of the environment. In our example of the virtual bird, this might mean that the bird responds to the songs of real birds, that its feathers move corresponding to real wind or that you might scare it away by making a sound. Multimodal AR is thus not just about user-interaction and it is not just about multimodal virtual content. It is likewise about sensing the real multimodal world, about listening to it, registering the temperature, the wind or even the smells present in the space and relating the virtual to these aspects.

INTERACTION The fact that we can interact with, or at least, have an influence on our real environment, also offers many possibilities for interaction in AR. One might argue that moving about real objects in an AR scenario or changing the physical aspects of the AR environment already are a basic form of interaction in AR. However, more interestingly, the fact that we can interact with or influence the real does not only allow for interaction with these real objects but likewise enables interaction with virtual objects."

An interaction with the virtual but also the real - a mediated real.

"If the virtual and the real are interrelated, one can also interact with the virtual content, simply through interacting with the real content (Schraffenberger and van der Heide, 2013a, 2013b). A basic example of this is a virtual marble on a real table that can be moved around by lifting one side of the table (causing the marble to roll to the lower side and fall off to the ground). Of course, again, there are even more possibilities if we incorporate more modalities and relate the virtual and the real not just in a visual manner. Imagine, for example, virtual creatures that are attracted by light, that change color according to the color of the objects behind them, that move faster if it is warm and that fear certain sounds. In such a case, simply interacting with our environment in a natural way will offer us a variety of possibilities to interact with the virtual content as well.

CONCLUSION AND FUTURE DIRECTIONS Little consensus exists on what is augmented in AR. In this paper, we have found that there is not just one right answer to this question. In fact, everything can be augmented. The possibilities range from augmented environments, objects, humans and (media) content to intangible, non-physical entities such as augmented stories, concepts or ideas. In the most general sense, the virtual augments that to which it relates. More importantly, the virtual and the real relate to, add to and augment one another.Regarding the fundamental characteristics of AR, we have identified spatial and content-based relationships between the virtual and the real as being decisive for AR. So far, AR research and practice have put much emphasize on spatial relationships. In the future, we can further study the possibilities of content-based relationships and, for example, investigate how the virtual can relate to thoughts, moods and feelings. We have explored the realm of existing AR works, categorized common forms of AR and illustrated them with AR (art) works. We do not claim that we have presented all possibilities of AR. Are there other – non-spatial and non-content-based – forms of augmentation? What about augmented events, processes and activities? Furthermore, we have defined Augmented Perception and discussed similarities and differences between Augmented Reality and Augmented Perception. It has become clear that AR and Augmented Perception are related fields that can contribute to each other. In the course of our investigation, we have gained many insights regarding ‘the real’ in Augmented Reality. Strikingly, the real in AR does not have to be real after all – for example, we can augment a fictional ghost that is depicted in a book.It furthermore became clear that the real provides many possibilities for interaction in AR. If there is a relationship between the virtual and the real, participants can interact with the virtual components through interacting with real objects and by influencing the real environment. Several discussed projects have shown that it can be fruitful to not just take the real for what it is but to also design, create or modify the real component in AR (for example, in a way that leaves out certain aspects to be filled in by the virtual content). With respect to the nature of AR, it became apparent that AR is about more than meets the eye. Even in cases where visual virtual content is integrated in our view of the world, AR is most commonly multimodal. This is because the real component in AR is usually multimodal."

I think the idea of multimodal is not necessarily important to my exploration as i'm abstracting the object - making it have an ambiguity - aiding in pure speculation - multiple representations - which the "real?" the most accurate? is important (Kosuth) . I'm not aiming for seamlessness - in fact the opposite - I want some glitchy - weird - ad-on , hybrid / chimer-ised nature to the objects relationship with the "real" space.

" (... )In the course of our investigation we have encountered the claim that reality cannot be augmented, since reality already is everything (Hughes et al. 2011). However, this claim is only true if we believe that AR is about the addition of some external content. In our understanding, AR is about the relationships between the virtual and the real. It is not necessary that the virtual is something apart from reality. In fact the opposite is the case: reality is augmented if the virtual becomes part of it, intertwines with it and relates to it. The fact that the virtual is part of reality hence does not make it impossible to augment reality; rather it suggests that our reality is, in fact, already augmented.

i'm interested in playing with tension between the to - this unknowing-ness - the ambiguity. I'm not interested in proclaiming reality.. just playing with its spectrum.. navgating it. It may feel inclusive to say that "reality is everything"... so nothing is unreal .... But really it seems reductive. A kind of is or is not - black of white mentality .. that isn't considered with possibility or speculation....or impossibility. The last line here is the moist poignant ...the fact we are considering an interaction that is already happening, the integration of the virtual and the real, proves its integration is possible - not that it is a whole, integrated from the beginning.

(...)

ENDNOTES [1] See, for instance, Hollands Doorzicht (Erens, 2006). This sound walk is made of sounds that were recorded in the Netherlands and took place close to the Dutch embassy in Berlin, 2006. [2] Other relationships between the virtual and real, such as interaction between virtual and real objects, are possible (Schraffenberger and van der Heide, 2013a). However, to the best of our knowledge, they are all based on underlying spatial or content-based relationships. [3] It would also be interesting to find out whether the clicks from a Geiger counter solely inform a user of the present radiation or whether the radiation is to some degree also experienced as in the environment and as distinct from the audible clicks. [4] This does not mean that all AR has to be multimodal. However, in the case of augmented environments and objects, this is usually the case. CITAR JOURNAL CITAR Journal, Volume 6, No. 1 – Special Issue: xCoAx 2014 27

REFERENCES Archer, N. (2010, February 14). Augmented Reality Tattoo . Designboom. Retrieved October 7, 2014, from http://www.designboom.com/technology/augmented- reality-tattoo/ Augmented reality. (2014, May 10). Wikipedia. Retrieved October 7, 2014, from http://en.wikipedia.org/wiki/Augmented_reality Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385. Billinghurst, M., Kato, H., & Poupyrev, I. (2001). The magicbook-moving seamlessly between reality and virtuality. Computer Graphics and Applications, IEEE, 21(3), 6-8. Craig, A. B. (2013). Understanding Augmented Reality: Concepts and Applications. Newnes. Erens, C. (n.d.). The Audible Space. Cilia Erens. Retrieved October 7, 2014, from http://www.cilia- erens.nl/cilia-erens-2/ Erens, C. (2006). Hollands Doorzicht [sound walk], see http://www.cilia-erens.nl/portfolio-2/hollands- doorzicht-berlijn-2006/?lang=en, accessed October 7, 2014. Gómez-Maureira, M. A., Teunisse, C., Schraffenberger, H., & Verbeek, F. (2014). Illuminating Shadows: Introducing Shadow Interaction in Spatial Augmented Reality. Creating the Difference, Proceedings of the Chi Sparks 2014 Conference 9 (pp. 11-18). Gradman, E. (2010). Cloud Mirror [interactive art installation], see http:// www.gradman.com/cloudmirror, accessed October 7, 2014. Harrison, C., Tan, D., & Morris, D. (2010, April). Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 453-462). van der Heide, E. (2000-). Radioscape [immersive electromagnetic environment], see http://www.evdh.net/radioscape/, accessed October 7, 2014. CITAR JOURNAL 28 van der Heide, E. , & Rekveld, J. (Curators). (2008- 2009). Wormhole Dordrecht [sound worlds in public space], see http://www.evdh.net/wormhole/, accessed October 7, 2014. van der Heide, E. (2014). Radioscape, Wormhole Dordrecht and Beyond. AR[t], Augmented Reality, Art and Technology, 5, 97-99. Hugues, O., Fuchs, P., & Nannipieri, O. (2011). New augmented reality taxonomy: Technologies and features of augmented environment. In Handbook of augmented reality (pp. 47-63). Springer New York. Mackay, W. E. (1996). Augmenting reality: A new paradigm for interacting with computers. La recherche, 284. (Accessed at http://www- ihm.lri.fr/~mackay/pdffiles/LaRecherche.English.pdf, 1-9). Manovich, L. (2006). The poetics of augmented space. Visual Communication, 5(2), 219-240. Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), 1321-1329. Nagel, S. K., Carl, C., Kringe, T., Märtin, R., & König, P. (2005). Beyond sensory substitution—learning the sixth sense. Journal of neural engineering, 2(4), R13- R26. Normand, J. M., Servières, M., & Moreau, G. (2012, March). A new typology of augmented reality applications. In Proceedings of the 3rd Augmented Human International Conference (p. 18). Ross, C. (2005). New Media Arts Hybridity: The Vases (Dis)communicants Between Art, Affective Science and AR Technology. Convergence: The International Journal of Research into New Media Technologies, 11(4), 32-42. Sandor, C., Cunningham, A., Dey, A., & Mattila, V. V. (2010, October). An augmented reality x-ray system based on visual saliency. In ISMAR 2010 (pp. 27-36). Scherrer, C., Pilet, J., Fua, P., & Lepetit, V. (2008). The haunted book. In ISMAR 2008 (pp. 163-164). Schraffenberger, H., & van der Heide, E. (2013a). Towards Novel Relationships between the Virtual and