Analysis Of An E-Learning Augmented Environment: A Semiotic Approach To Augmented Reality Applications

posted in: augmented reality, essays | 0

Analysis Of An E-Learning Augmented Environment: A Semiotic Approach To Augmented Reality Applications

Sergio Cicconi, Maurizio Marchese Dept. Information Engineering and Computer Science - University of Trento (ITALY)
12th annual International Conference of Education, Research and Innovation (ICERI) 2019, 11-13 November, 2019, Seville, Spain: ICERI2019 Proceedings, pp.4921-4931, doi:10.21125/iceri.2019.1204
In this paper, we analyze Augmented Reality (AR) in the context of a semiotic theory of new media. So, rather than analyzing AR meant as a new form of technology, we will analyze the new forms of communication which are produced with AR technology, that is, the Augmented Environments (AEs) generated by applications running on head-mounted devices created with AR technology. And we intend to use semiotics of media to conduct such analysis. In the first part of the paper, we outline the theoretical framework for our analysis. Particularly, following the suggestions coming from text semiotics, we investigate the concepts of media and text, so to be able to consider the AEs as multimedia texts. In the second part of the paper, we use semiotic theory to disassemble in many ways the AEs, so to identify all possible components of AEs and understand the role they play in creating the new complex form of experience that we call augmented reality. We believe that the semiotic approach to AEs can greatly increase the knowledge on AR and, consequently, help designers, developers, analysts, and users to better understand the new ways of perceiving the world offered by AR.
This work intends to use semiotics of media to acquire and systematize the knowledge related to Augmented Reality (AR), especially with regard to the objects developed using AR technology, that is, the applications that, once deployed in Head-Mounted Devices (HMD), are capable of generating immersive Augmented Environments (AEs). This approach to the AR is meant to be an alternative to another approach, certainly more common and widespread, that deals with AR from a technological point of view. Such technological approach often uses a technical language to describe AR that is not easily comprehensible by those people who are interested in AR but don't have a technological background. We believe that the discourse on AR cannot be limited to explaining how the technology makes it possible to create immersive AEs. Designing an AE and developing it using a certain technology are two different activities, that require different skills and different ways of looking at AR, and, therefore, that also require different people working together in collaboration. Unfortunately, we must say that the prevalence of the technical approach to AR does not favor those opportunities for collaboration. So, with this work we want to show that even a non-technical approach to AR is possible, hoping also to foster the dialogue between technologists and non-technologists.
Already in 2001, MacIntyre et al. wrote: "AR techniques will not move beyond specialized application areas, such as equipment maintenance, into areas of interest, such as entertainment and education, until we develop [the theories to understand how to design AR experiences] and the corresponding tools, and that the best way to develop these theories and tools is through tight collaboration between technologists and New Media theorists" [1, p. 197].
Our recent experience in designing and developing an augmented e-learning environment can only confirm the continuation of the situation described almost twenty years ago by MacIntyre et al.. Unfortunately, at least as far as AR is concerned, even today the collaboration between technologists and new media theorists (and designers, artists, humanists) seems rather fleeting [2], [3].
On the one hand, we still find many professional figures with solid skills in the humanities, but with limited experience with technologies, and who do not have the time or interest to acquire new knowledge coming from the technological world. These should be the professionals who decide how to design Augmented Environments (AEs), and what contents to use. However, since they do not know the full potential of AR, nor do they know what new forms of experience can be created with AR, they are not really able to define the characteristics of the AEs to be designed.
On the other hand, we have developers with a thorough understanding of the technology behind AR. They should focus on the development of the technology and the applications to be produced with the technology, without being concerned too much with the design of the AEs generated by those applications. Unfortunately, as we have just seen, it is rare for developers to have at their disposal a design for an app created by professionals with appropriate skills. Therefore, developers often develop apps that they themselves have designed. The results, in these cases, are not always satisfying. The apps are perfectly working, yet they often lack of an appealing look, or exhibit poorly designed or unclear interfaces, or deal with the contents with excessive simplification.
Our work wants to be a tool to stimulate and foster that collaboration advocated by MacIntyre et al., between two worlds that still do not talk to each other very often. With this work, we also want to provide a way to deal with AR that could help non-technologists to understand the potential of a technology -- augmented reality -- that technologists have developed.

We intend to provide some methodological suggestions to organize the semiotic analysis of augmented reality and augmented environments. We will draw on concepts from different, but somehow contiguous, research areas: theory of new media [1], [4], [5], [6], semiotics of media [7], [8], semiotic engineering [9], [10], semiotic textology [11], [12], [13], textual semiotics [14], [15].
Generally speaking, semiotics studies all cultural processes as processes of communication, and therefore deals with the underlying systems of significations permitting those processes [16, p. 8]. Semiotic analysis proposes to systematically break down the object to be examined into smaller and, at the same time, more general elements, so that they can be found both in the decomposed object and in other objects of the same type. The semiotic investigation begins by taking into account what we can call a "surface analysis", which describes the object in its materiality and exteriority, and then continues with ever deeper levels of analysis, each of which shows an ever-greater degree of abstraction and generalization. Each level of analysis is meant to disclose new features of the object under examination, and show new ways of looking at it and understanding it. In fact, the identification of different hierarchical levels of meaning, is one of the fundamental characteristics that distinguish the semiotic methodology from other methodologies typical of the human sciences [7].
We would also like to point out that there is not a unified and all-embracing semiotic methodology capable of capturing all aspects of any possible object investigated. This means that, depending on the nature of the object under investigation, on the type of analysis desired, on the depth of the analysis, and on the objectives set by the analysis, each individual analysis will identify and use the theories and methodological approaches that the analyst believes to be most appropriate and effective.
In this work, we want to introduce what we consider essential to set up a semiotic investigation on AR and on the immersive Augmented Environments for HMD generated by applications running on HMD (in short, from now on: AEs for HMD). We begin by presenting a general semiotic approach, which we consider to be the analytical ground for any semiotic investigation dealing with media, and we will then continue by showing a few specific analyses focusing on some relevant features of AEs.
Recently, we have developed an app that, once running on the HMD called Microsoft HoloLens, generates a learning AE which has been customized for older adults [17]. The learning environment displays all the most important features of AEs for HMD: it is immersive, 3-D, and fully interactive. Also, it contains several types of interfaces, and is capable of delivering educational material on digital culture. Occasionally, we will refer to such AE to provide examples on how to concretely use the theory discussed in this paper on an existing AE.
In the perspective of a semiotics of new media, before proceeding with the analysis, we examine a few fundamental concepts that we will use in our work.
2.1 Media
In ordinary language, we tend to use the word media in an ambiguous way to identify any channel of communication, from the printed text to the digital data, including radio, television, and any other form of communication. Using this uncertain notion of media, there is often confusion in identifying what we define as media. Sometimes, we might be talking about the devices that are used to transmit certain pieces of information, for example, when we use the word "newspaper" to refer to a particular set of printed sheets. Other times, however, we use the word "newspaper" to talk about a group of news that have been published by a publisher. Finally, other times, with "newspaper" we mean to refer to a group of people (the editorial office) involved in writing and assembling the news to be published.
In the context of a semiotics that deals with media, we must get rid of the ambiguity coming from a generic use of the term "media". We therefore propose a definition of media, or rather of medium, in its singular version, referring to a distinction that is quite common in the literature on the media between medium as a technology and medium as a form of communication (e.g. see, [1], [7], [8]). In the definition we propose, we make a distinction between:
  1. the medium as a technology, that is, as a variety of material components (sensors, micro-processors, monitors, cables, storage devices, etc.) that, once assembled into some device, allow the transmission, reception, and possibly the storage of information;
  2. the medium as a system of significations, that is: (i) a set of features (i.e., possible genres and formats of the objects that can be produced with the medium, possible types of communication allowed by the medium, etc.); (ii) a set of rules for the formation, organization and manipulation of the features; (iii) a set of conventions (culturally, socially and historically determined, and that people follow when they communicate using technologies) that can be used in certain communication contexts to produce particular and tangible forms of communication.
In the rest of this paper, we will deal only with media meant as systems of significations. In this regard, however, we should be aware that in many cases it is not possible to make a clear-cut separation between the two ways of understanding media. In fact, certain material aspects of the media as technologies impose constraints and limits on the characteristics of the media intended as systems of significations (e.g., the act of writing on the touchscreen of a mobile phone establishes a particular relationship between the hands of the user and the interface allowing writing, which, on turn, affects certain features of the text produced, such as brevity, synthesis, essential style, etc. [7]).
What we examined so far is enough to allow us to frame AR within the context of a semiotics of new media. Following our definition of medium, we can now treat AR as a particular system of significations - hence a particular medium - that can be investigated through the methods of semiotics. Moreover, we can also use semiotic analysis with the forms of communication produced with AR, that is, the AEs for HMD.
Here, it seems important to underline that the particular system of significations that we want to examine takes into account some specific features belonging to AR (i.e., the capability to work with virtual contents (holograms), the capability to mix virtual and real contents in a single blend, the use of 3D spaces in which the user is able to immerge, the use of forms of interaction between user and holograms, the capability to provide the user with the full control of his own point of view). These features belong (almost) exclusively to AR and, therefore, are the features that more than anything else contribute to make of AR a unique medium. So, it is exactly by exploring the many possibilities these features offer for the creation of new and original AEs for HMD that we can better understand the great potential AR has in reshaping our experience.
2.2 Multimedia Texts
We are now able to define more clearly what we called, in the previous section, the "forms of communication" created by media, that is, the multimedia objects produced in some communicative context using a medium or several media meant as systems of significations. We will identify each of those forms of communication with a term that is the key concept in this work: a text.
First of all, we need to get rid of the quite common idea that a text is only what is written in a book, or in an article. We need a definition of text capable of accounting for the text as a complex multimedia object. It is impossible here to report here, even briefly, the many definitions of text in the vast domain of textual semiotics (see on this the extensive bibliography in Nöth [18], Pozzato [14]). We have therefore chosen to refer to a definition of text (in Cosenza [7, p. 7]) which seems particularly useful for our purposes.
According to this definition, a text is any portion of reality:
  1. that has meaning for someone;
  2. whose limits can be clearly defined, so that it is possible to distinguish the text from what is outside it;
  3. that can be broken down into discrete units, according to several hierarchical levels of analysis, from the most concrete (which is at the most superficial level) to the most abstract (which lies at the deepest level);
  4. that can be decomposed according to objective criteria, that is, criteria based on motivations and arguments that can be found in the text itself.

Still following Cosenza's suggestions, we add to the above definition of text some further features to account for the fact that texts usually use more than one medium:
  1. the text described above can be said to be multimedia when it brings together several media, each of which is meant as a system of significations;
  2. such media are assembled together through a unitary communication strategy,
  3. that we enjoy through more than one sensory channel.

According to this well-articulated definition of text, we can think of all books or articles using verbal language as the primary medium of communication as texts. Yet, we can also regard as texts many other ‘objects’ produced by different media: a comic strip, a film, a theatrical performance, a website, a commercial, or a piece of software and an augmented environment.
We can now finally reformulate the purpose of our investigation: we want to examine, from the perspective of semiotics of media, the multimedia ‘objects’ that are produced by using AR as a system of significations, that is, the AEs generated by applications running on HMD. During the analysis, these AEs for HMD will be treated as multimedia texts.
We will now propose several ways to "disassemble" an AE for HMD, so to be able to examine its components from different perspectives. It should be noted that the analyses we propose presuppose a form of direct observation of the AE for HMD to be analyzed. And there is only one way for the analyst to have a direct observation of an AE for HMD:  immerge herself into the AE she wants to observe.
3.1 Types of Apps for Augmented Reality
According to some statistics published online [19], Google and Apple classify mobile apps available in their Stores according to 33 and 24 categories, with the top ten categories (games, business, education, lifestyle, entertainment, utilities, travel, book, health and fitness, food and drink) containing more or less 70% of the apps downloaded in 2019. At the moment, AR shows an enormous potential on the market [20], but the number of apps for AR is not yet big enough to allow AR to have its own category, and even less to have a subdivision into sub-categories within the category, especially if we talk, as in our case, about apps running on HMD and generating immersive AEs. Nevertheless, since the task we set in this work is to disassemble as much as possible the AEs for HMD we are dealing with, we begin our analysis by proposing a subdivision of our apps for AR into sub-categories (the categories listed above).
On first approximation, we use the purpose of one app (usually stated by the producer/designer of the app) to insert the app into one of the above categories. So, for example, if we look at the sentence describing the app we have developed ("it creates a learning environment in AR for older adults") we can include our app in the "education" category. We are aware that using the purpose of the app as a way to identify the app category is not particularly reliable. That is why, later on, while examining the AE components, we will try to find better ways to confirm the initial categorical attribution (e.g., in 3.4.1).
3.2 Text Segmentation
3.2.1 Paratext and Text
We consider an AE for HMD as a text. Let us now proceed to break down this text so as to acquire new information on its characteristics and properties. Particularly, we want to segment the text into blocks, each of which maintains a certain autonomy of forms and content.
In 1987, while working on the definition of some criteria for the analysis of literary works, Genette introduced a term to identify a set of heterogeneous elements present in a text: the paratext. The paratext is everything that, although is not really part of a text, accompanies the text and, in different ways and styles, helps it to make itself visible to its readers [21]. The texts Genette referred to were, of course, literary texts, books. So, the paratext was the set of elements typically present in a book: the book title, the author's name, the book cover, a particular graphic layout, the book index, and so on.
We intend to use the idea of paratext –modified to account for the type of multimedia texts we work with– to segment the text we are dealing with into two distinct spaces: the space of paratext and the space of content (containing the primary contents of the text, i.e., all that the text wants the user to perceive, understand, learn, and use). We propose to think of the paratext of a multimedia text as the set of those media components with a communicative function, that are somehow placed at the "periphery" of the text itself, and that help the user of the text to have the best possible experience while enjoying the text [13]. In an AE for HMD, there clearly exist some media components that help the user of the text to get the primary content of the text and yet, cannot be considered as part of the primary content: the interfaces.
By segmenting the text into two different sub-spaces we can keep the interfaces as separate as possible from the primary contents. In such way, not only can we analyze the two spaces separately, but we can also decide, for each space, the most adequate methodological approach to use for the analysis.
3.2.2 The space of Paratext: The Interfaces
It is beyond the scope of this work to propose more detailed analyses of the space of paratext. The study of interfaces is well-established within the domain of Human-Computer Interaction, and the literature on interfaces is huge. So, we let those studies deal with in-depth analyses of interfaces and of the complex relationships between users and interfaces.
For the purpose we set for this work, we are interested in identifying only a few features of the interfaces defined in the space of paratext. Particularly, we want to distinguish the interfaces according to the following simple typology: (a) help interface (providing the user with information enabling the user to deal with the AE); (b) navigation interface (allowing the user to move throughout of the AE); (c) configuration interfaces (allowing the user to personalize some features of the AE).
We can see an example of this subdivision of the interfaces in Figure 1, showing the space of paratext of the learning AE for HMD that we developed. Here, only two types of interface are visible: (a) the help interface (which, in turn, is made up of two interfaces: a “help panel” showing general and local information on the AE and its features; a “virtual assistant” who interacts with the user and guides her throughout the AE); (b) the navigation interface (made up of a “panel” showing the most common vocal commands (for voice navigation) and a more traditional “navigation” bar with links to the different scenes in the AE).

Figure 1. The Space of Paratext: Navigation and Help Interfaces
Figure 1. The Space of Paratext: Navigation and Help Interfaces
The reason why we want to clearly identify the navigation interface is that by examining the navigation interface we find a way to work out a further segmentation of the space of content.
3.2.3 The Space of Content: Content Macro-Units as Scenes
It seems reasonable to assume that, during the process of designing AEs for HMD, designers establish some criteria for segmenting the AEs primary contents. In fact, all the AEs we examined (i.e., Fragments, Holo Tour, Holo Anatomy, Land of Dinosaurs, RoboRaid, Inside Heart, just to name a few AEs for the HoloLens) show some sort of content segmentation. So, we need to find a way to identify and separate those textual segments – we call them content macro-units – that we see in the AEs for HMD.
We have a tool that can help us with this: the navigation interface, with its set of links. When we activate one of the links in the navigation interface, we force the change of a substantial portion of the content: the portion of content we were dealing with is removed from the AE and a new portion of content takes its place. These portions of content are the content macro-units we are looking for. We assigned a name to the content macro-units that together define the whole space of content of a multimedia text: scenes. The choice, of course, is not accidental, since we took the term from Unity, the game engine we used to develop our augmented learning environment. Unity organizes all the contents of a whatever project, often a game, into a hierarchy of scenes, each of which contains all the objects (graphics, environments, obstacles, scripts, characters, etc.) necessary to create a level or a section of the game.
As an example, we examine the learning AE we have developed. By means of the links we find in the navigation bar (see Figure 1), we can identify the scenes that are in the space of content of our AE. Figure 2 shows the segmentation of the space of content into scenes, and the scenes internal organization.

Figure 2. The Space of Content: Scenes segmentation and organization in our learning AE
Figure 2. The Space of Content: Scenes segmentation and organization in our learning AEs
3.3 Text Layering
So far, we found a way to separate the space of paratext from the space of content of a multimedia text, and also a way to decompose the space of content into scenes. Yet, we still do not know how to really distinguish one text from another. The simple segmentation of the space of content of a whatever multimedia text into scenes does not yet allow us to say much about the primary content of the text: what is the text about? what is the content the text wants the user to perceive, understand, know and use? How is this content organized?
It is then necessary to proceed with a further process of decomposition of the text. This time, however, rather than segmenting the text into smaller units, we focus on each of the units we found so far, that is, the scenes. We will continue by creating a layering of analyses [22, p. 33] focused on the components of each scene and on the relationships these components have with one another. Particularly, we will:

  1. identify, for each scene, its various internal components (e.g., the visual space, the sound space, the virtual objects present in the scene, etc.) – we call them content micro-units;
  2. analyze, one by one, the above content micro-units, each time using the theoretical and methodological tools that we believe to be the most appropriate for the task (e.g., we can analyze the sound space of a scene by separating the verbal component within the sound space from the non-verbal and from the musical components; or we can examine the distribution of the different sound components in the sound space, and so on);
  3. analyze the ways in which the different content micro-units interact with one another within the same scene (e.g., we can identify the relationships between the music and the setting, or between the arrangement of lights and the arrangement of virtual objects in the visual space, and so on).
3.3.1 Scenes as Augmented Stages
There are various ways to analyze the content micro-units of each scene. To begin with, we find it useful to treat the scenes as parts of a theatrical performance. In fact, AEs for HMD share many features with theatrical performances. And this means that many of the characteristics and elements of the scenes of an AE can be analyzed by resorting to the existing studies of semiotics of theater [23], [24].
By using semiotics of theater, we will be able to identify the various systems of significations used in theatrical performances. For example, depending on the goals set by our analysis, we can study the systems defining and organizing the visual and sound spaces of theatrical performances; or we can focus on the systems dealing with time, or with the ways actors use verbal and non-verbal communication; we can study the organization of the virtual objects used to create the scenography, or the forms of interaction between virtual actors (people or objects) and the user, and so on.
Of course, when examining AEs as forms of theatrical performances, it becomes necessary to take into account the differences and distances between the worlds represented in the various AEs and those proposed by more traditional theatrical performances. For example, it seems important to point out that the user’s point of view in an AE is very different from that of a spectator of a theatrical work. In a theater, except for rare cases, the spectator is placed in front of the stage and, in any case, outside the space that defines and delimits the space of the performance. In AEs, the user can change her point of view by simply changing her position; moreover, the user always remains somewhere on the stage, and within the space of the performance. This greatly affects the role of the user in relation with the contents presented. In an AE, the user is at the same time a spectator and a protagonist of the performance that has been created only for her. As main actor in the performance, the user is actively (and emotionally) involved in the construction and manipulation of the elements and events proposed by the AE. And we must remember that the ability to emotionally involve the user in the experience created by an AE for HMD is one of the peculiar characteristics of AR, which distinguishes it from almost all other media.
We must also consider that the space of a theatrical performance is less complex than the space defined in an AE. The space of the theatrical representation, with rare exceptions, is determined by the physical space available for the stage; in this space, actors and scenic objects are mainly distributed on a horizontal plane over the stage. The space of an augmented representation has no such limitations. By adapting itself to the size and shape of the real space in which the user is located, the space of an augmented representation coincides with the whole AE. Virtual people and objects can be literally anywhere within this space, with results that are impossible to achieve with the theater.
Despite these and other differences, here not examined, between theatrical and augmented representations, we believe that the semiotics of theater can be a powerful analytical tool to capture many aspects of the internal components of the scenes of a text, particularly when the text is built around one or more narratives  (as in the case of games, or cultural or artistic AEs).
3.3.2 AR Native and Imported Contents
There are further or alternative ways to analyze the content micro-units of a scene. We begin by making a distinction between two general types of content micro-units:

  1. AR ‘native’ content micro-units: the content micro-units that are created directly in AEs and that can only exist in AEs (e.g., in the learning AE we developed, we created an exercise that requires the user to manipulate (i.e., move, rotate, resize) a group of holograms until a certain spatial arrangement of the holograms is reached);
  2. imported content micro-units: the content micro-units that have been created with other media and are imported into AEs (e.g., a video of a lesson created for an e-learning website and imported unchanged into one AE, so that it is visible on a virtual player (a hologram) hanging on a real wall).

Case a: If we want to examine the native content micro-units, then we need to refer to sets of features, rules and conventions defined in the AR medium meant as a system of significations (see 2.1).
It seems important to remind that some of these features (the capability to work with virtual contents (holograms), the capability to mix virtual and real contents in a single blend, the use of 3D spaces in which the user is able to immerge, the use of forms of interaction between user and holograms, the capability to provide the user with the full control of her own point of view) belong (almost) exclusively to the AR system of significations, and contribute to make AR a unique medium.
As for the rules, they take into account the above features, and:

  • allow the definition (creation) of the possible holograms in AEs;
  • allow various forms of hologram manipulation in AEs (e.g., moving, scaling, rotating, etc.);
  • allow various forms of organization of holograms in AEs (e.g., the above exercise).

It should be noted that if we want to examine the forms of organization of holograms, we might find it helpful to refer to the forms of organization other media have already established to organize their own ‘objects’. These objects are different from holograms; however, the ways these objects are organized might be quite similar even in different media. For example, if we think about the exercise we previously described (the manipulation of holograms in one AE, looking for a certain arrangement of the holograms) we realize that even though only AR makes it possible to manipulate holograms within an AE, the idea of moving some objects in a space according to some logic and goal can be formulated and used in other media and independently from AR. So, while designing exercises for our learning AE, we took that idea and used it to define one particular exercise, and to organize the behavior of some holograms in our AE.
We can take advantage of this to somehow simplify our analyses. To analyze the forms of organization of holograms,  defined in the AR system of significations, we can use the extensive literature relating to the specific semiotics that study different (and more known) systems of significations, such as  semiotics of film, of video and video clips, semiotics of visual communication, semiotics of visual arts, semiotics of photography, semiotics of music, semiotics of marketing,  textual semiotics, semiotics of news, and so on (for bibliographical suggestions on these and other semiotic themes see [18]). After all, that is exactly what we did in the previously section on theatrical performances.
Of course, we must not forget that when we "import" into AEs certain forms of organization of objects defined in other media, we must inevitably adapt (that is, change, extend) those forms of organization in order to make them compatible with the AR system of significations. Therefore, we should account for all the possible transformations of whatever goes through the importing process (as we did in the previous section dealing with the difference between augmented and theatrical representations).

Case b: If we want to examine the imported content micro-units, then we must consider that such units are originally created with other media and are then (re)constructed by the medium RA (meant as system of significations). Dealing with such texts might require two different type of analysis:

  • on one hand, we can look at the content micro-units simply as objects going through a process of conversion due to their being imported into an AE. For example, let us consider again the video-lesson we previously talked about. The video is taken out of a web page and imported into one AE. By means of the process of importing, the video becomes a hologram (i.e., a virtual player showing the video, which maintains the exact content it had in the web page). Now, if we look at the video only as a hologram existing in the AE, then we can deal with it by using the rules for the manipulation and organization of holograms defined in the AR system of significations (i.e., move, rotate, resize, aggregate to other holograms, etc.), as we would do with native content micro-units.
    It should be noted that, in this case, we don't even need to consider the content micro-units as texts. We are more interested in identifying the rules handling the importing process, and according to which the content micro-units are ‘re-written’ to fit and work in the new AR system of significations. These rules are mostly technical, since, in a general sense, they are used to convert texts given in certain format into texts in a different format. As such, they do not really require a semiotic analysis.
  • on the other hand, we can analyze the content micro-units as autonomous texts, that is, regardless from the fact that those texts, due to the importing process, have become constitutive elements of one AE. In this case, we can again resort to specific semiotics (see case a in this section) and carry out particular analyses by examining each text from different perspectives and with different levels of penetration and abstraction. For example, if we consider, again, the video lesson we previously talked about, for our analysis we can use semiotics dealing with videos, films, or documentaries; or we can use others semiotics dealing with communication, or education, to explore the communication techniques used in the video to convey educational content in online courses; or we can examine the video editing techniques used to assemble the different sequences that make up the whole video; or we can identify the characteristics of the different media involved in the making of the video.

3.4 Text and Context
So far, we have focused our analyses on the AEs for HMD, considering them as texts. However, a text, any text, does not live in complete isolation. A text always exists within a context, that is, a culture that is historically, socially, economically determined, and has produced that text and/or uses it. So, to better understand the multiplicity of meanings that can emerge by analyzing a text from multiple perspectives, we place the text within a communicative situation capable of accounting for the complexity of the context.
Here, we cannot explore in detail the concept of communicative situation (for a more in-depth analysis on this, the reader can refer to Petöfi [11]–[13]). But we must at least mention the idea that in order to have a communicative situation we need at least two communicators that somehow interact by means of/with a text: one of the communicators is a producer who creates (by using one or more media meant as systems of significations) the form of communication that we call text; the other communicator is a receiver that receives the text created and uses it in some ways.
It is important to note that the production and reception of the text are possible because both the communicators possess a body of knowledge about the world that share, more or less extensively, with their social environment. In semiotics, this body of knowledge about the world is often called the personal encyclopedia of an individual. It is a whole, not always easy to systematize, made of knowledge, skills, experiences, beliefs and expectations. The personal encyclopedia is the necessary background knowledge each individual needs in any communication process [16], [25]. In fact, it is precisely by using what is present in their individual encyclopedias that communicators know, during a communicative act, all that is necessary to know to create a certain text and/or to use it.
3.4.1 Producers and Model Receivers of Augmented Environments
Since we propose a semiotic approach to AEs for HMD, we think it useful to talk about something that is within the personal encyclopedia of the producer of an AE: it is what, from a semiotic point of view, is defined as the model receiver. The model receiver is a more or less abstract and accurate profile, which is (more or less consciously) defined by the producer in her encyclopedia and which she will use during the process of design of the text. Therefore, when we talk about the model receiver, we are not talking about the empirical receiver, but rather about a set of predictions the producer makes on mental states, competences, interpretative steps, and concrete actions of the empirical receiver [7] [15].
The model receiver plays a fundamental role in the production of any text. In fact, we must remember that every text is produced not only for a purpose, but also for a specific target. So, the producer of an AE for HMD (as text) should always have that target/model receiver in mind (in her personal encyclopedia) during the design of the AE. Every aspect of the AE – from the organization of the scenes to the definition of the interfaces, from the graphics, to the music, up to the choice of the primary contents and of the different systems of significations to be used, etc. – should be carefully designed to allow the empirical users of that AE to have the best possible experience.
Given the importance the model receiver has in the process of designing AEs for HMD, once we undertake the analysis of an AE, we should be able to find, within the AE, the traces of the model receiver that the producer has somehow "embedded" in the AE.  Then, we can use those traces to try to reconstruct the profile of the model receiver of that AE.
In an AE for HMD, there are various elements/features that help us to figure out the AE model receiver:

  • the purpose of the AE (and, therefore, the type of app, identified in 3.1) and the information acquired from the analysis of the content micro-units of the different scenes (as seen in 3.3) can already provide us with an idea of what the model receiver of the AE is. We can then improve this first definition of the profile of the model receiver by examining the general features of already existing profiles (i.e., social, professional, cultural, or recreational operators; or, if we need a finer definition, we can even refer to more detailed profiles: doctors, students, engineers, humanists, gamers, visitors of museums or cultural events, users defined by age, or with specific impairments, etc.);
  • the in-depth analysis of the interfaces and of the content micro-units of the different scenes allows us to identify also the types of possible actions the user is allowed/not allowed to perform in the AE, and that can be considered as typical actions for a certain type of user.

For example, let's examine the learning AE for HMD we have developed. Its purpose is to create a learning environment in AR for older adults, so we can say that the right category for the app is "education". From the analysis of the content micro-units of the scenes we have the confirmation that the AE deals with educational contents on a certain domain of knowledge (digital culture). So, as a first approximation, we can say that the right user of our educational AE is a student. Moreover, again from the purpose of the AE, and from other information we find in the interfaces (particularly the help interfaces) and in other content micro-units in the scenes, we have hints on the fact that the AE is designed for elderly users. So, we refine our previous definition: the right user of our AE is an adult student (who is 65 or older). By looking further into the interfaces and content micro-units of the scenes, we find other clues that confirm the type of user we identified. For example, the large font size used in all written words visible in the AE facilitates the reading even for users with impaired eyesight (a typical problem in older adults); the AE generates holograms only in the space in front of the user, so that even users with limited motor skills (a possible problem in older adults) can easily perform the gestures necessary to manipulate those holograms. We are therefore able to confirm again that the AE has been designed for elderly students, and that the "elderly student" is the model receiver we are looking for. Now, if we need, we can refine the profile of model receiver of our AE by conduction further research from different sources (e.g., ISTAT, demographic, sociological or medical studies, etc.).

How can the semiotic analysis of an AE for HMD use the profile of a model receiver extracted from within the AE to produce new insights or new analyses? Before taking into account the model receiver, the semiotic analysis allowed us to answer to questions such as: what kind of AE are we dealing with? how does this AE work? how can I segment it? what are its interfaces, and how do they work? how can we access the contents of this AE? Once we introduce the concept of the model receiver of an AE, we can ask different types of questions: have the AE interfaces been properly designed for the target/model receiver (defined by the designer in her encyclopedia) of this AE? Are the various content micro-units of the scenes easily accessible (i.e., understandable, manageable, usable) by the model receiver? Or, more in general: is the AE generated by the app able to satisfy the needs, taste, emotions, capabilities, and competences of the user for whom the AE has been designed?
These questions open the way to new analyses focused on the entire design of the AE. Analyses that are, again, within the Human Computer Interaction domain, and that try to investigate all aspects of the interaction between the users of an app for AR and the AE generated by the app.
In this work we have adopted the perspective of a semiotics of media to propose many analyses on what we called the AEs for HMD. Through such analyses, we have been able to separate the interfaces of an AE from its primary contents; we have been able to disassemble the scenes that make up the primary contents of an AE; we have been able to identify the different semiotic systems involved in the analysis of the micro-units of content of each scene, whether they are AR native or imported into the AE from other media; finally we have been able to examine the consequences of placing an AE within a communicative context.
In this work we decided to deal with the semiotic analysis of AEs for HMD because we are quite confident that the opportunity to look at AEs for HMD from a multiplicity of perspectives, and the possibility to disassemble them to examine in detail their constitutive elements, can foster the understanding ─ from a non-technical point of view ─ of what AR is. Moreover, we believe that the semiotic approach here proposed might encourage reflection on how to concretely use the findings of the analysis to develop truly new forms of communication, which cannot be created with other media.

Also, there is a second and perhaps more important reason that led us to write this work. Surely, analyses such as the ones we proposed here can be quite useful when we want to examine the AEs for HMD of already developed apps for AR in an attempt to establish, a posteriori, whether the AEs generated by those apps satisfy certain requirements, or show certain features. However, we really think that the analyses presented in this work can be used more profitably if carried out not a posteriori, on AEs generated by apps already developed, but rather during the design process of the AEs. Normally, semiotics has a descriptive attitude, that is, it limits itself to describe “only what it sees” during the disassembling of a text. On the other hand, nothing prevents us from using semiotics even in a normative way. Then, the findings of a particular semiotic analysis conducted on an AE for HMD which is still at an early stage of development can be transformed into "guidelines" useful to define, change and improve the design of the augmented environment.
[1] B. MacIntyre, J. D. Bolter, E. Moreno, and B. Hannigan, "Augmented reality as a new media experience," in Proceedings - IEEE and ACM International Symposium on Augmented Reality, ISAR 2001, 2001, pp. 197–206.
[2] D. Schmalstieg and T. Hollerer, Augmented reality: principles and practice. Addison-Wesley Professional, 2016.
[3] S. Aukstakalnis, Practical Augmented Reality: A Guide to the Technologies, Applications, and Human Factors for AR and VR. Addison-Wesley Professional, 2017.
[4] J. D. Bolter and R. Grusin, Remediation. Understanding New Media. MIT Press, 1999.
[5] L. Manovich, The Language of New Media, vol. 27, no. 1. MIT Press, 2001.
[6] J. H. Murray, Inventing the Medium - Principles of Interaction Design as a Cultural Practice. The MIT Press, 2012.
[7] G. Cosenza, Introduzione alla semiotica dei nuovi media. Editori Laterza, 2014.
[8] M. Danesi, Understanding Media Semiotics. Oxford University Press, 2002.
[9] C. S. De Souza, The semiotic engineering of human-computer interaction. MIT Press, 2005.
[10] C. S. De Souza, S. D. J. Barbosa, and R. O. Prates, "A semiotic engineering approach to user interface design," Knowledge-Based Syst., 2001.
[11] J. S. Petöfi, Scrittura e interpretazione: introduzione alla testologia semiotica dei testi verbali. Carocci, 2004.
[12] F. Barbabella, "An Introduction to Semiotic Textology," in Family carers: technology-based support services, Aracne, 2015.
[13] F. Barbabella, "La comunicazione umana mediata. Proposte per un modello semiotico-testologico multimediale.," Macerata, 2008.
[14] M. P. Pozzato, Semiotica del testo : metodi, autori, esempi, 1st reprint, 2011. Carocci, 2001.
[15] U. Eco, Lector in Fabula. La cooperazione interpretativa nei testi narrativi. Bompiani, 1979.
[16] U. Eco, A Theory of Semiotics. Indiana University Press, 1976.
[17] S. Cicconi and M. Marchese, "Augmented Learning: an E-Learning Environment in Augmented Reality for Older Adults," in INTED 2019 Proceedings, 2019, pp. 3652–3662.
[18] W. Nöth, Handbook of semiotics. Indiana University Press, 1995.
[19] "What are the popular types and categories of apps," 2019. Retrieved from:
[20] "20 Augmented Reality Stats to Keep You Sharp in 2019," 2019. Retrieved from:
[21] G. Genette and M. Maclean, "Introduction to the Paratext," New Lit. Hist., vol. 22, no. 2, 1991.
[22] F. Casetti and F. Di Chio, Analisi del film. Bompiani, 1990.
[23] K. Elam, The semiotics of theatre and drama, 2nd ed. Routledge, 2002.
[24] E. Fischer-Lichte, The semiotics of theater. Indiana University Press, 1992.
[25] U. Volli, Manuale di Semiotica, 7th ed. 2003.

Leave a Reply

Your email address will not be published. Required fields are marked *