Augmented Classrooms: A Generator Of Augmented Environments For Learning

posted in: essays | 0

Augmented Classrooms: A Generator of Augmented Environments for Learning

Sergio Cicconi, Maurizio Marchese Dept. Information Engineering and Computer Science - University of Trento (ITALY)
in INTED 2023, 6-9 March 2023, Proceedings, Valencia: INTED 2023.
Augmented Reality (AR) is an emerging technology allowing the creation of augmented environments, in which computer-generated digital content is seamlessly combined and overlapped into our perceptions of the real world. In this paper we present an Augmented Environment dedicated to learning, capable of hosting and organizing a variety of learning contents. This Learning Augmented Environment has been built with the “Generator of Augmented Environments” (GAE), i.e. a software package we developed with the game engine Unity. The GAE allows the creation of Augmented Environments that run on head-mounted devices, providing a variety of immersive experiences. In the first part of the paper we examine the main features of the GAE. We start by defining an Augmented Environment (AE) as a set of immersive scenes created in AR and linked one another according to a given organization. We also define an immersive experience into a specific AE as the experience of a particular navigation throughout all the augmented scenes that make up that specific AE. By using selected content which has been virtualized (transformed into holograms), thanks to the GAE it is possible to generate a specific AE that allows the immersive navigation throughout the selected content. In the second part of the paper we provide an example of using GAEs to create a Learning AE on “digital culture”. Particularly, we examine the main steps of the process that, thanks to the GAE, transforms the learning material of the course into an augmented environment to be enjoyed with head-mounted devices, providing users with an immersive learning experience. We believe that the Generator of Augmented Environments provides an important contribution to the development of learning augmented environments. The GAE helps to simplify and speed up the still rather complex process of building Augmented Environments. Indeed, the GAE allows authors of learning content to focus on the development of specific content they want to use in one Augmented Environment, rather than on the process of creating the augmented environment that will host that content.
In recent years, Augmented Reality (AR) has gained widespread popularity and has been applied in a variety of fields, including entertainment, education, advertising, and professional training. Considering the growing number of AR applications, we thought it would be useful introduce with this work a tool called the Generator of Augmented Environments (GAE) that has been designed to generate customized augmented environments.
In the first part of this work, we will describe the features of the GAE and the context in which it has been developed. We will begin by discussing the main features of augmented reality and augmented environments, and then proceed to describe the features of the GAE.
In the second part of this work, we will present a case study demonstrating the use of the GAE in the context of learning. Particularly, we will use a specific learning course on Digital Culture, and examine the process of transforming it into an augmented environment that can be accessed using wearable devices (Head- Mounted Devices – HMDs), and that provides the users with an immersive and interactive learning experience.
1.1 Augmented Reality and Augmented Environments
Augmented reality is an emerging form of experience in which the real world is enhanced (or augmented) by computer-generated content. AR allows digital content to be seamlessly overlapped and combined into our perceptions of the real world in real time [1], [2], [3]. More advanced forms of AR create fully immersive experiences involving sight and hearing and add rich and significant extra-worlds to the real world (see Figure 1). Some recent AR systems actually ­leading the AR market, such as Microsoft HoloLens 2 [4], Meta Quest 2 [5], or Magic Leap 2 [6], allow users to interact with the virtual objects (holograms) existing in the augmented environments. In some cases, such as the Meta Quest 2, haptic gloves connected to the HMD can even be used to simulate the sense of touch [7].
AR is accessed through an app installed on a HMD or mobile device. While the device’s camera captures the image of the real world and sends it to the device’s display in real-time, the device’s processor generates on the device’s display a virtual world that perfectly overlays the real world. If the device is a HMD, then we have an immersive form of augmented reality ( Figure 1 , Figure 2 -left). If the device is mobile then we have a non-immersive form of augmented reality ( Figure 2 -right). In either case, the augmented spaces perceived by the user, which are a synthesis of the real and virtual worlds, are referred to as Augmented Environments (AE).

Figure 1. The Space of Paratext: Navigation and Help Interfaces
Figure 1 : A conceptual diagram of an AR system, with a HDM and see-through displays. The see-through displays allow the user to see the real world as if looking at it through a normal pair of glasses. Through sensors, tracking systems and cameras, the device acquires information on the user’s position and on the objects in the real world surrounding the user. Virtual objects are then generated in real time in the see-through displays, so that they are seamlessly integrated with the objects in the real world.
As mentioned, the GAE is a tool for generating augmented environments. Now, we can add that all the Augmented Environments created making use of the GAE have necessarily to display the typical features of augmented spaces [[1], [8], [9]. Which means that they have to be able to seamlessly combine the real world and the virtual world; and that they must also be fully 3-D and immersive, and that, finally, they must allow interactivity in real-time, so that the users can interact with the holograms as if they were real people and real objects. Therefore, it is necessary to use an HMD in order to fully experience and enjoy AEs generated using the GAE.”

Figure 2. Immersive and non-immersive forms of Augmented Reality
Figure 2: Immersive and non-immersive forms of Augmented Reality.
2.1 Two different perspectives on Augmented Reality
With this work, we intend to continue an investigation that we started with a previous work dealing with a semiotic approach to augmented reality applications [10]. Here, we want to further deepen our knowledge related to augmented reality, this time by focusing on the products developed using the AR technology. Specifically, we will deal with augmented environments meant as new forms of communication created using the new medium called augmented reality. Additionally, we will focus on a particular type of augmented environments, that is, on augmented environments dedicated to learning.
Our approach to AR is intended to be an alternative to the more common and widespread approach that views AR from a purely technological perspective. In most cases, such a perspective deals with the study of sensors, microprocessors, displays, cables, speakers, storage devices, and other hardware components. This approach often seeks to answer questions about how these components work, how their efficiency can be improved, and how they can be assembled into devices capable of creating augmented experiences.
We believe that the discussion on AR should not be limited to explaining how technology enables the creation of augmented experiences. The technical language used by this approach often does not allow for adequate examination of the objects created with the AR technology. It does not allow for adequate explanation of the ways augmented environments are designed and developed. Therefore, in this work we will not address technical aspects associated with the development of AR hardware components, and will instead focus on problems related to the design and development of augmented environments. However, despite the removal of the above technical aspects, if we deal with the creation of augmented environments, we cannot avoid dealing with other types of technical aspects, this time associated with the development of these augmented environments. Indeed, while the design of augmented environments requires a diverse range of knowledge and competencies in various fields (e.g., the knowledge of the fields the augmented environment is about, and also a more technical knowledge on the design of virtual worlds [11]), the development of augmented environments also requires proficiency in a variety of programs (such as Unity, Maya, Blender, Character Creator), and programming languages (such as C#, C++, Java, and JavaScript).
So, the creation of augmented environments remains a complex task. And it is precisely to address this complexity that we have developed the Generator of Augmented Environments. We looked for a way to reduce, at least in part, that complexity. Indeed, with the help of the GAE, we can make the task of creating augmented environments a little faster, and a little easier.
2. The Generator of Augmented Environments
The GAE is a tool for the development of augmented environments. At the same time, the GAE is itself an augmented environment developed with Unity, which is a cross-platform video game and simulation engine. The GAE is designed to host any type of content (acoustic and visual) that can be represented within a virtual space. And it is also able to expand and customize itself according to a set of requirements provided by a designer/developer.
We want to make it clear, however, that the GAE is not a tool for designing augmented environments. The GAE works by using a design that has been previously created by an author working on a specific project. Project that exists before the GAE, and outside the GAE.
The creation of a customized augmented environment is carried out through two distinct stages of development, and only the second stage involves the use of GAE. In addition, the creation of a customized augmented environment requires at least two different professional figures, each of which coordinates the internal activities of one of the development stages.
During the first stage of creation, one author/designer takes care of working out the design of the representation of an idea. In particular, this author/designer identifies a discipline, or a certain domain of knowledge and, within it, chooses suitable contents to represent a certain idea; then, the author/designer segments the chosen content into a finite number of relatively self-contained and easily identifiable portions of content; finally, he clearly defines the organization of those portions of content.
During the second stage of creation, one augmented environment designer uses the design create during the previous stage to develop within an augmented environment an adequate representation of the idea conceived by the author. And it is exactly in this stage that the GAE comes into play: the augmented environment designer uses the GAE to facilitate and speed up the task of creating a customized augmented environment in accordance with the requirements defined in the design.
2.1 Components of the Generator of Augmented Environments
The GAE uses two key elements to build and customize an augmented environment according to the design requirements: the Virtual Room and the Connections Panel.
The Virtual Room
the Virtual Room (VR) is a hologram shaped like a room, regular and square, made up of 4 virtual walls (3 meters wide by 3 meters high), a virtual ceiling, and a virtual floor. Therefore, like any real room, the VR is displayed to the user wearing a HMD as an immersive, three-dimensional object.
Here are some properties of the Virtual Room:
  • The VR recognizes the features of the real room the user is in (shape and dimensions of the real room, objects and furniture in the room) and reshapes itself to overlap the real room.
  • The VR is empty, that is, without any content. Whatever type of content will be added later, when the whole structure of the augmented environment has been completed (on this, see more in Section 3.3). On the other hand, at the time of its creation, the VR already contains two main interfaces: the help interface (for assistance and information), and the navigation interface.
  • The VR can easily be duplicated, and any new VR maintains all the properties of the VR it was duplicated from.
  • Each VR is designed to contain a variety of content in the form of holograms. Particularly, each VR will host what we previously defined as relatively autonomous portions of content.
  • A finite set of VRs connected to each other according to some criteria specified in the design is the complete structure of one particular Augmented Environment. Any augmented experience a user can enjoy, always happens within one VR that is part of one particular Augmented Environment.

The Connections Panel
We saw that any augmented environment can be thought of as a finite set of connected VRs. While the VRs are in charge of hosting those relatively autonomous portions of content identified in the design, the Connections Panel − a panel inside the GAE − is responsible for acquiring and managing the organization of those portions of content. In other words, the Connections Panel manages the connections among the VRs according to the criteria that have been established in the design. As a matter of fact, the Connections Panel builds a guided navigation for the user by providing the navigation interface of each VR with information on which other VRs are connected to it. In this way, the user can navigate from the VR where he is in only to those VRs connected to it.
Figure 3 shows two different configurations of a Connections Panel for an Augmented Environment with 6 VRs. In the panel on the left, the Augmented Environment Designer, following the criteria that have been established in the design, has set only the connection between the VR 3 and the VR 4. This information will be sent to the navigation interface of the VR 3, which will thus allow the user to go from the VR 3 to the VR 4. The panel on the right shows the Connections Panel of an Augmented Environment with 6 VRs with a much more complex set of authorized connections between VRs. This panel will be explained in Section 3.

2 different configurations of  a Connections Panel for an augmented environment with 6 VRs.
Figure 3 : 2 different configurations of a Connections Panel for an augmented environment with 6 VRs.
2.2 The generation of augmented environments
In the initial stages of development of a specific augmented environment, the Augmented Environment Designer will provide the GAE with the information necessary to customize the original Augmented Environment and transform it into the desired Augmented Environment. Particularly, the Augmented Environment Designer will insert into the GAE the number n of VRs necessary to complete the structure of the desired Augmented Environment. He will also provide the GAE with information on the organization of the different VRs, that is, the set of connections allowed between the different VRs.
At start, the GAE only contains one generic and empty VR with the features described in the previous Section. Based on the information provided by the Augmented Environment Designer, the GAE:
  • Creates a complete augmented environment with a number n of copies of the initial VR, with n equal to the number of relatively autonomous portions of content identified during the creation of the design. All the created VRs will be empty (but will include the help and navigation interfaces).
  • Creates one Connections Panel with n rows x n columns (similar to the one in Figure 3-right) capable of accounting for all possible connections between the n VRs created above. It will then be up to the Augmented Environment Designer to appropriately configure the Panel so as to reproduce in the Panel the organization of the VRs (and therefore of the contents) defined in the design. The map of connections will be transferred to the navigation interface of each VR, so that each VR will allow the user to navigate only to the connected VRs.

Once this empty augmented environment has been built, the Augmented Environment Designer will use the GAE to facilitate the population of the different VRs with the specific portions of content identified by the author/designer of the representation of the idea or event to be experienced in the Augmented environment.
2.3 Populating an Augmented Environment
This is not the right place to describe the set of required skills and the activities involved during the creation of virtual content for any specific Augmented Environment that we may have built with the GAE. In general, the creation of virtual contents can be carried out with or without Unity. If the Augmented Environment Designer keeps working with Unity, then he will also work within the GAE, since the GAE is a piece of software developed with Unity. On the other hand, the Augmented Environment Designer can also decide to create the virtual contents using dedicated programs for creating and modeling virtual objects of any type and shape (from avatars to complex objects). In this case, these virtual objects, once completed, must be imported into Unity and placed into the VRs of the specific Augmented Environment to be populated.
What we would like to emphasize here is that, in the phase of populating the Augmented Environments, the GAE plays a secondary role, while the role of the Augmented Environment Designer becomes fundamental. As we said, the GAE manages to speed up and simplify the construction of complete (and empty) augmented environments by automating those often long and tedious processes that are necessary to build that set of virtual objects, general features, and functionality that is shared by all augmented environments.
However, when it is time to work on the creation of digital contents for the augmented environments generated by the GAE, the work of the Augmented Environment Designer, or of a 3D modeler, closely resembles the creative work of a traditional artisan, or of a carpenter or a sculpturer. And in this case, it becomes really hard to think about automatizing those creative processes. At this point, the GAE’s work is over.
It seems useful to conclude this work with an example of use of our GAE in the domain of augmented learning [12], [13]. For the sake of concreteness, let’s say we want to create a learning course on Digital Culture, even though the choice of a topic is not strictly necessary, since the method we show here can be used with any topic.
First of all, we want this course to be usable in immersive augmented reality, so we want a course capable of providing users with an immersive and interactive 3D learning experience. Also, we want the course to be a self-paced course, taken by a solo learner (so, no other classmate shares the augmented learning experience at the same time). Finally, but that is just it an option that we propose, we want the course to be conducted by a virtual instructor, which means that we want an avatar/hologram that can interact with the user, can help him or her to move around the Augmented Environment, and can use images, video and audio files, documents, exercises, and, more in general, any kind of learning material useful to accomplish a learning experience.
We saw that the creation of the representation of a whatever kind of idea or event within an augmented environment can be divided into two stages: (1) designing the representation of the idea or event, and (2) developing that representation within an augmented environment.
Stage 1: In Stage 1, we took the role of Authors/Designers of the representation of the event which we called “Course on Digital Culture”.
  • We identified some design principles and methodologies, learning theories and learning tools to define an adequate theoretical framework for designing a learning course [14], [15], [16], [17], [18], [19], [20].
  • We identified a domain of knowledge (related to a general idea of “digital culture”) with content relevant to the topics to be covered in the course we want to build (see Figure 4; to identify and select the contents within such a domain we relied on a few introductory books or manuals on ICTs, computers and digital tools [21], [22], [23]).
  • We selected those didactic contents that we thought of as most suitable for the course, and then we built an initial, rough organization of these contents.
  • Relying on Instructional Design methodologies for the segmentation of learning materials, and following some specific logical, temporal, causal, and pedagogical principles, we segmented the chosen learning materials into more or less complex learning units (e.g. lessons, topics, sub- topics, activities, exercises, tests, assignments, and so on – see Figure 5-a).
  • We reworked the first rough organization of the contents, and built a solid hierarchical structure of interconnected learning units with different levels of complexity (see Figure 5-b). Such a hierarchical structure, together with all the learning units (also including all the contents of the learning units ), is what we call the design of the learning course on Digital Culture. We will use this design in Phase 2 to develop the Augmented Environment with the representation of the course.

The domain of knowledge we used to select the contents for the course on digital culture.
Figure 4: The domain of knowledge we used to select the contents for the course on digital culture.
Stage 2: In Stage 2, we took on the role of Augmented Environments Designer of the representation of the event which we called “Course on Digital Culture”. Based on a subjective evaluation (which, therefore, can vary from course to course, and from designer to designer) we decided that the topic (that is, what we defined as a relatively autonomous portion of content) is our basic learning unit. We also decided that each Virtual Room of the Augmented Environment we want to create will contain only one basic learning unit. Therefore, the total number of VRs of our Augmented Environment will be equal to the number of topics covered in the course.

The domain of knowledge we used to select the contents for the course on digital culture.
Figure 5: (a) Learning Units and their components. – (b) a possible organization of the different learning units of a course. At the lowest level of complexity, we find learning activities, each of which uses one or more multimedia forms of communication. By assembling together several learning activities we create topics; lessons are created by assembling different topics together. Finally, a Learning Unit which includes the whole course is made up of all the lessons in the course.
To simplify, we continue our example taking into account only 4 topics in Lessons 1, and 2 topics in Lesson 2, as shown in Figure 5-b. It is clear that whatever we will do with Lessons 1 and 2 can be repeated with all the Lessons of the course.
  • First, we inserted into the GAE the number of VRs to be generated (in our case, we have 4 topics in Lesson 1, and 2 topics in Lesson 2, for a total of 6 topics; therefore, we need 6 VRs).
  • The GAE generated 6 empty Virtual Rooms (but with navigation and help interfaces in each VR) with all the features examined in the previous Section.
  • The GAE also generated a 6 rows x 6 columns Connections Panel like the one shown in Figure 3-right, to account for all the possible connections between the different VRs. This Panel, once customized, will transfer to the GAE the information defined during the creation of the design on the organization of the relatively autonomous portions of content (the topics).
    It should be noted that the connections between the VRs are much more than what we might expect. Indeed, in the Panel in Figure 3-right, we see the connections linking together the VRs one after another in an ordered sequence (i.e., VR1→VR2→VR3, and so on), as it is logical to expect. But we also see other backward connections (i.e., VR4→VR2, VR5→VR1, and so on), that is, connections that allow the user to go back to previously visited VRs. Alas, this freedom of movement of the user within the augmented environment is an inevitable complication. Or, at least, it is in our case, when the augmented environment we want to build is a learning environment. In fact, if it is true that the user of the proposed course should have access to Lesson 2 only after having completed all the topics in Lesson 1, it is also true that once inside Lesson 2, the user could also want to go back to Lesson 1 (to revise, or to improve the understanding of certain topics). And this applies to all the topics of the course. Consequently, a mapping of the connections between VRs must show the possibility of free navigation between topics, therefore, also between already visited topics.
  • The GAE transferred the information on the connections to the navigation interface of each VR, so that the user of the Augmented Environment will be able to move between the different VRs according to the criteria defined in the course design.
By this time, the Augmented Environment that will be used to represent the learning Course on Digital Culture is complete, at least in its structure. Figure 6 shows the first 6 VRs of the Augmented Environment created by the GAE, and the allowed connections between the different VRs.
We do not need to further explore our example. Obviously, the next and final stage of development in Augmented Reality of the course deals with populating the newly built Augmented Environment. However, as we clarified above, creating virtual content for the Augmented Environment is a complex and time-consuming task that cannot really be automated. So, the close examination of such process of content creation is beyond the scope of this work.

The domain of knowledge we used to select the contents for the course on digital culture.
Figure 6: A portion of the Learning AE representing the learning course on Digital Culture. Each VR contains one basic learning unit (a topic) and is connected to many (but not all) other VRs.
We believe that the Generator of Augmented Environments provides an important contribution to the development of augmented environments. Overall, the GAE helps to simplify and speed up the still rather complex process of building Augmented Environments. As for learning, the GAE allows authors of learning courses, usually not really familiar with the more technical aspects of the development of augmented environments, to focus on the development of specific learning content they want to use in one Learning Augmented Environment, rather than on the process of creating the augmented environment that will host that content.
We would like to add a further consideration regarding the contents of augmented environments. As we stated, the creation of digital content for augmented environments is difficult to automate, and therefore remains one of the most complex tasks during the creation of any augmented environment. However, there is more to it. Increasing the complexity of an already complex task is the fact that virtual content creation should always take into account the features of the medium (in our case, Augmented Reality) that will allow for the enjoyment of that content.
In other words, referring back to our example about the course on Digital Culture, if we want to create a good course that produces true augmented learning experiences, it will not be enough to simply borrow/copy learning material from a course that has already been defined in a certain medium (for example, an online course) and convert/recreate that learning material into a format suitable for being enjoyed in Augmented Reality. The process of content creation cannot be reduced to a purely technical process of converting contents from one format to another. We must understand that every process of transferring contents from one medium to another is never neutral or transparent. If certain contents that exist within a certain medium are imported into another medium, they change, they are re-mediated [24], and they acquire different meanings.
Just to make an example on this, let’s think about one video enjoyed on a web page of an online course on a certain topic X. Then, let’s think of that same video within a learning and immersive augmented environment dedicated to the same topic X. In both situations, the content of the video is the same. However, the media that allow the enjoyment of that video are different, and operate with different modes. So, the experience of enjoying that video by a user will be different, depending on the medium the user is using: it will have different emotional impacts on the user, and will therefore have different consequences for the user. As a matter of fact, that same video enjoyed through the two different media will not be the same.
If we want to create a good course that produces a fully augmented learning experience, we will have to do something more and something different than the simple act of copying/converting content from one medium to another. We will have to rethink about the learning materials we want to use in the course so that, when those learning materials are placed within an immersive augmented environment, they fully exploit all the features of AR, those features that make AR a new and unique communication tool.
We will have to ask ourselves: what does it mean to have, in augmented reality, a course on a certain topic? Are we really talking about the same course we would have if that course were online, or in a real frontal class? What are the differences between those courses that present “the same material” through different media? Will a form of communication that is appropriate in a certain medium be equally appropriate in augmented reality? What is the added value that augmented reality can bring to the course?

[1] R. Azuma, “A survey of augmented reality,” Presence Teleoperators Virtual Environ., vol. 21, no. 6, pp. 34–47, 1997.
[2] R. Azuma, “Location-Based Mixed and AR Storytelling,” 2015.
[3] W. Barfield, “Wearable Computers and Augmented Reality: Musings and Future Directions.” Fundamentals of Wearable Computers and Augmented Reality, Second Edition, London: CRl Press, pp. 3–11, 2015.
[4] “Microsoft HoloLens | Mixed Reality Technology.”
[5] “Meta – VR headsets, smart displays and AR glasses,” Meta.
[6], “Magic Leap 2.”
[7] “Inside Reality Labs Research,” Tech at Meta, Nov. 16, 2021.
[8] M. Akcayir and G. Akcayir, “Advantages and challenges associated with augmented reality for education: A systematic review of the literature,” Educ. Res. Rev., vol. 20, pp. 1–11, 2017.
[9] B. Ralhan, “AR: Transform your surroundings to learn anytime, anywhere!,” 2017.
[10] S. Cicconi and M. Marchese, “Analysis of an E-Learning Augmented Environment: A Semiotic Approach to Augmented Reality Applications,” in ICERI2019 Proceedings, Sevilla, Nov. 2019, pp. 4921–4931. doi: 10.21125/iceri.2019.1204.
[11] R. R. Bartie, Designing Virtual Worlds. 2003.
[12] J. Bacca, S. Baldiris, and et. al., “Augmented reality trends in education: a systematic review of research and applications,” J. Educ. Technol. Soc., vol. 17, no. 4, p. 133, 2014.
[13] M. Kljun, V. Geroimenko, and K. Čopič Pucihar, “Augmented Reality in Education: Current Status and Advancement of the Field,” in Augmented Reality in Education, V. Geroimenko, Ed. Cham: Springer International Publishing, 2020, pp. 3–21. doi: 10.1007/978-3-030-42156-4_1.
[14] A. W. (Tony) Bates, Teaching in a digital age. 2015. [Online]. Available:
[15] J. Dirksen, Design for how people learn. New Riders, 2015.
[16] R. Gagne and L. Briggs, Principles of instructional design., 4th ed. Holt, Rinehart & Winston, 1974.
[17] W. Horton, E-learning by design, 2nd edition. 2012.
[18] W. Horton and K. Horton, E-learning tools and technologies: A consumer’s guide for trainers, teachers, educators, and instructional designers. John Wiley & Sons, 2003.
[19] W. W. Lee and D. L. Owens, Multimedia-based instructional design: computer-based training, web-based training, distance broadcast training. John Wiley & Sons, 2004.
[20] J. H. Murray, Hamlet on the holodeck: The future of Narrative in Cyberspace. The MIT Press, 2016.
[21] F. Lunghezzani, D. Princivalle, P. Camagni, and R. Nikolassy, Clippy Zoom: Corso di Informatica per il Primo Biennio. Hoepli, 2011.
[22] F. Tomasi, Metodologie Informatiche e Discipline Umanistiche. Carocci Editore, 2008.
[23] R. White and T. Downs, How computers work, 10th edition. Que Corp., 2014.
[24] J. D. Bolter and R. Grusin, Remediation. Understanding New Media. MIT Press, 1999. doi: 10.5860/CHOICE.36-4895.