Innovation in Optimization of Virtual Space Experience Using Interactive Engine and Device —Example of a Song Dynasty Landscape Painting

Computer visual simulation has been an important digital tool in environmental planning, design, learning, and cognition since digital tools began to be used in the landscape architecture industry in the 1960s. In recent years, computer software and hardware technologies have developed rapidly, especially in virtual reality and interactive design tasks. However, regarding architecture and related designs, users are most of the time still passively viewing the results of prerendered scenes, and designers usually produce scenes into a movie, a picture, or postproduction visuals for users to view, which however do not come with functions for instant interaction between users and scenes. Thus, the purpose of this study is to establish a preliminary model with a series of operating methods and processes for visual landscape simulation using an appropriate interactive engine to optimize the virtual space experience. Through the actual operating process using a Song Dynasty (960–1279) landscape painting as an example, we carry out the following five steps: (1) interactive software and engine analysis, (2) ancient landscape painting interpretation, (3) three-dimensional computer model and material reconstruction, (4) digital model conversion and modification, and (5) visual effect and interactive outcome optimization using sensors.


Introduction
Since the first use of digital tools in landscape architecture and design industries in the 1960s, digitalization has brought about extensive changes in these fields. One major contribution of digital tools is enhanced design understanding and communication between designers and users. (1) In 1963, the first geographic information system (GIS) was born, and then in 1965, Harvard University's Computer Graphics and Spatial Analysis Laboratory was established to develop graphics software, which promoted the subsequent development of GIS software. (2) In 1969, Ian McHarg in his Design with Nature proposed the concept of several modern GIS analyses including overlay analysis. (3) However, not until 1981 was the commercial GIS software ARC/INFO and, in 1982, the graphics software AutoCAD made available. Digitalization has changed the way landscape planning and design are represented. (4) Computer vision simulation is an important digital tool in landscape design. A typical method is to prerender it into a movie or a picture using drawing software or use the techniques of matte painting to create a rendering using drawing software. In recent years, computer software and hardware technology have developed rapidly, and game engines can simulate extremely realistic and interactive scenes. In this study, we apply the Unity game engine to create virtual reality for instant, interactive, and highly realistic visual simulation, (5) using gardens in Song Dynasty (960-1279) paintings of the 12th century as an example.
The ruled-line painting is a special style of Chinese painting involving the drawing of even and straight lines, which is suitable for illustrating buildings. (6) Some Song Dynasty paintings accurately depict the characteristics of architecture and scene objects that are worthy of appreciation; however, the actual Song Dynasty gardens are gone, unlike those built in the Ming (1368-1644) and Qing (1636-1912) Dynasties, which have become historical sites available for visits. Therefore, one of the ways to experience a garden of the Song Dynasty is to recreate scenes into virtual models in accordance with paintings to reproduce the historical garden.
The graphics technology of computer games could not achieve real-time three-dimensional (3D) rendering until the introduction of the Quake engine in 1996. Therefore, related research on landscape visual simulation by game engines gradually appeared after advent of the 21st century. (7) For a comparison with the similar digital tools and studies, there are many practical examples of visual simulation applications, such as Lumion and Twinmotion, which are rendering software based on game engines. However, the application of game-engine-based rendering software operations is very different from that of direct-operation game-engine rendering in terms of, for example, the achievable effect, the interactive model, and the hardware requirement.
Current research on the direct use of game engines for visual simulation of landscapes mainly focuses on the introduction of pure GIS data into the game engine and its visual display. For example, Buyuksaliha et al. (4) provided cases of solar power in Istanbul using the Unity engine. Herrlich used CryENGINE to automatically generate scene data from GIS, but research on visual simulation of design cases using game engines, as well as on how the Unity engine automatically generates scenes from GIS, (8) was lacking, other than a study of how to import terrain data from the GIS into the Unity engine. Thus, our research objective is to establish a series of operational processes for the visual simulation of landscapes using game engines and to create a 3D model by adopting a Song Dynasty garden painting as the source of data to be input to the game engine to produce a set of innovative virtual reality software for display. The characteristics of the Song Dynasty garden and the knowledge on ruled-line landscape painting can be presented vividly and interactively in the setting of the painting.

Materials and Methods
The possibility of using a game engine for visual landscape simulation has been explored globally. Herwig and Paar pointed out that in computer games, virtual environments that are astonishingly realistic have been developed, and this trend is currently changing GIS and other drawing software. (9) In an experimental project, professional users are aware that they are using the A5 game engines Unreal and Conitec to show the planning of the outskirts of Strausberg, Germany. Herrlich pointed out that although the game engine has many advantages, it has some shortcomings compared with professional visual software. (8) For example, the rendering of the game engine is focused on visual effects and immediacy but lacks physical correctness.

Interactive engine
There are several sets of rendering software based on game engines, such as Lumion (from the Quest3D engine) and Twinmotion (from the Unreal engine), which are software typically developed using different game engines. Such rendering software is mainly developed for instant visualization synchronized with the modeling software and for architectural and landscape design. Taking Lumion's LiveSync function as an example, users can modify the model in Sketchup while rendering it in Lumion, and synchronously reflect the modified result. In addition to instant visualization, this type of rendering software differs from traditional rendering software in that it is developed for building and landscape design and for functions such as measuring tools or reading real-world maps and architectural volumes ( Fig. 1). Since such rendering software is developed using a game engine, some functions are augmented by components of the game engine, such as a sound engine, physics engine, and artificial intelligence. The traditional interpretation software only processes rendering and does not have such components. (10) It has been pointed out that in the computer game industry, interest in the so-called Serious Game, in which the technology of computer games is applied to science or education, is growing. (11) In one experiment, researchers transferred the data of terrain and surface vegetation distribution from GIS to CryENGINE, and used the sandbox editor of the game engine to automatically generate surface maps in accordance with different vegetation types and plant the vegetation, demonstrating the automated use of GIS data to generate terrain and vegetation in a scene. (12) Many studies are focused on how to generate scenes using game engines with GIS data or to test the effects of real-time visual simulation, whereas Buyuksaliha et al. (4) used the Unity engine to combine urban volume models with solar power generation potential. The information gained shows the potential for solar power generation and results for different time units that can be used to set reference data for solar panels. (13) Fig. 1. Process of creating 3D landscape simulation.

Ruled-line painting
The Song Dynasty was a brilliant age in Chinese history and lasted for 320 years. It can be subdivided into the Northern and Southern Song Dynasties. During the Song Dynasty, royal painters of the court focused on the pursuit of realism and detail, and ruled-line paintings were the ultimate expression of architectural structure. (14) The perspective or projection method in the ruled-line painting is usually the one-point perspective, scatter perspective, reverse perspective, parallel projection, or oblique projection, and the choice of perspective or projection is usually related to the shape of the painting. Although the rigorous definition of ruled-line painting is the depiction of buildings or vehicles using ruled-line pens and rulers, the lines in Song Dynasty paintings were drawn by hand, so the drawing method can be categorized into freehand drawing. (15) The lines are either drawn using a ruler, the appearance of which is shown in Fig. 2.
Works on architecture and gardens in the Song Dynasty paintings are rare. The Southern Song Dynasty Architecture in Song Paintings by B. X. Fu (14) contains a large number of paintings and explores the characteristics of architecture and gardens in the Southern Song Dynasty, (16) such as the garden of a house, the flexible layout, changes in the building decoration, the promenade, and a pavilion completely built over the water. Some examples of Song Dynasty paintings were based on gardens existing at that time. For example, Zhang Xian's "Shiyan Map" shows a corner of the Wuxing South Garden. It can be seen that the four corners of the waterfront are opposite the pavilion built along the water.
The paintings named "The Sycamore Garden Map" and "Songyin Courtyard Map" show that there are exquisite square flower beds in the courtyard for placing and planting trees and valuable stones. Other paintings such as the famous "Tongyin Moon Picture Page" show pots, planted lotus flowers and hexagonal flower stands with trees, and lake stones in the garden. The above three works are all part of a residential courtyard. It can be seen that in the Song Dynasty, the courtyard was planted with potted plants or had flower beds and separately placed lake stones, but no stacked stones appear in the Song Dynasty paintings. In the Northern Song Dynasty, the book "Luoyang Mingyuan Ji" recorded that only bridges were built with stacked stones, while the mountains were depicted as earthen. (17) It is speculated that stacked stones in the Song Dynasty are not universal and seems to be limited to royal imprisonment.
"Four Scenes of Landscapes" by Liu Songnian, a court painter of the Southern Song Dynasty, shows a private villa in Lin-an in the Southern Song Dynasty. It can be seen that the building complex is not only more freely combined, but also has a landscaped layout. The facade of the building is mostly lattice windows or wide open to show more perspective. Within the building complex, Xia Jing shows a pavilion that is completely separate from the water. This is often seen in Song Dynasty paintings but is rare in the Ming and Qing Dynasties. M. Lin's "Butterfly Night Tour" depicts a corner of the palace garden on a moonlit night. There is a hexagonal spire pavilion with a corridor attached to it. The candlesticks are juxtaposed in front of the pavilion. This is a gallery connecting buildings in the Song Dynasty. There are many examples of this, and in some paintings, there are also multistory corridors in an open space or on the water. For example, W. Li's "Moonlight Watching the Moon" and S. Zhu's "Liu Fengshui" seem to indicate that the use of the corridor is later in the Song Dynasty gardens than in the Ming and Qing Dynasties. More extensively, it can be concluded from the above related literature that the Song Dynasty paintings reflect some of the characteristics of gardens at that time, such as the color of the houses depicted with the gardens, relatively free planes of buildings, buildings that are completely independent of the water surface, and earthen mountains and lakes placed separately for better appreciation. The villa is characterized by no folding mountains and extensive use of corridors to connect buildings (Fig. 3).

Results and Discussion
The ruled-line painting selected for this study is the "Four Scenes of Landscapes-Autumn", because we can see a more complete layout in this particular painting, instead of a drawing of only one corner of the garden as in many paintings, so we are able to analyze the various landscape elements in the painting, and finally draw the plan-view configuration for subsequent modeling. The implementation of the main practical process of rendering the scene can be  interactive in the game engine to verify its possibilities. The operation mainly transforms the planar configuration model into a stereo model, converts it into a format suitable for the game engine, and then imports it into the game engine for related settings. Finally, the entire database is output with the virtual reality software, and the operation details are described through the implementation process for reference. If there is a construction or other structure involved in the modeling process, it must be restored by referring to the book "Building Method" or other related literature and accurately and meticulously modeling the details of the structure to simulate the building of that time.

Target painting analysis
Liu Songnian painted small landscape gardens and was one of the representative painters of courtyard paintings. "Four Scenes of Landscapes" depicts the scenery of a villa of the scholar-official class of the West Lake, Hangzhou.
Regarding the building, the paintings clearly depict the appearance of houses, fences, railings, doors, pavilions, bridges, and foundations. One of the major features of the house is that the roof type is the so-called suspended hill top, which is a special style of roof in the Song Dynasty. This type of roof can still be seen in modern-day houses, such as the former residence of Su Buqing, a renowned Chinese mathematician, in Pingyang, Zhejiang province. The façade of the house comprises two inner and outer lattice windows. The inner floor lattice window has been removed, and the outer lattice window is divided into upper and lower parts. The lower half is detachable and has been removed. Only the upper half is fixed. The lattice window can be disassembled in accordance with the season to maintain warmth in winter and provide coolness in summer.

Site plan renovation
Because the layout cannot be seen in the painting, the overall layout has been slightly adjusted, so it may be slightly different from the description in the original painting. The terrain is referenced to the slope settings that may appear in the West Lake and is as close as possible to the conditions described in the painting (Fig. 4).
According to the plane configuration diagram, the scene contains seven elements: (1) landform, (2) base structure, (3) building, (4) paving, (5) water features, (6) plants, and (7) others, which are classified in accordance with the classification of landscape elements in the book Basic Elements of Landscape Architectural Design by N. K. Booth. (18) However, the breakdown of each category is not classified following the design method in the book because of the difference between design techniques and construction methods. The elements in the scene will be sequentially produced and demonstrated in a series of systematic processes in the Unity engine, and some of the various landscape elements will not appear in this implementation (such as fountains), but they will be processed differently from that of other similar elements. In this study, we applied these seven elements.
In the painting, two large buildings can be seen, and there is a presumed use of a plurality of suspended hills that are parallel or perpendicular to each other. The buildings that are covered behind are presumed to locate by the hanging hills on the waterfront of West Lake and the seat is indoors in the hall. The stacked stones behind the water bank of West Lake are speculated from the painting that the plane of the building is relatively simple and rectangular, whereas the front hall is more complicated, not only the non-cross-orthogonal huts, but also the height of their respective roofs. The hall itself seems to be surrounded by two layers of lattice windows. Although the façade of the waterside building cannot be seen in the painting, the raft that can be seen in the water also has a double-layer lattice window, and there is a hook in the middle of the two-layer lattice window on the water side. The urns of the two houses are also invisible in the paintings, so it is assumed that there are straight mullion windows.
Furthermore, because the drawing is hand-painted, old, and historical, important construction materials are faded and unidentifiable, so much archaeological work must be carried out for verification. Parts such as tiles, bricks, and small pieces of woodwork are made in accordance with the provisions of the "Building Law". The foundation of the building is made of bricks, the corners are laid with corner pillars, and the surrounding wall is whitewashed above the bricks, which is in line with the method of alkali isolation under the wall to prevent loess salinization. The building materials and construction methods of the Song Dynasty are brought to life through the use of interactive software in this study.

Virtual reality representation
One of the most important parts of interacting with the scene is to give the user the ability to freely browse the scene. In this implementation, designers must set a character that can move freely. The user can control the character to browse within the scene and interact with the objects in the scene. In Unity's official assets package, there is a first-person role model that has the relevant code. The Prefab named FPSController in the Prefabs folder of FirstPersonCharacter under Characters can be dragged directly into the scene. This character can be moved after the game is executed (Fig. 5).
In addition to the character itself, designers must set the interaction between objects in the scene. For example, the most basic physical collision prevents the character from penetrating a scene object or going where it should not. The sunlight needs to be suitably located, and its position should be set to detect collision that blocks its path. In this implementation, there is also an interactive commentary and viewing section. (19) When the user is close to certain objects, a sensor can trigger a guided commentary or a special lens that presents a unique view of the landscape in the scene. To achieve this, designers must set the Collider at the appropriate position in the scene and modify coding for sensors (Fig. 6).

Conclusions
In this study, we attempted to explore the feasibility and practice of interactive visual simulation using a game engine and to output the results using virtual reality software as a demonstration. The progress of construction and methods may be better visualized and discussed, and sensors embedded in the interactive engine may enable users to vividly experience the architecture and landscape in the Song Dynasty on the basis of the scenes transformed from a ruled-line painting from the Song Dynasty. In practice, some key points are as follows. (1) To improve the rendering performance, it is necessary to convert the created picture into a scene using the game engine in real time.
(2) Compared with other modeling and rendering software, the game engine operation is different in some points; designers must deal with subjects that traditional rendering software usually does not handle, such as a physics engine, coding, and the user interface. (3) We were able to produce a good scene model to be displayed by the game engine and to import interactive elements into it. (4) Users can virtually experience a tour of the scene that introduces relevant knowledge of Song Dynasty ruled-line paintings. (5) The rendering performance met the hardware requirements and showed good visual effects, demonstrating the feasibility of interactive visual simulation using a game engine.
Finally, during the exhibition period, more than one hundred users verified that they enjoyed a positive experience and testified to the effectiveness of this virtual reality system, including the good quality of scene rendering, vivid spatial perception, and the fun of learning professional knowledge through the virtual interactive tour.