Fabricating Behavior Sensor Computing Approach for Coexisting Design Environment

1Idea Factory, National Yunlin University of Science & Technology, 640 No. 123, University Rd., Section 3, Douliou, Yunlin, Taiwan, R.O.C. 2Dept. of Digital Media Design, National Yunlin University of Science & Technology, 640 No. 123, University Rd., Section 3, Douliou, Yunlin, Taiwan, R.O.C. 3School of Architecture, Tsinghua University, No. 30 Shuangqing Rd., Haidian, Beijing, China. 4International Faculty of Science and Engineering at Curtin University, 6102 Kent Street, Bentley, Perth, Western Australia, Australia 5Dept. of Electrical Engineering, National Yunlin University of Science & Technology, 640 No. 123, University Rd., Section 3, Douliou, Yunlin, Taiwan, R.O.C.


Introduction
Digital fabrication often requires a mixed initial interaction among the fabrication tools and the information behind it that is hard to align with users owing to the nature of a coexisting environment. (1,2) Thus, sensor devices are indispensable in a coexisting environment, not only to obtain the static environment properties but also to reflect the feedback of the dynamic situation. Therefore, we focus on the development of an effective sensing system to continuously format the human behavior pattern into discrete variables.
In addition, the traditional sensors built for robotic arms are very expensive. These sensors are generally used for very tough remote conditions, such as to dismantle explosives or to explore the deep ocean. However, these sensors may not be very practical in human-machine collaboration circumstances. In Refs. 3 and 4, we proposed the coexisting fabrication system (CoFabs), a fabrication process that combines visualization sensors and behavior feedback based on an agent-based sensor-computing approach. Within CoFabs, the user can explain the reasons behind design decisions by unleashing the process and visualization associated with the process. It includes a coexisting space representation. (5,6) This allows the modeling of such interaction from cyber to physical spaces. (7,8) CoFabs uses media feedback for design and fabrication. When the fabrication process is too dangerous and requires the professional division of labor, the user can constantly revise by observation. It takes a lot of time to consult, wait, and manufacture during this process. Thus, the digital fabrication process needs a new hybrid sensor that embeds an agent system within the virtual and physical environments as a coexisting environment.
CoFabs proposed a control mechanism called fabricating behavior sensor (FBS), which improves the accuracy of sensing and helps users manage the process between the virtual data and physical environments. We intend to use gestures, behaviors, and interfaces to improve the precision of the fabrication process, and provide users with a smoothness controlling experiences.

Methods and Technology
The CoFabs process integrates both real-world and virtual environments based on the digital twin (DT) concept of a system, where physical entities and virtual information are referenced to each other in a recursive way through a series of physical changes, information analysis, and generative fabrication suggestions. In this way, the workflow is optimized. (9) To integrate the virtual and the physical, we need to combine a camera and infrared sensor in FBS. We use the "Seeing-Moving-Seeing" design-thinking model to help users refine their ideas. (10) This process of FBS can be separated into two recursive sequences. One is to generate a virtual interface layer on a physical user perspective view to build a corresponding environment understandable for users. The other is to sense users' gestures and movements to control the robotic arms executing the manufacture or fabrication.
Users can observe the subject in the physical space, interact with the virtual objects, then realign the design properties by comparing the corresponding mixture environments and virtual model during the fabrication process.
To render the model and generate the model interface in physical space, we use a Grasshopper plug-in called Fologram, to convert the geometric information into the FBS, and execute the robotic end effector to move along the path. (11) The FBS for the user interface (UI) and the computer server are built for fabrication tool code generation. The FBS can communicate by scanning a QR code, thus enabling the virtual model in the FBS to generate data strings and pass the data back to Grasshopper. Following this, we code several C# scripts to execute some custom fabrication calculations, such as moving path calculations, which contain geometric data, behavior variables, and remapping parameters between the fabrication tools and the FBS UI system (Fig. 1).

Implementation of FBS
In this paper, we use a mixed reality (MR) device to control the physical machine for fabrication and assembly. We use a weaving structure project for our test. (12) It is composed of continuous elastic members, such as bamboo or fiber-reinforced plastic (FRP) rods, which can bear bending forces. (13) The advantage of the weaving structure system lies in its formal representation, structural performance, and construction method. The Seeing-Moving-Seeing design-thinking model provides different observation methods so as to prototype and interactively revise in real time. It requires a lot of training hours for simulating and assembling, and in order to manage the workflow properly the worker leader needs a system to minimize the assembling crew and simplify the fabrication process.
Thus, we build a virtual interface through the Grasshopper plug-in and simulate the weaving structure on a wearable device. Users can use this wearable device to confirm the relative properties of assembling components and manage the fabricating process. It compares the information of the simulation process with the actual process to identify any problems within the process. These problems can then be revised in real time.

Sensor control on the FBS
MR is an integration environment, where both real and virtual worlds coexist and correspond to each other in real time. (14)(15)(16)(17)(18) There is a wireless, augmented reality, head-mounted, MR display device called HoloLens. (19) Thus, we implement the FBS through HoloLens to assist in the fabrication process; we use this MR interface to sense human behavior, (20) and we use a physical control system for fabrication and assembling. In this paper, the FBS based on HoloLens is developed to expand the CoFabs. Currently, we focus our research on the HoloLens graphical user interface (GUI) and interactive behavior. We import the digital model to HoloLens and use visual technology to establish an immersive environment that matches the user's perspective and use operative gestures to accomplish the fabrication process. To finish the HoloLens GUI and interactive behavior of the fabrication tool, we divide the sensor and feedback thread into four parts: selector identification, data retrieval, gesture confirmation, and custom-method creation.
• Selector identification: Use the ray cast at the center of the sight view to apply the selector category in the environment thread and judge if there is a collider event with the virtual object. • Data retrieval: Listen constantly to the selector's selected event through sight view and use HoloLens to display the data of the selected objects to help the user make a design decision. • Gesture confirmation: Confirm the detected states of the index finger and thumb gestures as clicks or perform a dragging motion. • Custom-method creation: Code a customized Grasshopper plug-in script with gesture behavior, taking the weaving structure as an example, to lock, unlock, or move the nodes on the weaving structure.

The CoFabs process
Although a virtual graphic computation system allows users to see the combined digital model and the physical environment in an efficient way, it needs detection tools to ensure that the material is simulated in a reasonable way. We have to optimize the digital model when we operate the deformation of virtual objects in this coexisting fabrication process.
First, we use the Kangaroo plug-in to simulate the weaving structure and import the model into HoloLens. Then, we mark every node on the weaving structure to check that each node is in position. Every node can interact with one simple clicking gesture. This coexisting model will calculate every intersecting point through the plug-in script computation system once the nodes have been clicked. The data of the selected subject, such as the number of joints or the direction of the surrounding bending curve material, will be displayed. Finally, the coexisting model helps the user determine which location must be positioned precisely and also allows them to identify whether there is a need to use additional fabrication tools for different joint types [ Fig. 2(a)].
To make the weaving structure of the fabrication process easier, we need to optimize the fabrication sequence. The construction of the weaving structure needs to be defined via an initial point cloud, and this coexisting model shows users the next reasonable joints in a step-bystep process. There are four states in each joint: choose, near, finished, and possible [Fig 2(b)].
• Choose state: Tie up in the space after selecting the node (red).
• Near state: Calculate the surrounding approach point according to the computing system after selecting the node, thereby sorting the next to tie up nodes (blue). • Finished state: Tie up the material in actuality by participants after the first state; this makes the simulation and process easier to match (white).
• Possible state: Hide the nodes as quantum states while the nodes have not yet passed by the computing system so as to avoid judgment errors in the process. In a step-by-step manner, users can build the weaving structure through this process of sensing and progress. Through this coexisting fabrication sequence, designers can operate by matching, adjusting, and decision-making through the virtual and real states of this weaving structure.

Material Evaluation
Through the model, we can check the position of every node after completing the weaving structure. Users can select node, then drag to revise the weaving structure. We discovered that every node changes its position, even if we just drag any node on this structure. Thus, we add a variable parameter (Pa) to express the degree of bending. As shown in Table 1, we adjust Pa to refer to the difference between the virtual and physical changes. We drag the selected point and observe the displacement of the point with an actual hand and measure the displacement via CAD software (Rhino). The dragged node will change the entire weaving structure owing to its material characteristics; in this case, Pa refers to the inner tension of FRP.
According to the results, the gesture shifted by 24 cm and the overall structure variable changed by 100.6 units when Pa was 100 [ Fig. 3(a)]. The gesture shifted by 21 cm and the overall structure variable changed by 178.7 units when Pa was 150, which significantly reduced the deviation of gesture displacement [ Fig. 3(b)]. The gesture shifted by 16 cm and the overall structure variable changed by 259.1 units when Pa was 200. This is the closest figure to the overall structure variable of the gesture displacement [ Fig. 3(c)]. Thus, we adjust the Pa to 200 to improve the accuracy and adjust the virtual data with the physical environment.
Finally, we tried to use FBS to build a practical weaving structure. We can see that the difference is very small between the virtual [ Fig. 4(a)] and physical weaving structures [Fig 4(b)]. The FBS increases the precision between the virtual and the physical.

Conclusions
We proposed the CoFabs framework to help fabrication threads, in order to match and to adjust the limited problem space. Team members can share the progress of the fabrication in an easy and clear way. Besides, users can not only retrieve the real-time DT model with this immersive perspective, but also manage the entire fabrication workflow.
We propose the FBS as a specific mechanism to improve the sensing accuracy and help users control the coexisting virtual and physical environment smoothly. We use some key parameter interventions to reduce the diversity of user differences, and ensure the virtual data and physical environment corresponding to each other properly.   Based on the Seeing-Moving-Seeing design-thinking model, we also encourage users to observe the finished product with different view scopes to help realign their models and environment. Thus, the CoFabs process is suitable for users, as it allows them to become immersed in an interactive coexisting environment.
In addition, the users can not only witness the progress of the model's creation but also determine the manufacturing details of the fabrication by prompting feedback. Designers can also modify several specific parameters to change the morphing relationship between the simulated model and real materials. Through this human-machine collaboration, we can make design decisions and fabrications more simple, intuitive, and communicable in real time.