A STUDY OF THE USE OF MIXED REALITY FOR CAPTURING HUMAN OBSERVATION AND INFERENCES IN PRODUCTION ENVIRONMENTS

Milos Stojkovic, Rajko Turudija, Milan Trifunovic, Marko Pavlovic, Ivan Jovanovic, Nenad Uzelac, Vladeta Milenkovic

DOI Number
10.22190/FUME220714047S
First page
Last page

Abstract


Augmented and mixed reality is already considered as needful technology of the modern production systems. It is primarily employed to virtualize proper digital content, mainly related to 3D objects, into the human visual field allowing people to visualize and understand complex spatial shapes, their mutual relations, and positioning. Yet, the huge potential of the technology is waiting to be revealed in its usage for collecting and recording human observations and inferences about the context of the production environment. Its bi-directional interface makes it the most direct and the most efficient knowledge capturing means to date. The paper presents the challenges and benefits that come from the usage of a conceptual interface of an mixed reality application that is designed to collect data, semantics and knowledge about the production context directly from the man-in-process. As a production environment for the development, implementation, and testing of mixed reality applications for this purpose, various processes for the assembly and maintenance of medium-voltage equipment were used.

Keywords

Augmented Reality, Mixed Reality, Knowledge capturing, Industry 4.0, Human-to-Machine Communication

Full Text:

PDF

References


Pace, F.D., Manuri, F., Sanna, A., 2018, Augmented Reality in Industry 4.0., American Journal of Computer Science and Information Technology, 6(1), doi: 10.21767/2349-3917.100017

Sorko, S.R., Brunnhofer, M., 2019, Potential of Augmented Reality in Training, Procedia Manufacturing, 31, pp. 85-90.

Gorski, F., Grajewski, D., Bun, P., Zawadzki, P., 2021, Study of Interaction Methods in Virtual Electrician Training, IEEE Access, 9, pp. 118242-118252.

Mourtzis, D., Zogopoulos, V., Katagis, I., Lagios, P., 2018, Augmented Reality based Visualization of CAM Instructions towards Industry 4.0 paradigm: a CNC Bending Machine case study, Proceedings of the 28th CIRP Design Conference, 70, pp. 368-373.

Mourtzis, D., Siatras, V., Zogopoulos, V., 2020, Augmented Reality Visualization of Production Scheduling and Monitoring, Procedia CIRP, 88, pp. 151-156.

Mumtaz, S., Alsohaily, A., Pang, Z., Rayes, A., Tsang, K.F., Rodriguez, J., 2017, Massive Internet of Things for industrial applications: Addressing wireless IIoT connectivity challenges and ecosystem fragmentation, IEEE Industrial Electronics Magazine, 11(1), pp. 28-33.

Meng, Z., Wu, Z., Gray, J., 2017, A Collaboration-Oriented M2M Messaging Mechanism for the Collaborative Automation between Machines in Future Industrial Networks, Sensors, 17(11), 2694.

Berges, I., Ramírez-Durán, V.J., Illarramendi, I., 2021, A Semantic Approach for Big Data Exploration in Industry 4.0, Big Data Research, 25, 100222.

Ceravolo, P., Azzini, A., Angelini, M., Catarci, T., Cudré-Mauroux, P., Damiani, E., Mazak, A., Van Keulen, M., Jarrar, M., Santucci, G., Sattler, K.U., Scannapieco, M., Wimmer, M., Wrembel, R., Zaraket, F., 2018, Big Data Semantics, Journal on Data Semantics, 7, pp. 65-85.

Jain, V., Aggarwal, S., Mehta, S., Hebbalaguppe, R., 2019, Synthetic video generation for robust hand gesture recognition in augmented reality applications, ICCV 2019 Workshop: The 5th International Workshop on Observing And Understanding Hands In Action, arXiv preprint arXiv: 1911.01320.

Kolsch, M., Bane, R., Hollerer, T., Turk, M., 2006, Multimodal interaction with a wearable augmented reality system, IEEE Computer Graphics and Applications, 26(3), pp. 62-71.

Aouam, D., Benbelkacem, S., Zenati, N., Zakaria, S., Meftah, Z., 2018, Voice-based Augmented Reality Interactive System for Car’s Components Assembly, Proceedings of the 3rd International Conference on Pattern Analysis and Intelligent Systems (PAIS), Tebessa, Algeria, pp. 1-5.

Bertoli, P., Corcoglioniti, F., Francescomarino, C.D., Dragoni, M., Ghidini, C., Pistore, M., 2022, Semantic modeling and analysis of complex data-aware processes and their executions, Expert Systems with Applications, 198, 116702.

Paulus, A., Burgdorf, A., Pomp, A., Meisen, T., 2021, Recent Advances and Future Challenges of Semantic Modeling, Proceedings of the IEEE 15th International Conference on Semantic Computing (ICSC), Laguna Hills, CA, USA, pp. 70-75.

Bottani, E., Vignali, G., 2019, Augmented reality technology in the manufacturing industry: A review of the last decade, IISE Transaction, 51(3), pp. 284-310.

Marinkovic, D., Zehn, M., 2019, Survey of finite element method-based real-time simulations, Applied Sciences, 9(14), 2775.

Marinkovic, D., Zehn, M., Rama, G., 2018, Towards real-time simulation of deformable structures by means of co-rotational finite element formulation, Meccanica, 53(11-12), pp. 3123-3136.

Zhou, F., Duh, H.B.L., Billinghurst, M., 2008, Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR, Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 193-202.

Kim, K., Billinghurst, M., Bruder, G., Duh, H.B.L., Welch, G. F., 2018, Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008–2017), IEEE Transactions on Visualization and Computer Graphics, 24(11), pp. 2947-2962.

Ceruti, A., Marzocca, P., Liverani, A., Bil, C., 2019, Maintenance in aeronautics in an Industry 4.0 context: The role of Augmented Reality and Additive Manufacturing, Journal of Computational Design and Engineering, 6(4), pp. 516-526.

Masoni, R., Ferrise, F., Bordegoni, M., Gattullo, M., Uva, A.E., Fiorentino, M., Carrabba, E., Di Donato, M., 2017, Supporting Remote Maintenance in Industry 4.0 through Augmented Reality, Procedia Manufacturing, 11, pp. 1296-1302.

Pierdicca, R., Frontoni, E., Pollini, R., Trani, M., Verdini, L., 2017, The Use of Augmented Reality Glasses for the Application in Industry 4.0, Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics, 10324, pp. 389-401.

Cukovic, S., Devedzic, G., Ghionea, I., Fiorentino, M., Subburaj, K., 2016, Engineering design education for industry 4.0: Implementation of Augmented Reality concept in teaching CAD courses, Proceedings of the Augmented Reality for Technical Entrepreneurs International Conference – [ARTE2016], Bucharest, Romania, pp. 11-16.

Maly, I., Sedlacek, D., Leitao, P., 2016, Augmented reality experiments with industrial robot in industry 4.0 environment, Proceedings of the IEEE 14th International Conference on Industrial Informatics (INDIN), Poitiers, France, pp. 176-181.

Doshi, A., Smith, R.T., Thomas, B.H., Bouras, C., 2017, Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing, The International Journal of Advanced Manufacturing Technology, 89(5-8), pp. 1279-1293.

Flotynski, J., Sobocinski, P., Strykowski, S., Strugala, D., Bun, P., Gorski, F., Walezak, K., 2021, Semantic Representation of Domain Knowledge for Professional VR Training, Proceedings of the 24th International Conference on Business Information Systems, 1, pp. 139-150.

Gorski, F., 2017, Building Virtual Reality Applications for Engineering with Knowledge-based Approach, Management and Production Engineering Review, 8(4), pp. 64-73.

Fernández del Amo, I., Erkoyuncu, J.A., Roy, R., Wilding, S., 2018, Augmented Reality in Maintenance: An information-centred design framework, Procedia Manufacturing, 19, pp. 148-155.

Mourtzis, D., Vlachou, E., Zogopoulos, V., Fotini, X., 2017, Integrated Production and Maintenance Scheduling Through Machine Monitoring and Augmented Reality: An Industry 4.0 Approach, Proceedings of the IFIP International Conference on Advances in Production Management Systems, Hamburg, Germany, 513, pp. 354-362.

Bruno, F., Barbieri, L., Marino, E., Muzzupappa, M., D’Oriano, L., Colacino, B., 2019, An augmented reality tool to detect and annotate design variations in an Industry 4.0 approach, The International Journal of Advanced Manufacturing Technology, 105(1-4), pp. 875-887.

Zhao, G., Liu, S., Zhu, W.J., Qi, Y.H., 2021, A Lightweight Mobile Outdoor Augmented Reality Method Using Deep Learning and Knowledge Modeling for Scene Perception to Improve Learning Experience, International Journal of Human–Computer Interaction, 37(9), pp. 884-901.

Johnson, T.L., Fletcher, S.R., Baker, W., Charles, R.L., 2019, How and why we need to capture tacit knowledge in manufacturing: Case studies of visual inspection, Applied Ergonomics, 74, pp. 1-9.

Fominykh, M., Wild, F., Smith, C., Alvarez, V., Morozov, M., 2015, An Overview of Capturing Live Experience with Virtual and Augmented Reality, Proceedings of the 11th International Conference on Intelligent Environments, 19, pp. 298-305.

Dudek A., Patalas-Maliszewska, J., 2016, A Model of a Tacit Knowledge Transformation for the Service Department in a Manufacturing Company: A Case Study, Foundations of Management, 8(1), pp. 175-188

Hashimoto, J., Park, H.J., 2020, Capturing tacit knowledge with smart device augmented reality (SDAR), Proceedings of the ECAADE 2020, Berlin, Germany, pp. 165-172.

Stojkovic, M., Trifunovic, M., Misic, D., Manic, M., 2015, Towards Analogy-Based Reasoning in Semantic Network, Computer Science and Information Systems, 12(3), pp. 979-1008.

Trifunovic, M., Stojkovic, M., Misic, D., Trajanovic, M., Manic, M., 2015, Recognizing Topological Analogy in Semantic Network, International Journal of Artificial Intelligence Tools, 24(3), pp. 1550006-1 - 1550006-25.


Refbacks

  • There are currently no refbacks.


ISSN: 0354-2025 (Print)

ISSN: 2335-0164 (Online)

COBISS.SR-ID 98732551

ZDB-ID: 2766459-4