Building Smart System by Applied Deep Learning and Spatial Indoor Agent Based Model for a New Adaptation University Learning Process Post Covid-19

Abstract

The impact of COVID-19 implied various restrictions on people’s mobility, especially for the higher education communities, by implementing the Learning from Home approach. This approach has altered the behavior of a human on a daily basis for a year long. Subsequently, the global vaccination program has been the advent of a “New Normal” approach as it reenables the direct human interactions by following health protocols to abide such as social distancing. This study investigated the pedestrian flow in the Department of Urban and Regional Planning (DURP) lecture building, Brawijaya University, and predicted the potential crowd spots using the Integrated Agent-Based Model (ABM), Computer Vision, and the Geographical Information System on an Indoor scale. Additionally, alternative designs of pedestrian flow were proposed to prevent crowds from occurring. The results showed the East and West entrance paths of the DURP building have high traffic, so the proper response is to organize the Southside door as an alternative entrance for pedestrian access. Moreover, the opening of the south gate could reduce the crowd spots on the 2nd Floor of the DURP lecture building.

1. Introduction

The rapid development of science on a global scale has affected the potential evolution of 3 groups of traditional scientific branches formal science, social science, and natural science [1, 2, 3, 4]. Nowadays, these scientific branches are associating with each other, thus forming new clusters, such as Social Sciences, Humanities, Arts for People and The Economic (SHAPE), and the Science-Technology-Engineering-Math (STEM). STEM cluster is formed by the combination of Formal Sciences (Mathematics and Statistics) with Natural Sciences (Biology, Physics, and Chemistry) [5]. In contrast, the SHAPE cluster is formed by the social life linked to scientific fields consisting of politics, psychology, and sociology [6, 7].

The merging of scientific clusters in regards to addressing the global issues related to human life still has some discrepancies. The gaps are still present between the natural science from the STEM cluster and the social science field from the SHAPE cluster. Therefore, the development of a new curriculum consisting of the combination of STEM and SHAPE clusters is proposed, namely Humanitarian Engineering (HE), through the ENHANCE project composed of the works of researchers from the Warwick University of U.K., along with the academics from Greece, Bangladesh, Vietnam, and Indonesia.

In the traditional engineering academic texts, it is challenging to find the term HE thus, some researchers have been trying to define it as the development of science progresses. Passino [8] stated that humanitarian engineering is an approach to constructing technologies that could assist the engineering in helping people, while VanderSteen [9] identified HE as a tool to solve social issues. Moreover, Hill and Miles [10] recognized HE as the solution to social problems by investigating the achievement of sustainability in developing countries. Therefore, this study regards HE as a scientific field that focuses on addressing complex humanitarian matters through the perspective of the SHAPE cluster using a STEM approach to propose smart, equitable, and harmonious solutions.

The purpose of this paper is to analyze the humanitarian engineering field through the micro-scale of the planning field by re-designing the pedestrian flow inside a lecture building concerning the new normal learning process to prevent the higher risk of COVID-19 transmission by avoiding the potential crowd spots using Integrated Agent-Based Model (ABM), Computer Vision, and the Geographical Information System on an Indoor scale.

2. Methods

Replace the entirety of this text with the main body of your chapter. The body is where the author explains experiments, presents and interprets data of one’s research. Authors are free to decide how the main body will be structured. However, you are required to have at least one heading. Please ensure that either British or American English is used consistently in your chapter.

This study aims to analyze the lecture building of the Department of Urban and Regional Planning, Brawijaya University, along with its surrounding environment. The descriptive and evaluative analysis is used to investigate the pedestrian’s flow, namely Integrated Agent-Based Model (ABM), Computer Vision, and the Geographical Information System on an Indoor scale. The descriptive statistical method is used to investigate the characteristic of the pedestrian consisting of movement, speed, and density. Moreover, evaluative analysis is taken to calculate the density of pedestrian traffic by utilizing the time series data of pedestrians’ peak volume during each working hour/day.

Decision making which utilizes images acquired from sensors is known as Computer Vision (CV) [11, 12, 13]. The purpose of the CV is to construct an intelligent machine with “see” ability. The Agent-Based Model (ABM) is known as an individual-centric and decentralized approach, whereas the modeler is tasked to pinpoint the agent or active entity (in this case, the person), characterize their behavior, detects agents in a specific environment, create connections in between, and establishes the simulation. Moreover, Geographical Information System (GIS) on an indoor scale is defined as a complete mapping system to make the disconnected project data practical, operate complex artificial environments, track indoor devices, evaluate space allocation in confined spaces, and recognize and react to the real-time events.

The detection of the pedestrian can be achieved by using various computer vision methods. One of the classical human detection methods was invented by the Voila-Jones algorithm, which aims to recognize human faces with a fast detection rate at the cost of low accuracy [14]. It is also revealed that the accuracy drops even more for non-frontal faces [15].

In regards to the development of human detection, the Histogram of Oriented Gradient (GOD) was later proposed in junction with linear Support Vector Machine (SVM), which offers an accuracy rate of up to 89% [16]. In addition, the drawback of the HOG algorithm is required expensive computational resources to operate [17].

Nevertheless, the rapid development of CV human detection through neural network algorithms utilizes the classical methods’ base concept, namely, You Only Look Once (YOLO) [18]. Single Shot Detection (SSD) [19], and Faster Region-based Convolutional Neural Network) (Faster R-CNN) [20]. Compared with classical image processing methods, improved robustness and reliability are expected from AI-based human detection [21].

The detection of the pedestrian in this study utilizes the You Only Look Once (YOLO) method. The YOLO algorithm was considered to be used regarding its reliability with a fast detection rate [22]. YOLO has been recognized to be able to outperform HOG-SVM. Therefore, it’s been widely used for many purposes, such as operating autonomous cars [23].

YOLO implementation was accessible on DARKNET (open source neural network). The idea of YOLO is to see the whole image once and later passes through the neural network once it immediately detects actual objects, thus later known as the name of the method, YOLO, from the abbreviation of You Only Look Once. The purpose of YOLO is to perform real-time object detection. A localizer or repurpose classifier is used by the detection system. A model is utilized for an image at various locations and scales. The highest-rated area image will be regarded as a detection.

YOLO utilizes the Artificial Neural Network (ANN) approach for object detection in an image by dividing the image into regions and predicting each bounding and probability box from each region. The bounding boxes are later compared with each expected probability. In addition to that, there are several advantages of YOLO compared to a classifier-oriented system. YOLO can carry out the test of the entire image with predictions informed globally on the image. YOLO is also several times faster than the Region Convolutional Neural Network (R-CNN) due to its ability to synthesize the neural network for making predictions than the R-CNN, which needs thousands of images to operate.

In order to represent the actual condition of pedestrian traffic and other geographic features, the ABM is assisted by spatial and geographic visualization data. Several ABM and GIS integration applications have been recognized at the macro scale, such as in the region and urban areas. Hartmann and Zerjav [24] revealed that the assimilation between ABM and GIS is proven effective in planning the optimum health service location concerning the nature of the urban population. The ABM and GIS are generally used for building simulation modeling, such as to estimate the impact of resource investment decisions concerning the health costs, population development of an area, and burden and spread of disease. Nonetheless, the integration of ABM at the microscale with GIS at the indoor scale is still limited. Therefore, to fill this gap, this study aimed to incorporate the main idea of CV as the novelty of the research.

3. Research results

This chapter has three sections: dataset development, human object detection process, and Agent-Based Model and Geographical Information System Indoor. The linkage between these sections will be investigated in the context of research steps for collecting the pedestrian behavior inside the DURP lecture building to rearrange pedestrian traffic to reduce the COVID-19 transmission.

3.1 Dataset development

Datasets collection is the first step before training the YOLO algorithm to recognize the object. The data collection can be conducted through video recording and later exported as images or downloading related pictures from the internet. Figure 1 describes the collection of the dataset before being used to train the YOLO algorithm.

Figure 1.

Video data source capture flow.

There are several criteria considered in data processing testing experiments, which consist of:

• Raspberry Pi 3

• Mobile Video Camera

• Video in .mp4 format

• 30 recorded videos condition in total.

• 320 × 240 pixels video resolution with a frame rate of ±20 frames/second.

• Videos were recorded in a stationary state.

• ± 10 seconds of the recoding time of each video.

• There are several conditions to be considered when recording the video, such as indoor environment, dark light (night), outdoors environment or lots of light illumination, and different background conditions.

A camera with a VGA resolution (320 × 240 pixels) and frame rate of ±20 frames/second was used to record the video through the Open Camera application. Static exposure value (camera’s sensitivity to light) conditions were used as the camera settings. The bright condition was recorded during the daytime outside the room and indoors with lights on, while the dark condition was recorded during the evening, and indoor lights were off. The recorded video has a duration of ±10 seconds, and pedestrians walk five meters away from the camera position. The different conditions for recording the video are being considered to investigate the lighting effects, environment (indoor or outdoor), and the presence of other objects than humans, such as shadows, would affect the detection results.

3.2 Human object detection process

The selection of human objects was conducted through blob analysis in MATLAB by examining the size of each object. The purpose of the blob analysis function is to determine the area of a human object based on the minimum and maximum blob area. The elimination of objects with a size of fewer than 10,000 pixels and larger than 980 pixels was considered to eliminate non-human objects. The other stages were conducted, including extraction of video frames, normalization of images, background subtraction, morphological operations, and object detection. Subsequently, the images extracted from the video frames were normalized. The image was later reconstructed in the form of opening, closing, and filling operations through a background subtraction approach and morphological operations. Specific values are determined at each stage for human object detection.

The conducted test results based on 30 videos were reported in the form of “correct” and “incorrect” information. The test could be considered appropriate if the system detection results match the manual calculation results and vice versa for inappropriate information results. Table 1 explains the human object detection testing, which revealed 11 incorrect tests out of 30 tests conducted. There were two tests with incorrect information from 10 video tests with bright lighting effects. On the contrary, the number of tests with incorrect information was higher in dark lighting effects, with nine out of 10 tests. These results revealed that the system could detect human (pedestrian) objects better in bright lighting than in darker lighting due to the distraction from the shadows, which were later misinterpreted as human objects. The other distractions that could affect the detection are light bias and other moving objects such as smoke.

Lightning condition	ΣTest	Detection result
		ΣTrue	ΣFalse
Bright	10	8	2
Dark	10	2	8
Low	10	1	9
Total	30	11	19

Table 1.

Human object detection testing.

The tests during dark conditions revealed that the poor lighting and unstable camera sensor caused the lower detection capability and exceeded the number of detected objects from the manual calculation. Therefore, the DURP lecture building was set with bright conditions (lights on) at the later stage of pedestrian data collection through video recording, as described in Figure 2.

Figure 2.

Video recording of pedestrian detection and tracking on 1st and 2nd floor of the DURP building.

3.3 Agen based model and geographical information systems indoor integration process

The agent-based model (ABM) is considered a computational technique that aims to reinforce the analysis of the artificial environment utilized by interfacing the agents in nontrivial ways. The response from every agent is demonstrated separately. Agents who act with agents and later respond to their ongoing case as a set of attitude rules are subsequently derived from the principal theory actions and connections within a definitive framework [25].

When the agent initiates communication with the Geographic Information System (GIS) by sending a “seek to migration” message to the GIS, the evaluation of topological connection and geographic coordinates provides the agent the attitude of being authorized. Subsequently, the GIS responds by updates (renewing the GIS database and related graphical demonstration) or returns a message to the agent mentioning why the migration failed to perform, such as the area is as of now involved, or no permission could be given for the development [26].

4. Discussions

4.1 Observed pedestrian traffic on the 1st floor of the DURP building under normal conditions

The investigation of pedestrian traffic was done through CCTV recording and later processed with CV to conduct the tracking analysis. It is revealed that the pedestrian traffic on the 1st Floor of the DURP lecture building was concentrated in the corridor connecting the West and East Gate with a very high-density category (1.5). On the contrary, the lowest level of pedestrian density, with values ranging from 0.1 to 0.25, was revealed on the rotation path of the DURP building from the north to south, then turned west towards the Faculty of Engineering Administration building as it is described on Figure 3. This route was chosen as the most preferred path by the existing pedestrians because the intersections could connect to many other possible directions.

Figure 3.

Observed pedestrian traffic on the 1st Floor of the DURP Building.

Another dense pedestrian traffic was recorded in the Plaza area and North-South corridor of the DURP building, representing the traffic from U.B.’s academic community. This was caused by the path connecting the main rooms on the 1st Floor of the DURP building. Considering if the offline teaching and learning process takes place, then the daily movements could reach 600 based on the number of active students from four grades (2017 to 2020 grade and the average number of students per class is 30 people). Therefore, this situation has the potential for COVID-19 transmission through direct contact, thus worsening the pandemic situation over the university.

4.2 Re-designing pedestrian traffic on the 1st floor of the DURP building for minimizing crowd spots

In response to the observed pedestrian traffic and density in the DURP lecture building, scenarios are proposed to reduce potential physical contact. Opening the south gate with restrictions on the number of the academic community allowed up to 360 from 600 people. Therefore, this proposed scenario could help minimize the COVID-19 virus transmission through potential physical contact.

In addition to the proposed scenario, offline lecture classes are designed for students from grade 1 (batch 2020) and grade 2 (batch 2019), whereas grades 3 and 4 (batch 2018 and 2017) are proposed for online learning procedures. Finally, the simulation of integration of ABM and GIS Indoor scale to re-design pedestrian traffic on the 1st floor could significantly decrease pedestrian traffic and density, as explained in Figure 4.

Figure 4.

Re-designing the access for minimizing pedestrian traffic on the 1st floor of the DURP building.

4.3 Observed pedestrian traffic on the 2nd floor of the DURP building under normal conditions

Based on the observed pedestrian traffic on the 2nd floor of the DURP building, the stairway and plaza in front of the stairs and the route to the DURP library are to be considered to be traversed, as explained in Figure 5.

Figure 5.

Observed pedestrian traffic on the 2nd floor of the DURP building.

It is revealed that the plaza in front of the stairs, the stairway, and the DURP library are the most preferred route by pedestrians because there are no classrooms available on the 2nd floor, only consists of lecturer’s room, library, and the academic’s meeting rooms. Therefore, a low magnitude of pedestrian traffic was observed through the North-South corridor area. Furthermore, a high magnitude of pedestrian traffic is expected on the 3rd floor since it primarily consists of classrooms and a computer laboratory.

4.4 Re-designing pedestrian traffic on the 2nd floor of the durp building for minimizing crowd spots

According to the observed pedestrian traffic on the 2nd floor, scenarios are proposed to re-design the traffic. The restriction of the number of the academic community and the opening of the east gate could produce a significant decrease of pedestrian traffic, especially in the plaza on the north and the staircase corridor, as explained in Figure 6.

Figure 6.

Re-designing the access to minimize pedestrian traffic on the 2nd floor of the DURP building.

The proposed scenario could help reduce the pedestrian traffic in the staircase corridor at the north, since there is no classroom on the 2nd floor of the DURP building. Thus, dividing the entrance to the 2nd floor, which formerly could only be accessed through the staircase, could help reduce the pedestrian traffic and later could help on minimizing the potential transmission of COVID-19.

5. Conclusion

The relatively new scientific field of Humanitarian Engineering (HE) could potentially fill the gaps between STEM and SHAPE clusters. HE aims to investigate humanitarian trends and issues mainly studied in the social science field from the SHAPE cluster and later proposed the solution through engineering perspective from natural science and formal science from the STEM cluster. HE approaches are considered in this paper, referring to the latest social issues concerning the COVID-19 pandemic. Human activities such as school shopping and work are expected to return to normal once the global vaccination program has been completed. On the contrary, there were no optimal results reported from the studies related to the impact of global vaccination. Therefore the ‘new normal’ approach is proposed while maintaining the health protocols, in which avoiding crowd spots is part of the protocols.

It is revealed that the HE approach by studying pedestrian traffic in the DURP lecture building through CV and ABM-GIS Indoor simulation could helps on minimizing the crowd spots. The north and west entrance paths on the 1st Floor were observed with a high magnitude of pedestrian traffic. Therefore, the east side door opening could be an alternative for new accessibility for pedestrians. A similar approach applies to the 2nd Floor by opening the east gate could help minimize crowd spots.

The restrictions on the number of academic communities entering the DURP building are considering the need for empirical and site visits to case studies for grades 1 and 2. In addition, grades 3 and 4 have fewer classes to attend, and the learning patterns of senior students are emphasized critical thinking through the exploration of literature studies outside the classes.

Acknowledgments

This study was conducted by developing Humanitarian Engineering Curriculum under ENHANCE Project, funded by Erasmus+ with partners from Warwick University (U.K.), University of West Attica (Greece), Universitas Brawijaya (Indonesia), Institut Teknologi Bandung (Indonesia), Universitas Gadjah Mada (Indonesia), Bangladesh University of Engineering and Technology (Bangladesh), University of Dhaka (Bangladesh), Ho Chi Minh City University of Transport (Vietnam), and Ho Chi Minh City University of Technology (Vietnam). Furthermore, this research combined with Universitas Brawijaya’s research under the Faculty of Engineering, Universitas Brawijaya’s non-tax revenue (PNPB) research scheme. Therefore, at the end of this writing, the researchers would thank Erasmus+ and the Faculty of Engineering—Universitas Brawijaya.