Virtual reality [VR] is generally understood as the simulated, therefore not real, environment that can resemble the physical world. In most cases virtual reality refers to the human versus environment interaction and the creation of sensory experiences. Technical limitations, the computing power, as well as the basic understanding of the concept of reality, are the limiting factors that still hold back the progress. Virtual reality is also used to describe a wide variety of applications commonly associated with the 3D environments.
Human beings depend on 3D visualization since it allows us to see patterns and relationships that would otherwise remain hidden or at least wrongly interpreted. New VR technologies provide tremendous breakthroughs in visualization of any scenarios from any perspective. There are trends to join two completely different environments such as real and virtual into one merged reality.
VR is mainly centered to the human perception, with the final goal being the complete deception of human senses in a way that the users should not be able to detect that they are taking part in a virtual reality system. This is the so called VR Touring test. [1, 2]
In our case this is not a mandatory condition, VR contribution to the modelling of the materials is entirely different. VR models should provide the accurate ground for deeper understanding and finally for the reliable prediction of material properties. VR technology must be used to support the modelling of extremely complex environments (such as complex polycrystalline materials) to the level never possible before.
Neural networks are systems that actually create the virtual reality. Their performance depends on the dataset called the training set used to train the neural network and define its behaviour. Once trained, the bond between the real and virtual established through the training set remains, while everything outside this set, that is almost the complete model of reality, is virtual. Neural networks produce models that may be very close to the reality (which is the ultimate goal) but nevertheless they will always remain virtual.
Neural networks are used to perform the modelling of the real data. Most often the measured data points are, due to the practical limitations, scarce but we need the information on the observed system in the complete data sub-space. Models are then often used to perform various optimization or system control tasks. No matter what the purpose of modelling is, it always produces virtual reality. Modelling actually simulates-recreates the reality.
The organization of the chapter is following:
The sub-chapter “VR and modelling” starts with an explanation of why the VR is so important for the modelling of the material microstructures (or as the matter of fact at any other modelling).
Follows a brief description on how the data on metallic material microstructure is obtained providing the entry point and the connection to VR.
The fourth sub-chapter covers the modelling steps that produce the building blocks of virtual material. The sub-chapter concludes at the point where VRM (Virtual Reality Material) is ready to be fine-tuned according to the data gathered on real material to produce the VRM as close as possible to represent the reality. The presented approach is the non-mesh based representation of the material. The core idea is to generate randomly shaped grains, where each grain represents a stand-alone object existing as separate object bound to its’ surrounding, just like it is the case at real polycrystalline materials.
In fifth sub-chapter the metrics to assess the real and virtual building blocks of material is presented. The established metrics gives us the means to compare the shapes of material’s building blocks. In order to make a realistic model of the observed material, the shape of the grains represented by the neural network must come as close as possible to the observed material. The virtual grains are first generated. Separately the microstructural properties of the observed material are gathered and the process of grain-shape optimization is used to modify the shape of virtual grains to come as close as possible to the observed real sample.
The sixth sub-chapter explains how the virtual grain boundary can be changed in order to come closer to reality.
The seventh sub-chapter provides the evidence on how the VR approach provides additional information not accessible before.
At the end conclusions summarize the covered topic.
2. VR and modelling
There are many reasons that speak in favour of using VR and modelling together. VR and modelling are separate fields that live their own life, where VR tends to be popular and modelling remains narrow in serving only narrow purposes. VR itself must have the underlying model that defines the behaviour, but the model as such has no other purpose as to provide the ground for the VR to live. On the other hand modelling of materials did not show any tendency to go into the domain of VR, i.e. into the 3D visualisation. The existence of such modelling is justified by providing the information useful for the engineers needing the data on material’s lifetime expectancy, durability, mechanical, electrical, thermal, and other properties. The use of VR brings the totally new perspective to the modelling of materials since it provides additional information and therefore provides the insights that were not possible before. Material properties like lifetime expectancy cannot be predicted without knowing the 3D structure. Cracking of material is another phenomenon that can only be properly studied and explained in 3D. The diffusion process is also the phenomenon that can be studied on completely different level using 3D VR environment.
The process of creation and exploitation of VR model (Fig. 1) takes several steps:
In geometric sense the only reliable information on the material can be gathered from the 2D samples. The variety of shapes, composition, and different phenomena detected in the real material is tremendous. Actually it can never again be repeated in the same composition as it is on the observed sample. Despite the fact that natural laws always govern the observed material, the tremendous number of material’s constituent elements that took part in the process of material fabrication, result in completely chaotic structures. Therefore no 2D sample can provide the representative information on the observed material to the extent that would suffice for the prediction of material’s behaviour and properties.
On the basis of the description of the real world the virtual material is created. It is the virtual reality in every sense. Although the visualization of the virtual material is not the goal of the virtualization it adds the value to the system since never before the exploration of 3D material structures was possible with such ease and clarity. The virtual material is built on information gathered from the probed sample. Despite the fact that the information was gathered from the 2D samples, the virtual material exists in 3D.
VR model is being used to predict the important material’s properties. Along the way the discrepancies between the VR and R are detected, new facts are being revealed and the VR model is being corrected in order to provide better results.
3. Gathering the information on material
The characterization of microstructure plays a central role in material science and engineering. Significant effort was invested in the development of new characterization and analysis techniques. There are several approaches to the characterization of materials such as:
Chemical composition characterization mainly involves the determination of the composition and the distribution of the chemical components through the material.
Structural morphology .
Stereology (optic microscopy) provides information on grain size, grain boundary, etc., although it relies only on a 2D observation. A three-dimensional description of the microstructural features, such as grain-boundary geometry, crystallographic grain orientations, compositional variations, and experimentally derived microstructural models, are used to predict the material’s temporal behavior . For the polycrystalline metallic materials, the basic constituent element is grain. The generation of a virtual microstructure therefore depends on obtaining a reasonably reliable spatial arrangement of grains, along with their crystallographic orientation . In order to build more realistic VR microstructures the information on morphology of the grains is needed (the data on crystallographic grain orientation and grain-boundary properties). The source of such information is the electron backscatter diffraction (EBSD) technique, which has proven useful for the characterization of the crystallographic aspects of the microstructure. EBSD is used to perform quantitative microstructure analysis in the scanning electron microscope (SEM) on a millimeter to nanometer scale. EBSD has significantly improved the process of characterization of materials by linking the microstructure and the crystallographic texture. The potential of EBSD to obtain mapping orientation and crystal type for various materials in SEM makes it a powerful tool for a reliable information gathering used to build the VR model of the material . To further characterize the microstructure, Jinghui et al. proposed a new procedure using EBSD pattern imaging. The image quality enhancement procedure combined with EBSD analysis was considered to quantify the complex microstructures. Moreover, this application is important to relate the microstructural and mechanical properties of the observed material .
SEM equipped by EBSD was extended to a three-dimensional analytical technique. A focused ion beam (FIB) is used to remove layers of material (slices) from the sample. After the removal of each slice EBSD is used to collect the data from the newly uncovered surface. High-resolution two-dimensional cross-sectional images are then joined and interpolated into 3D space to visualize the structure of the material . When surface topography is of interest, SEM has been demonstrated to be successful for both characterization and geometry measurements. The computer imaging tools are used for the stereoscopic matching (2D to 3D), for calculation of the surface elevation and for the data interpolation. The spectroscopic matching algorithm produces quite accurate three-dimensional image of the material .
The objective of metallography/stereology is to describe the geometrical characteristics of the 2D microstructural shapes such as the amount, number, size of grains, etc. A material’s microstructure is chaotic in nature. The variety of grain shapes is tremendous (size, morphology, orientation, etc.).
The material’s sample surface is mainly represented in a 2D image (Fig. 2).
Fig. 3 illustrates the detection of grain (domain) borders, and separation of closed contour objects that define separate objects – grains.
It is important to note that the term grain is used throughout the chapter. It is not always clear whether the shape is actually describing the grain or it is describing the domain of some prevailing property.
Separate grains are the core elements where real and virtual materials meet. According to the information gathered from the separate grain shapes, the VR model is created and fine-tuned.
4. The neural network model of material
The goal is to create the virtual environment composed of a vast number of basic building blocks – virtual grains. We want to recreate the microstructure of the polycrystalline materials with a special emphasis on metallic materials. The literature provides the basic understanding of the so far used modelling principles and techniques leading to more or less practically useful models. Many researchers are trying to build models which will finally provide the reasonably reliable reconstruction of real materials and thus enable to provide insight into the micro to macro properties of the modelled materials which has for quite some time proven to be the unreachable goal.
The chapter describes a somewhat different approach to the same problem. The microstructure of the material is the consequence of a very complex evolutionary process, where the microstructure represents the unique state of the material. The state of the material can be registered in 2D with a reasonable accuracy using the relatively accessible equipment. The microstructure of the material can be understood as the geometric manifestation of the complex chaotic process that actually produced it.
Currently we do not know the process that would satisfactorily describe the 3D properties of polycrystalline materials. Since in reality we cannot easily observe the material in 3D, we use simplified methods for microstructure assessment (i.e. line intercept method …). Current methods used to describe the real materials microstructure do not suffice to build the adequate virtual model usable for material performance modelling.
The software installed on the optical or scanning electron microscopes provides the grain size distribution by one of the standardized line intercept method. We have shown that no reasonable correlation exists between the measured linear grain size distribution and 2D cross-section size distribution. Far more complicated is the case between the 2D cross-section grain size distribution and the 3D grain volume distribution. From the linear assessment of grain size distribution we know far too little regarding the actual grain size distribution of the observed material. This is one of the elementary reasons why the modelling of micro to macro properties did not bring good results.
The main building block of the material microstructure is a grain which is also understood to be the main building block of its’ virtual counterpart. In the virtual environment such grain represents an object described by the main properties such as: border area defining the shape, orientation, etc. From the computational point of view the grain object has its’ own processes such as: random generation of the grain, volume calculation, cross-section area, determination whether the given point belongs to the grain surroundings or to the grain itself, grain re-shaping procedures etc.
The neural network was introduced for the generation of virtual grain. It makes the grain manipulation easier and the representation of grain very compact. The virtual grain generation is only the first step in the process of building the fully 3D polycrystalline material model.
4.1. The random grain geometry
The randomly generated grain is the main building block of the virtual material. It must fulfil the following conditions:
The shape of each grain must be the consequence of the random process.
Each grain is uniquely shaped object.
It must resemble the shape of the grains met in the modelled material – the grain shape must be random but with the control of important properties (size distribution, grain roughness, etc.).
The representation must be as compact as possible.
The shape must be easily transformed.
The position, orientation and movement in the coordinate system must be simply controllable.
In our work we have used two steps to generate the virtual grains, the first is geometric, the second uses the neural networks. The geometric generation of grains is just an intermittent phase used to provide means to present grain shapes to the neural networks. Once neural network is trained the grain shape is completely represented by the neural network parameters and the formal geometric representation is no longer needed.
The process of the random grain generation starts within the polar coordinate system (Fig. 4) and is conducted through four steps.
For each consecutive corner point the pairs
The 2nd step is conducted in Cartesian coordinate system. For the neighbouring corner points the grain border carrier lines are calculated and denoted by two parameters – the slope and the y-intercept (Fig. 5).
In step 2 the general linear equations for the grain border lines were defined. In order to conclude the grain boundary generation, the valid line sections (those parts that really represent the boundary) must be found (Fig. 6).
The analytical part of the grain generation process ends with the number of representative points in the R/Angle coordinate system (Fig. 7 left). The function depicted in Fig.7 right forms the training set used for the neural networks to learn the grain boundary.
4.2. Neural networks
Neural networks are model-less approximators, meaning they are capable of modelling regardless of any relational knowledge of the nature of the modelled system.
In our work we are using the feedforward neural networks with supervised training scheme.
The basic building element of any neural network is an artificial neural network cell (Fig. 8 top).
Artificial neural network consists of a number of inputs (synapses) that are connected to the summing junction. The values of inputs are multiplied by adequate weights w (synaptic weights) and summed with other inputs. The training process changes the values of connection weights. The value of summed and weighted inputs is the argument of an activation function which produces the final output of an artificial neural cell. In most cases, the activation function is of sigmoidal type .
Artificial neural network cells are combined in the neural network architecture which is by default composed at least of two layers that provide communication with outer world (Fig. 8 bottom). Those layers are referred to as the input and output layer respectively. Between the two, there are hidden layers which transform the signal from the input layer to the output layer. The hidden layers are called hidden for they are not directly visible to the input or output of the neural network system. These hidden layers contribute significantly to the adaptive formation of the non-linear neural network input-output transfer function and thus to the characteristics of the system.
The process of adaptation of a neural network weights is called training or learning. During supervised training, the input – output pairs are presented to the neural network i.e. for each presented input value the desired output value (target value) is also known for they are both part of the training set. The training algorithm iteratively changes the weights of the neural network in order to get closer to the presented output values.
The data points are consecutively presented to the neural network. For each data point, the neural network produces an output value which normally differs from the target value. The difference between the two is the approximation error in the particular data point. The error is then propagated back through the neural network towards the input, and the correction of the connection weights is made to lower the output error. There are numerous methods for correction of the connection weight. The most frequently used algorithm is called the error backpropagation algorithm.
The training process continues from the first data point included in the training set to the very last, while the queue order is not important.
When the training achieves the desired accuracy, it is stopped. From here on, the model can reproduce the given data points with a prescribed precision for all data points.
In our case the training set was composed of 50 points representing the grain boundary (Fig 9). The number of 50 points was experimentally chosen. With the neural network it is only possible to model unique functions. The grain boundary represented in Cartesian coordinate system (Fig. 7 left) does not fulfil the condition of uniqueness, while the same boundary represented in the polar coordinate system (Fig. 7 Right) is unique. Once trained the complete grain boundary is represented by the neural network weights (Fig. 9). For the selected neural network architecture (1-10-10-1) 141 weights are needed (Fig. 9 left). For the grain boundary definition no further calculation is needed, the grain is fully described.
In the computational sense the grain is an object combining data and methods. Grain data is included in the neural network weights, methods (processes) are:
Random generation of the grain,
Cross-section area calculation,
Determination whether the given point belongs to the grain surroundings or to the grain itself,
Grain re-shaping procedures,
From Fig. 9 one can observe that the grain shape has no sharp edges as it is the case at geometric shape (Fig. 7). The roughness of the grain boundary is within the neural network representation quite easily adjustable.
By implementation of the described steps we can generate a vast number of random virtual grains. The shape of each grain is unique and from the practical aspect it is not likely ever to be repeated. This is exactly the case we encounter at real materials. As we can observe from Fig. 10 right the neural network representation is smoother and closer to the reality as its’ geometric counterpart.
5. The grain similarity metrics
Many existing methods attempt to describe the roughness of grains in 2D and 3D, but none of them is capable of uniquely describing this parameter. A common way to describe the angularity of different particles is to use a numerical parameter called the spike parameter. In this case the particle abrasivity is related to the size and sharpness of triangles. Maximum and minimum circumscribed circles can also be used, but they have certain limitations. Many other methods were developed to describe the particle boundary, such as: shape factor, aspect ratio, roundness, etc. . Since there are no solutions that would suit our needs, the new assessment method that takes into consideration many details of the grain boundary was developed.
5.1. Roughness histogram method
For each generated grain, the roughness histogram is obtained. In general, for the grains generated by the neural network 300 points are used to describe the grain boundary. The roughness histogram method measures the grain boundary’s departure from an ideally round object (circle).
The tangent on the circle takes the angle of
The angle φ representing the measure of the departure of the grain boundary from the round shape in the point Tn is obtained. The results of the calculation are given in the histogram in Fig. 12. The histogram consists of 18 (+1 for error detection) classes therefore it can be treated as a vector in 18 dimensional space. To calculate the distance
The coordinate system for each generated grain must be put into the grain’s centre of gravity (Fig. 13).
The grain-centring process takes the following steps: grain is sectioned in a series of triangles defined by the coordinate centre and the two adjacent grain-boundary points. In the case of the two-dimensional shape the intersection of all the straight lines that divide the shape into two parts of equal momentum about the line defines the centre of gravity. It is the average (arithmetic mean) of all the points composing the shape .
The grain coordinate system origin is moved to the gravity centre of the grain.
The roughness histogram method only gives comparable results when used on centred grains. In Fig. 14 there are two instances of the same grain. The first one is not centred, and the second one is placed in its gravity centre.
Many tests show that the method used is fast, stable, and can represent the roughness, regardless of the size, rotation, and mirror image of the grains in the coordinate system.
6. Joining the properties of real and virtual material
The basic virtual grain shape is provided by the neural network. The desired (achieved by the analysis of real material) grain-roughness histogram normally differs from the currently observed virtual grains. The main goal of the procedures that perform the grain-shape optimization is to obtain the virtual grain-roughness that is as close as possible to the desired one. We know that the match between the targeting grain-roughness and the achieved one should not be too strict since the basic shape of the grains influence the grain-roughness histogram significantly and our goal is not to reproduce the grain shape as well.
The grain-shape optimization process must fulfill the following conditions:
It must not alter the starting grain shape significantly.
The main influence must affect the grain boundary.
It must be a controllable and convergent process.
Let us assume that the neural network produces the grain-boundary function denoted by . The random grain-boundary modification must fulfill the condition that the point at the angle φ=0 remains the same as at the angle φ=2π. The modification of the grain-boundary function must provide the rougher boundary, while at the same time it must be random in shape. Equation (2) describes the circumstances.
where represents the modified boundary function, represents the original grain-boundary function provided by the neural network, and represents the random function that actually modifies the grain boundary. An example of the modified virtual grain is shown in Fig. 15.
The complete random process of the grain-boundary modification is controlled by only three parameters.
7. The grain size distribution — The first overlooked detail in classic modeling
Basically three ASTM (American Society for Testing and Materials) methods are used for the evaluation of the grain-size distribution :
The existing methods provide the size distribution of grains only in 2D. However, many attempts were made to describe the grain-size distribution in 3D from the data obtained in 2D. Due to the computational complexity researchers replaced the realistic grains with spheres, which eventually lead to a grave systematic error. More complex grain model proposed by Zhao  uses polyhedral grains for the grain-size assessment. The results have shown that this model is more efficient in calculating the 3D grain-size distribution from 1D and 2D distributions.
However, comparing the 2D and 3D grain size distribution in a cellular material shows similar results. The areal distributions of planar sections taken for various polyhedral shapes demonstrated a good agreement between the expected and the calculated areal curves for an assumed complex polyhedral symmetry . When a sectioning plane intersects the features in the microstructure, the image obtained reveals features that are reduced in dimension by one , as follows:
Volumes (three-dimensional) are described by areas,
Surfaces (two-dimensional) by lines,
Curves (one-dimensional) by points.
The shape of the grains can be relatively simply quantified by measurements made on grain structure. The ratio of the grain’s longest chord and its perimeter to the equivalent diameter can be analysed using image analysers . However, among others, particle shape angularity and roughness are the most difficult parameters to define. The development of new methods to characterize particle properties has been motivated by the need to improve particle modelling. A new technique was proposed by Sukumaran et al.  to quantify particle shape and angularity using an image analyser. The true shape of the particle is approximated by an equivalent polygon and a new shape factor is defined as the deviation of the global particle outline from a circle. Most techniques developed so far tend to reduce a shape into a simpler shape representation. The common criteria cited by researchers while evaluating the shape representation are: scope, uniqueness, stability, sensitivity, efficiency, etc. . The problem associated with particle surface characterization is that they provide statistical functions and parameters that are not unique for a particular particle surface . Very popular methods for the roughness characterization of particles in general are fractals. However, the methods used to calculate the fractal dimension can be effective when applied under some limiting conditions.
Using the described modelling the line intercept method [25, 26] was tested for its’ validity. For this purpose we have generated a virtual material with an area of 1 mm2. The goal of the test was to analyse the differences between the grain size distribution obtained by the line intercept method and the exactly known grain size distribution of the modelled virtual material. At the line intercept method the line crosses the underlying grains at arbitrary position (from the grains perspective). Since the grain shapes are very non-uniform the representative crossing of the grain that would correlate with its’ actual size does not exist.
If we take into examination one single virtual grain and perform its’ slicing then one single grain can produce the size distribution. Furthermore, depending on the angle at which the slicing is performed, the distributions for one grain are very different (Fig. 16). From the experiment it is clear that line intercept method, where the line crosses the grain at arbitrary position, cannot represent the adequate information regarding the involved grain size.
It is generally accepted that the large number of involved grains eventually statistically correct the errors made at each grain by the line crossing. 1 mm2 of the virtual material was generated and the line intercept method covered 1mm of it. The process was repeated several times and the results are presented in Fig. 17 where only two experiments are shown.
From Fig. 17 it is evident that there are substantial differences between the line intercept distribution and the real grain size distribution. It is also evident that the random grain generation process produces the virtual material of almost normally distributed grain sizes. It has to be noted that the random generation process was not influenced by any means to produce such a distribution. The real distribution is very stable and was almost the same for all experiments. On the other hand the distributions obtained by the line intercept method vary significantly. The lower parts of the size distributions falsely indicate that there is a large amount of small particles in the sample. This anomaly is due to the fact that a large amount of grains is always crossed at the grain sections having low crossing length.
VR technologies that provide the enhancement of modelling will provide significantly deeper understanding of real materials. This is also true for the nano-structures where the geometry plays a central role.
The entry point to the VR is the information gathered through the various techniques. Generally main geometric properties are gathered through the various microscopies. Preparation of samples for the microscopy is critical since errors at sample preparation stage are directly integrated in the model.
From the microscopy images the separate grains (domains) are extracted. This is a second critical process since it adds additional errors to the model supporting the VR. Extracting the closed contours from the digital image that contain grains (domains) is not a trivial task. For some microstructures this task is next to impossible.
Once the grain shapes are extracted, their boundary roughness histogram is calculated and averaged over the grains. The boundary roughness histogram is the one that model is taking as target, so the modelled material will exhibit almost the same histogram. From the extracted grains their size distribution is determined. Boundary roughness histogram and grain size distribution are used to create the model.
The main building block of the virtual microstructure is the random grain. Virtual grain is first generated and described by geometric means. The second step is training the neural network to hold the grain boundary. Once trained the geometric form can be discarded, grains are completely represented by separate sets of neural network weights, no complex computations are needed to provide the grain volume, grain cross-section area, grain reshaping in order to get it closer to the reality, etc. The grain is an object. It joins its own data as well as methods such as:
Random generation of the grain,
Determination whether the given point belongs to the grain surroundings or to the grain itself,
Grain re-shaping procedures,
By using the generated virtual material of 1mm2 area it was shown that commonly used line intercept method cannot represent the real size distribution of the material although it is generally used under ASTM (American Society for Testing and Materials) standardization.
The finding is the first and direct consequence of using VR to support modelling.
The research was financially supported by the Slovenian Research Agency program group P2-0132 (Institute of Metals and Technology, Ljubljana, Slovenia)