Information Technology Drivers in Smart Farming Management Systems

Abstract

The chapter describes the possibilities of collecting digital data on crop and livestock production and their use in “smart farming” systems. Earth drone and spectral mobile mapping technologies can provide plant production-related measures with high temporal and spatial resolution. Remote sensing helps better understand farming patterns and crop management. Improving understanding of the link between remotely sensed data and risk assessment and management in “smart farming” is very important. Controlled-environment agriculture takes advantage of light recipes, related to spectral light-emitting diode (LEDs) and sensors. In livestock farming, analyzing a database of digital data on the environment and livestock individuals can help farmers make decisions better. The heterogeneous digital data from plant and livestock production are collected into a Data Lake. Then the data are processed to transform the data into the proper format for data analytics. Data Warehouse should be integrated into an ERP system that is dedicated to the agricultural environment.

1. Introduction

The transition from experience-driven to data-driven decisions in agriculture production is unthinkable without the use of digital tools and solutions. Given the traditional nature of agriculture production and its custom-based practices, this is not a quick process. During the last decades, farming has been forced to implement measures to increase efficiency and productivity at the expense of resilience in the face of climate change and environmental variability. Intensification in agriculture has been causing a serious impact on environmental sustainability. The industry is facing a reduction in the workforce, and consumer demand is growing for more transparent, sustainable, ecological, and high-quality products. Moreover, the new common agricultural policy of the EU aligned with the European Green Deal is focused on environmental protection in rural areas. The use of precision methods – i.e. information and communication technologies (ICT )tools, processes, and methods – in the production of agricultural products is becoming increasingly important in production practice. In the two major sectors of agriculture, crop production and large-scale livestock farming, precision technologies are enabling the creation of “smart farming” systems. In data-driven farming, the expertise of the farmer is becoming more valuable. The results of data analysis that exploits large-scale databases can be incorporated into decision support systems. This data analytics will enable actors in the agricultural sector to rely on accurate, traceable, and credible production results to make decisions that will help them manage cost-effectively and optimize the environmental impact of production.

Digital tools and analytical methods are ready for use in agriculture. It will take time for them to become widespread in farming practices and part of farmers’ animal and crop production procedures. The application of data analytics will validate the added value of precision practices and their added value for farmers in real-life farming environments through lots of good practices. This chapter reviews the most relevant precision methods in crop production, large-scale livestock production, and their usability in the decision support process. The latter leads to “smart farming” technology.

2. Precision plant production

The real benefit of proximal and remote sensing is the capability to characterize spatial or field variability that cannot be parameterized more effectively by any other way. This function has great potential for all land-use practices to increase information availability in everyday farming using proximal and remote sensing technologies. Generally, remote sensing performs nondestructive chemical measurements without intrusion into the material, while providing the possibility of a broad spatial overview and high temporal flexibility.

When spatial thematic information is requested for large-scale areas at regular times, satellite remote sensing is often applied in agriculture. Nowadays, many traditional remote sensing tools are available for both large- and small-scale farming entities as well. Spectral imaging and non-imaging sensors are powerful bio- and geochemical data acquisition tools that can play a crucial role in the early detection of crop management risk factors, such as soil nutrition supply, pests, and diseases.

From a practical point of view, those research studies in the paper are considered that are using field spectrometers and/or spectral cameras and attempt to understand the agricultural values, biophysical and biochemical properties, or reactions of cultivated plants. Only outdoor or field-related applications are included in this discussion of the paper.

2.1 Remote sensing data acquisition

The fineness of the spatially distributed data depends on the sensor and platform. There is a technical limitation to the spectral and spatial resolutions of the satellite platforms. This constraint causes that high spectral resolution and high spatial resolution cannot be achieved at the same time from the same satellite altitude. It has technical aspects, one of them being a justifiable signal-to-noise ratio. The signal-to-noise ratio (SNR) compares preferred signal levels to unpreferred ones. It is complex to give an average SNR for a sensor or multispectral data because it depends on wavelengths, radiance levels, and other technical parameters. Generally, non-imaging spectrometers provide higher SNR values compared to imaging ones. Satellites with less than 1-m pixel size have less than 10 broad bands in the spectrum typically, while satellite images with more than 10 spectral bands have larger pixel sizes than 10 m on the ground typically. One way to increase the spatial and spectral resolution is to change the sensor and reduce the altitude of the data capturing. This demand initiated many different forms of terrestrial and near-ground imaging and non-imaging spectroscopy.

The spectral resolution describes the electromagnetic spectrum to sense material properties and characterizes the number and width of the spectral channels available for spectroscopic sampling. The spectral resolution could be also interpreted as the “chemical resolution,” since the spectral resolution resolves the apparent spectral material properties and links chemistry to spectroscopy. Accordingly, the higher spectral resolution provides more detailed chemical insights [1].

Temporal resolution is a factor in agricultural remote sensing that controls flexibility and data availability. The periodical returns of satellites are typically not demand-driven, and the airborne campaigns with the high-temporal resolution are very cost-intensive and complex.

Radiometric resolution is a technical term that characterizes the sensitivity of the detector or the wavelength-dependent energy resolving power of a sensor. It is quantified by bits, typically. Accuracy and stability are essential in radiometric calibrations to calculate radiance and/or reflectance that are the derivatives and representative outputs (information carriers) of the remotely sensed data and the primary inputs for further statistical analyses.

In color imaging, three broad bands (blue, green, and red) are used to reproduce real-life object properties in a virtual form the best. The RGB (red, green, and blue) bands are broad spectral channels.

When the number of spectral channels is increased (over 100) and the spectral range is extended (400–1000 nm or more), imaging spectroscopy or hyperspectral imaging is applied.

2.2 Characteristics of data sources

Spatial scales of field phenomena are not absolute and are customized to specific needs and applications. From a global (earth-observing) point of view, scales smaller than 104 km² could be referred to as local scales, which are higher by several magnitudes than the common agricultural management scales in Europe. For site-specific observations, further downscaling is needed. For crop management, the variability on the field and subfield scale are of interest, and the variability at distances of 50 m or less is mainly related to management practices [2].

Considering options for the remote sensing application in agriculture is one of the most time-critical. The entire crop sector and production are based on time-critical processes that contain sowing, plant protection, fertilizing, irrigating, and all management decisions.

In spatial downscaling when the measurement height drops down to 100, 10, and 1 m, the temporal, spatial, and spectral resolution can be significantly increased and new demands or application needs such as mobility (e.g. on the fly) and flexibility (e.g. vehicle-based) can be considered.

The temporal resolution affects not only the process accuracy but also the imaging process. Recent developments show that a novel kind of imaging technique (e.g. snapshot spectroscopy) enables high-rate spectral images to generate spectral videos that are an obvious advantage in online process monitoring and controlling of agricultural conditions both in field and indoor.

Ref [3] accentuated that characterizing vegetation, soil, or environmental parameters through spectrometers would offer new opportunities. Meantime, many new opportunities for application were found in science and research, primarily. Remote sensing research topics often focused on stress caused by pest or disease incidences, yield and biomass estimation, nutrition deficiencies, drought, frost, etc. Vegetation stress may cause anomalies in the cellular or leaf structure affecting the pigment system or the moisture content in canopy, which could be detected and mapped by optical sensors as can be seen in Table 1.

Passive remote sensing		Active remote sensing
Multi-/Hyperspectral	Thermal	Radar	Lidar
Plants: Leaf pigments Phenology Cell and tissue structure Water content Biochemical processes and products (lignin and cellulose) Diseases Soil: Clay minerals Humus content TNC CEC	Plants: Water stress ET stress Pathogens Harvesting Yield estimation Soil: Moisture Texture	Plants: Canopy height Canopy density Plant height Canopy structure Biomass Soil: Soil roughness Soil moisture DEM	Plants: 3D plant model Volumetric parameters Plant morphology Canopy structure Soil: High res. DEM Erosion Geomorphology

Table 1.

Passive and active remote sensing tools used to characterize plant and soil parameters.

2.3 High-resolution crop remote sensing

Remote sensing of biophysical parameters such as phytomass, leaf area index (LAI) and canopy structure have intensively been analyzed, [4, 5, 6, 7]. Behind the biophysical parameters, numerous papers have been devoted to biochemical components such as foliar constituents, chlorophyll a and b, carotenoids, lignin, cellulose, protein, water, and other elements [8, 9]. Many of the studies used high-resolution full-range (FR) spectra (400–2500 nm), because some foliar chemistry components give indications only over 2000 nm (e.g. lignin and cellulose) [10, 11]. Our study focuses on narrowband indications in the range of 400 to 1100 nm. Our study focuses on narrowband indications in the range of 400 to 1100 nm.

Multispectral, satellite remote sensing used broad (50–100 nm) spectral bands initially, which have been narrowed by scientific high-resolution sensors over the last decades [12]. The VNIR (400–1100 nm) spectral range will remain significant in future crop sensor developments as well, but it will be spectrally enhanced likely, to produce high-resolution crop or soil sensors. Band comparisons highlight the best benefits of the narrowband indices such as non-saturating behavior or high sensitivity in vegetation dynamics (e.g. phenology) [13]. The narrowbands can be classified as very narrowbands (1 nm to 15 nm), narrow bands (16 nm to 30 nm), intermediate bands (31 nm to 45 nm), and broadbands (greater than 45 nm) [13]. For future VNIR crop sensor developments, the following spectral narrow bands could be of interest (Table 2).

Wavelength (nm)	Parameter	Indications	References
375	Biochemical	Leaf water content	[13, 14]
466	Biochemical	Leaf chlorophyll
515	Biochemical	Leaf nitrogen	[14]
520	Biochemical	Pigment content	[15, 16]
525	Biochemical	Leaf nitrogen	[17, 18]
575	Biochemical	Leaf nitrogen	[19, 20]
675	Biochemical	Leaf chlorophyll	[14, 21]
700	biochemical	Nitrogen stress	[14, 22]
720	biochemical	Nitrogen stress	[13, 23, 24]
740	biochemical	Leaf nitrogen	[14, 25]
490	biophysical	Crop yield	[14]
550	biophysical	Biomass	[23, 26]
682	biophysical	Crop yield	[14]
845	biophysical	Biomass	[27]
915	biophysical	Crop yield	[14, 24]
975	biophysical	Leaf moisture	[28]
1100	biophysical	Biomass	[29, 30]

Table 2.

Narrowband sensor wavelengths for measuring crop parameters.

Narrowband studies showed that classification accuracies have increased. Generally, the hyperspectral narrowbands explain about 10–30% greater variability in quantitative biophysical models in comparison with broadband and are not sensible to saturation problems in biophysical estimations [31]. These benefits are to be considered in future high-resolution imaging or non-imaging crop sensors. There are known important parts of the VNIR spectrum: the so-called red-edge region, which is likely becoming increasingly important for novel optical field sensors as well (Table 2).

An auspicious tool to detect vegetation conditions is to study the sharp rise of the reflectance curve between 670 and 780 nm. This segment is called the red-edge region. Both the position and the slope of the red-edge point (REP) change due to physiological conditions and can result in a blue- or redshift of the inflection point. The red-edge index is defined as the position of the inflection point of the red-NIR slope of a vegetation reflectance curve. The reliable detection of this index requires high-resolution spectral measurements [27]. The well-known methods are to define red-edge position (REP) [32]. The reflectance curve’s numeric derivation and interpolation techniques are also widely used. The REP is correlated strongly with foliar chlorophyll content which is a sensitive indicator for various environmental factors. A comprehensive spectral analysis has been conducted on fruits and other agricultural products in scientific studies [33]. Recent developments in REP offer new perspectives and approaches for spectral mobile mapping services.

2.4 New demands and tendencies

The demand for out-of-the-lab devices initiated the early field spectroscopy and sensor with non-imaging measurements, which originated from laboratory spectroscopy and required respective developments in optics and portable platform techniques. From the beginning, portable or handheld field spectroradiometers were very popular in geology, soil, and vegetation spectroscopy as they provide flexible and rapid field data acquisition [34]. Thus, the spectroscopy in the visible (VIS) and near-infrared (NIR) has been widely used either in the laboratory [35] or for in situ monitoring [36].

There is an apparent gap between integrative point measurements and airborne or even space-borne image data. Field imaging line scanners are less widespread in ground-truthing than portable point spectroradiometers. Non-scanning or snapshot hyperspectral imaging is one possible solution to overcome this limitation of in-field usability [37]. Snapshot hyperspectral imaging enables rapid data acquisition as the entire image with all spectra is captured, at once, within a few milliseconds by a handheld or portable mode [38].

Optical field data acquisition has been reshaped and extended by new platforms in the last few years. This kind of platform liberalization changes our ground-truthing attitudes and fieldwork traditions. Traditionally, field spectroscopy was used to support airborne and space-borne campaigns (Figure 1).

Figure 1.

A non-imaging (A) and an imaging spectrometer (B) in field use. Source: A. Jung.

The proximal and remote sensing spectral detectors are either imaging or non-imaging sensors. Until recently, light-weighted spectral scanners were not used widely because of technical limitations. One of the first successful fix-wing miniature spectral scanning measurements was achieved by [39]. The light-weighed scanners (< 1–2 kg) mainly work in the spectral range from 400 to 1100 nm. These typically utilize push-broom spectral imaging. Hyperspectral cameras with the scanning principle cannot capture random moving objects. Mobile imaging field spectroscopy requires sensors that are flexible and easy to operate. Non-scanning hyperspectral imaging has been recently introduced for many outdoor applications. Non-scanning spectral imaging is called “snapshot imaging spectroscopy” [37], and it has a different principle from the push- and whiskbroom sensors (Figure 2).

Figure 2.

Working principles of hyperspectral imaging, colors represent different wavelengths. Source: Courtesy of Cubert GmbH, Germany.

A snapshot light-splitting architecture integrated on a sensing sensor chip with appropriate spatial resolution captures the full-frame image with a high spectral (> 100 bands) and radiometric resolution (> 14 bit). The image capturing process benefits from a powerful light collection capacity [37]. For a hyperspectral snapshot camera, in a sunlight situation, the integration time of taking one hyperspectral data cube is about 1 ms. Such a camera can capture more than 10 spectral image data cubes per second, which facilitates hyperspectral video recording. The commercially available snapshot imaging spectrometers record hyperspectral full-frame images with more than 20–100 bands in a spectral range of 400 and 1000 nm.

The snapshot advantage prefers time-critical applications either in the laboratory or in the field. This fact is significant for vegetation studies and crop management, especially because of physiological and phenotypical changes [31]. Knowing more about temporally resolved spectral crop information is of high importance for agriculture among others because of timely and targeted nutrition supply, and preventive and precision pest control. Beyond the temporal aspect, there is a general and global demand for high-resolution data in agricultural process control. The technical paradigm change in imaging field spectroscopy will enhance the effectiveness and availability of commercial sensors in smart farming.

The real-time image capturing capability of the proximal snapshot imaging spectroscopy is essential for capturing moving objects (i.e. leaves and canopy) or being on a moving platform (i.e. vehicle, unmanned aerial vehicle (UAV), robot, or human being) at high-resolution scales. Smart farming applications could offer individual detection and treatments of species or canopies that are of interest in the viewpoint of economic, environmental, and professional. Reference [40] used a snapshot hyperspectral imaging camera in a farming experiment to study its usability on a UAV platform to monitor crops. This study concluded that the combination of 3D imaging techniques and snapshot hyperspectral imaging enables the precise and accurate monitoring of dynamic crop growth through phenological changes. A multi-temporal crop surface analysis enables the precise monitoring of plant height and plant growth, while hyperspectral analysis derives physiological vegetation parameters like chlorophyll or nitrogen content and others. To monitor crop growth behavior, crop vitality, and crop stress snapshot hyperspectral imaging may be an ideal sensor [40].

3. Precision livestock farming

In all farm animal species, farmers have the same goal: to produce animal products profitably. The scope of inputs is similar: feed, water, livestock, medicine, infrastructure, and human resources. The big difference is that the animal species have different external and internal characteristics, the farms have different technological standards, and the management requirements are different. These factors determine the structure, quality, and use of the resources listed earlier. The digital technologies ought to fit into the operational framework of a given livestock farm and create a well-defined added value for farmers. This is not an easy task under the conditions in which livestock products are produced, the often difficult to predict market environment affecting profitability, and changing social and regulatory factors.

The driving force behind the demand for precision technologies in large-scale livestock farming is the possibility of early detection of diseases primarily [41]. In the case of breeding animals, this offers the opportunity to reduce the additional costs of culling due to the late detection of disease. In these groups of animals, the farmer wants to keep the farm animal in breeding and reproduction as long as possible. The objective of rearing animal products for human consumption (fattening and/or laying flocks) is to produce the animal product of the quality required by the market in the time available, using the optimum quantity of inputs (feed, drinking water, etc.) to reduce the negative environmental impact.

The early detection of diseases and their electronic alerting to the farmer through the analysis of digital data collection tools and the database they produce is just one of the possibilities offered by precision livestock technologies. Achieving this goal also provides the producer with a range of other useful information: the appearance of the disease is indicated by changes in animal behavior. While the digital devices collect data on individuals and send a signal when they change, analysis of the data in the database can reveal several important facts. These include the time spent eating, drinking, and resting, which is typical of all our farm animals. Species specificities can be observed, such as scratching in poultry, wallowing in pigs, or ruminating in cattle. In addition to studying individual behaviors, the observation of social behavior also provides the farmer with useful information about certain farm animals (e.g. fighting or playing patterns in pigs).

These observations have been made by farmers in the past and present without digital tools. The difference is that the use of precision technologies reduces the need for personal presence and allows continuous monitoring of individual and group animal behavior instead of periodic observations [42]. An additional advantage is that farmers can gain practical knowledge and experience not at the time of data collection, but also at a later point in time. The data can be saved and examined in other using methods that were not previously available.

The central issue of precision livestock farming (PLF, smart livestock farming, and smart animal agriculture) is how to increase food production sustainably. While, the farmers should care about animal welfare and reduce the environmental burden. This goal can be achieved by merging data that originated from data acquisition (sensors) and Internet of Things (IoTs). The data transformation along with predictive analytics can be applied by using artificial intelligence (AI) tools. Ref [43, 44] indicated that PLF uses principles and technology of process engineering to manage livestock production through smart sensors to monitor animal growth, milk and egg production, endemic diseases, animal behavior, and components of the microenvironment within the production unit (in Ref [44]). Precision livestock technologies are playing an increasingly important role in response to the factors that hamper the production of animal products. These include reducing the environmental impact of intensive livestock production systems, reducing the cost of inputs by optimizing their quantity (including human labor), and keeping up-to-date knowledge of animal health by studying individual and group animal behavior [42, 45]. A more detailed in-depth analysis of data from the farming environment will help livestock keepers make decisions and indirectly improve their ability to generate income. These technologies consist of digital data collection, the creation of databases from that data, the analysis of the data, and the presentation and visualization of the patterns found in the data. The big difference with traditional methods is that databases containing data on animals or environmental parameters collected by digital means cannot be successfully analyzed using traditional statistical methods. This requires the use of data science methods to identify internal patterns in the data sets that are not possible with traditional analysis methods.

It is important to note that precision technologies, i.e. the use of informatics in the collection and processing of data from livestock, cannot be generalized in practice. There is no universally applicable device or procedure that answers the questions of livestock farmers. The simplest, most reliable, and least costly IT solution must be tailored to the circumstances of the farm. Digital data collection tools can be categorized according to the type of data they collect, and the most appropriate analysis methods should be selected from those already available. This requires knowledge of the specificities of the farm. In practice, these precision solutions will only be widely used if they are validated in on-farm projects.

This section describes the framework for precision farming technologies.

3.1 Data acquisition

Recently, there are several publications on precision livestock farming [46, 47, 48, 49, 50, 51]. Several authors have concluded that although the IT solution that are used works well, its practical uptake remains to be seen. In [52], the major limitation of practical applications includes high installation and maintenance cost, difficulties in using the new technologies due to lack of knowledge or skill of farmers, lack of confidence in the manufacturing companies, etc. How can this be changed, and how could this be improved? (Figure 3).

Figure 3.

Farm with precision livestock farming technology [52].

The use of precision livestock farming technologies would contribute to consistent objective and regular welfare monitoring of livestock in real time, allowing farmers expeditiously to identify problems and implement preventative measures to avoid critical failures [53]. In large-scale livestock farming, the data that come into the earlier-mentioned database can be divided into two broad categories: direct (i.e. digital data coming in via IT tools) and indirect (digital data recorded by the farmer). Digital data must be collected from all possible sources to have an accurate database of the production activities of a given livestock holding. This database is necessary to achieve the objective. The so-called analog and historical data that are collected traditionally should also be incorporated into the database.

Another equally important aspect of data collection is the environment-oriented (i.e. data collected directly on the farming environment) and animal-oriented (data collected on livestock individuals) data sets. The use of both animal- and environment-oriented data supports easy and proper monitoring of health, welfare, production, and risks [54]. Farmers have been collecting these data with varying degrees of attention for years. After all, a careful farmer wants to know exactly how much it costs to produce the animal product and where there are points for improvement. However, we can only talk about precision livestock farming if the database also contains digital data on the individuals in the livestock.

3.1.1 Housing system

Table 3 lists the input data that can be measured and analyzed to provide the farmer with information on the farming environment. There are three main types of farming in large-scale livestock production: confined, semi-confined, and free-range. The more enclosed and controlled housing is (poultry and pigs), the easier it is to collect data on the microenvironment of the barn using sensors. These data were collected by farmers for decades, especially in confined housing, because it is the basis for automatic ventilation, cooling, heating, feeding, and water technologies. This approach represents an opportunity for precision livestock farming because farmers do not use this large amount of data for data analysis purposes in most cases. In semi-confined systems, animals are affected by the external environment, and human influence is minor (dairy cattle). In free-range conditions (beef cattle and pigs), the role of humans is even reduced, with external weather factors fully determining the environmental conditions around the animals. In this type of farming, it is also possible to collect environmental data, e.g. by placing weather stations in the pasture.

Sensors	Housing system
	Closed	Semi-closed	Free-range
Temperature	xxx	x	x
Humidity	xxx	x	x
Air speed	xxx	x	x
Ammonia	xxx	x	—
Carbon dioxide	xxx	x	—
Air pressure	xxx	x	x
Feed level	xxx	x	—
Drinking water flow	xxx	x	—

Table 3.

Data in the farming environment, Alexy M.’s own research.

The way how animals are kept also has a big influence on the individual data that can be collected about them. Table 4 lists and groups the most popular digital data collection tools according to their practical application in different housing systems.

Device (based on [55])	Housing system
	Closed	Semi-closed	Free-range
RFID (passive or active)	xxx	xx	xx
Rumen bolus	—	xxx	xx
Walk over weigher	xxx	xx	x
Cameras	xxx	xx	—
UAV	—	x	xxx
GPS	—	x	xxx
Accelerometer	xx	xxx	xx
Pedometer	x	xxx	xx
Microphones	xxx	xx	—

Table 4.

Applicability of PLF tools in different housing systems, Alexy M.’s own research.

Table 4 shows that not all digital devices are suitable for all types of housing systems. The usability of the tools, the cost of acquisition and operation, and the quality and quantity of data that can be collected are of important considerations. The matching and integration of data in different formats into a common database is of particular importance. Ensuring data transmission is one of the most difficult tasks (interoperability) because local data storage must be implemented if there is insufficient bandwidth. These are practical problems that can only be solved, tested, and developed, in the context of pilot experiments.

3.1.2 Livestock characteristics

A key element of precision livestock technologies is to acquire data on individuals of livestock. The data acquisition requires either placing a digital device on the animal or inside a part of the animal or collecting data on the stock. In the latter case, it is necessary to segment the images during data analysis after data collection, by labeling every animal. In all cases, the behavioral and body characteristics of the animal must be known.

Table 5 presents the digital data collection tools that can be considered for three farm animal species (cattle, pigs, and poultry) and assesses their applicability. The digital tools presented are those included in Table 4.

Device	Species of livestock
	dairy and beef cattle (semi-closed)	Dairy and beef cattle (free-range)	Pig (closed)	Pig (free-range)	Poultry (deep litter)
RFID (passive)	xxx	xxx	xxx	xxx	x
RFID (active)	xxx	xxx	—	—	—
Rumen bolus	xxx	xxx	—	—	—
Walk over weigher	x	x	xxx	x	xx
Cameras	xx	—	xxx	x	xxx
UAV	—	xxx	—	xxx	—
GPS	—	xxx	—	xxx	—
Accelerometer	xxx	xxx	—	—	—
Pedometer	xxx	xxx	—	—	—
Microphones	xx	—	xxx	—	xxx

Table 5.

Applicability of digital devices in cattle, pig, and poultry, Alexy M.’s research.

In cattle farming, the milk and meat production directions are indicated separately. The reason for this is the different production purposes of the cattle and the different farming environments. In the case of pigs (although almost 95% of the world’s pig population is kept in confined, intensive conditions), free-range farming is also observed (the purpose is the same, here free-range means organic farming and meeting the needs of other target groups of customers). All digital data collection tools can be used in cattle farming. The possibility to collect individual data from poultry flocks that are kept in completely enclosed conditions is hampered by the body structure of the birds. Namely, birds have no external ears to which, e.g. radio-frequency identification (RFID), tags can be attached and their fast growth rate causes problems since devices that can be placed on the neck or limbs cannot be used because the animals’ body size changes so rapidly that it may damage the animals’ physical integrity. In their case, the use of cameras and microphones is an option. Attempts have been made to use RFID technology on the wingtip, but in practice, this is not a feasible and worthwhile investment.

In livestock farming practice, the specificities of the farming environment largely determine the quality of the data that can be collected. A good example of this is our experience in one of our pilot projects, in which we analyzed camera images of poultry flocks to estimate individual weights and detect behavioral anomalies. In this case, the issue was not with the application of the model (artificial neural network), because the results were surprisingly good, but with devices for the data collection and storage. The metal parts of the computer in the enclosure, which collected the camera images, were so corroded after only two fattening cycles that the device was unusable. The camera lenses had dust on them, a spider had woven a web in front of the lens (only whiteness was visible), and the high temperatures and humidity required to capture the day chicks had fogged the lenses. Consequently, we were able to use only some images for analysis. It was not possible to store the images taken by the cameras on a remote computer (Cloud storage), because the site did not have strong reception (edge storage).

3.2 Data storage and preprocessing

At this stage, the skills of data scientists are needed. In the first two steps, the experts sit at the same table, and in this phase, the domain expert leaves the table but stays in the room. Data preparation consists of several steps: cleaning the data, sorting data in different formats, performing anonymization, reconciling data in different tables, etc. Then the most appropriate data analysis model can be selected. One of the cornerstones to select a data model is the type of data collected (image, sound, number, etc.). By analyzing the aggregated database, data scientists can build a data analysis model by identifying the internal patterns in the data series. The data must have the characteristics that allow a credible and correct analysis to be performed. Data science defines at least six important characteristics (6 V’s, Figure 4), each of which must be present in the data for the database on which the data analysis is based to be usable.

Figure 4.

6 V data characteristics. Source: https://nix-united.com/blog/how-big-data-is-transforming-the-education-process/.

Although this process may seem simple, practical experience shows that it is not. We have experienced this in one of our pilot experiments by ourselves, in which we collected individual data on the daily activity of Mangalica breeding sows kept outdoor. Passive RFID tags were inserted in the ears of 20 sows and readers were installed in an area of the pasture, at the wallowing area. Data were collected on three weather parameters at hourly intervals. The database included the time of arrival and departure of the sows (from which the duration of presence can be calculated) and the data of the three weather parameters (temperature, humidity, and air pressure). After cleaning the database, the frequent item sets method from data mining (apriori algorithm) was chosen to obtain information on the daily activity and social behavior of the sows [56]. However, during the evaluation of this model, we found that it was not suitable to evaluate the database over time and to determine the activity trends of sows. After several months of work, a new model was set up and is currently under analysis. Although the business and data understanding was formulated and the database preparation was successful, the use of an inappropriate model could not answer the practical question.

However, the artificial neural network algorithm used to evaluate images of poultry flocks successfully recognized the birds. This is illustrated in Figure 5.

Figure 5.

Poultry is detected by the neural network algorithm on images. Source: Alexy, M.’s own research.

3.3 Evaluation and deployment

In this step, the domain expert returns to the table: s/he evaluates the patterns established by the model and determines whether the question asked has been answered correctly. The data scientist may have found correlations that are flawed or irrelevant from an expert’s point of view in a livestock domain. If the evaluation shows that the result has added value for the domain expert, the practical application of the solution in the field can begin. This solution will be presented to farmers, then the results are validated whether it has value to farmers. As the practical example aforementioned shows, in the peer review process, it is possible that the desired result is not achieved by applying the wrong model. A livestock professional knows the farming and breeding characteristics of the livestock and understands the complexities involved in producing animal products. They know whether the data science answer to the question asked at the start of the project is appropriate. If so, the results obtained can be applied in practice and disseminated to livestock farmers.

The results of the precision method, validated in a real farming environment, can be successfully integrated into the decision support system of the agricultural enterprise. The outputs of precision methods provide real-time and reliable input information to information systems, which can be used as the basis for complex, strategically important economic decisions. This is presented in the next section.

4. Architecture solution for the problems of smart farming and information supply chain

Recently, agriculture became one of the primary sources of data through the applications of various sensors and IoTs (Internet of Things). In the ecosystem of agriculture, horticulture, and farming, the efficient and effective utilization of data has gained momentum and become an essential issue. Transformation of data collection from the simple substantiated financial data to data that originate from large-scale monitoring and controls of operation led to the requirement of disciplined data analytics. There are data that can be listed as traditional data of company operation and operational data of farming originated from different devices, IoTs. These latter types of data are unstructured and heterogeneous; either we consider their structure or their content. Some pieces of data are accompanied by metadata that may describe the essential information about the content and can be utilized to categorize them and organize them into a data catalog to exploit them for advanced data analytics.

4.1 Agriculture and information management

The development of past decades transformed agriculture and farming regarding digitization and data processing profoundly. Some data sources are available and can be utilized as social media. Web information systems, Internet of Things/sensors, and management information systems of the agricultural enterprises, and electronic images, which were generated by various equipments, have an important role in the assessment of several relevant factors within the production process of farming.

The application of Data Warehouses is apt to structured data that originated from structured databases [57, 58]. The processing of unstructured data by advanced algorithms machine learning and data science, e.g. images, audio, and text, requires cleaned data, a so-called single source of trust that encompasses reliable and trustable data for prediction and prescription. The aforementioned, different types of data can be gathered into an appropriate data architecture that would provide step-by-step transformation (Figure 6).

Figure 6.

Data sources and their utilization in smart farming.

4.2 Data Lake for information management

We suppose that the data originating from the disparate source system are of good quality; if this is not the case, the data preparation process has a built-in procedure for quality improvement and data cleansing. The typical life cycle of data related to farming is showcased in Figure 6. The left-hand side of the diagram contains the potential sources of data that can play an essential role in farming, either agriculture, horticulture, livestock management, or any other modern branches.

A hybrid data warehouse that includes a robust Data Warehouse, as a consequence, leads to a data life cycle that begins with transformation, cleansing, and integration. The purpose of constructing a Data Lake is to have day-to-day operations that separate transactional data from data collection devoted to reporting and retrieving. The original goal of the services of Data Warehouse was to query historical data and to carry out complex data analysis. In the data preparation stage, the data are cleansed and sieved according to the data structure of the target Data Warehouse, and the major constituents of this general structure are the fact table and the associated dimension tables. The next phase includes activities as follows: data migration, data integration, the transformation of the codes that are used in the succinct description of the data, and conversion to transport data between database management systems. The primary objective of the development of the Data Warehouse was to lay a sound foundation of data analysis in a separate system from the production system to avoid performance problems that may have been caused by complex queries. The ETL (Extract, Transform, Load) process is used to load data into a Data Warehouse. In this step, the general data cleaning and transformation took place, e.g. removal of the last and introductory spaces of data items, removal of redundant zeros, standardization of identifiers/identification numbers, making effective restrictions of data fields; and e.g. conversion of imperial units into metric units of measure or vice versa.

While the aforementioned data manipulation is performed, relationships among data items, tables, and schemas may be violated or lost. Analogously, integrating and combining the data from multiple resources can lead to defects that are transferred into the Data Warehouse. To prevail over the data quality restrictions that were caused by the transformations in the Data Warehouse, the idea of the Data Lake was brought into existence. The idea advertised by the notion of Data Lake is to place the data in its original form in storage areas, i.e. in the Transitive and/or Raw Data Zone after the aggregation phase (Figure 7; [59]). Thus, Data Lake can store and obtain data from RDBMS (Relational Data Base Management Systems), semi-structured data (XML, binary XML, JSON, BSON, etc.), and unstructured data (e.g. images), moreover metadata, which are usually displayed in the semi-structured form and characterize the data fed into the Data Lake too. The data entry and preprocessing phase can involve loading, batch, and stream processing for source data while performing the necessary quality checks with the Map-Reduce capability [60]. A vital feature of the Raw Data Zone is that it can be regarded as the “single source of truth” because it retains the data in its original form; however, the data can be anonymized, masked, and tokenized in this zone. Data scientists and business/data analysts can come back to this zone when seeking original connections and relationships among data items that may have become absent during data transformation, conversion, encryption, and encoding. The Trusted Zone implements functions for data processing to ensure quality assurance and to guarantee compliance with standards, data cleansing, and data validation. In this zone, plenty of data alterations take place by the predefined local and global standards, through which the data can be considered “the only version of the truth.” This zone encompasses master data and fact data that are registered and tracked by a data catalog that is automatically or semi-automatically populated with metadata. The data in the Refined Zone endure several additional alterations that are designed to make the data usable in data science algorithms. These data manipulations include structuring of the data format, possible detokenization, a quality check of data to meet the yearnings of the algorithms when models of the subject area (e.g. agriculture and smart farming), and data analysis are developed. In this way, procedures for knowledge acquisition through data exploration and analytics can be performed, and comprehension of the data sets can be achieved. Within each zone, user access rights must be rigorously controlled through adequate methods, e.g. role-based access control or other combined ensemble methods that fit the specific environment. In the event of temporary and ad hoc requirements to deviate from the basic settings, attribute-based access rights or any other adequate approaches can be used. For researchers, executives, authorized employees, and other experts who want to conduct exploratory data analytics, sandbox makes it possible to create models for data analytics and discover associations and relationships between attributes, without involving external or internal experts, without other additional costs [61]. The researcher can feed data from any other zone into the sandbox in a controlled way. Interesting results that came into existence can be sent back to the Raw Data Zone for reuse.

Figure 7.

Data Lake architecture for agriculture.

4.3 Agriculture and information system architecture

Zachman’s framework contains various viewpoints of business stakeholders and a set of models describing the essential facets of overarching information systems. In Table 6, the perspectives represent the various layers of the enterprise architecture in the sense of modeling tools, software, and operational infrastructure. The aspects can be perceived as a line of models, in that the lower-level model is a refinement of the upper-level model. The claimed advantage of the Data Lake is that the data extracted from the source systems are transformed before the actual use of the data for analysis. This approach permits more adaptability to requirements than the controlled, structured environment of Data Warehouses.

Aspects perspectives	What	How	Where	Who	When	Why
Contextual	Fact, business data/for analysis	Business Service Business Intelligence	Chain of Business Processes, Workflows	The business entity, function	Chain of Business Process, Workflow	Business goal	Scope
Conceptual	Underlying conceptual data model/structured, semi- and unstructured data	Business Intelligence with added value originated	Workflow	Actor, Role	Business Process Model	Business Objective	Enterprise Model
Logical	Class hierarchy, logical data model structured, semi-structured and unstructured data	Service Component Business Intelligence, Data Analytics	Hierarchy of Data Analytics Service Component	User role, service component	BPEL, BPMN, Orchestration	Business Rule	System Model
Physical	Object hierarchy, data model (object store)	Service Component Business Intelligence, Data Analytics	Hierarchy of Data Analytics Service Component	Component, Object	Choreography	Rule Design	Technical Model
Detailed	Data in SQL/NoSQL and other file structures	Service Component Business Intelligence, Data Analytics	Hierarchy of Data Analytics Service Component	Component, Object	Choreography, Security architecture	Rule specification	Components
Functioning Enterprise	Data	Function	Network	Organization	Schedule	Strategy

Table 6.

A mapping schematically between Zachman architecture and components of Data Lakes [62].

The purpose of the Data Lake and Data Warehouse dedicated to research within agriculture informatics is to lay the foundation of data analytics workflows. The architecture should support several requirements as follows: (1) achieving processing speed through adjusting configuration parameters; (2) exploitation of the distribution of data among cluster nodes; (3) usage of provenance data; and (4) data placement and scheduling algorithm for input data to prepare the efficient data processing.

The contextual perspective describes the goal to which the system is dedicated. In the case of data analytics workflow in e-agriculture, the objective is to assist the management to formulate the research questions in terms of business processes. At the conceptual level, the disparate models that are devoted to specific research questions and associated with various approaches to data analytics appear. In the logical layer, the specification of the data analytics workflows is exhibited using business process description languages, algorithms of data analytics, and services for data access. The logical level should contain the explicit specification of access rights taken into account of the different regulations, especially GDPR and sector-specific prescriptions. The physical layer contains the technology-specific arrangements and solutions. Besides the actual programs containing algorithms of Data Science, and data preparation activities, the physical level should contain the number of physical-level data processes (executors) that carry out the particular workflow, the number of concurrent tasks, the allocable memory, etc. The implementation and operational layer (the detailed level) depicts the scientific workflows on an infrastructure that contains nodes of data processing that are realized as commodity hardware and the decomposed models into details, respectively (Table 6).

4.4 Enterprise resource planning systems, Data Warehouse/Lake for information management in agriculture

In large and small- and medium-sized enterprises (SMEs), the data is transmitted from disparate systems. The essential constituents of the data sources are the Enterprise Resource Planning Systems (ERP), Customer Relationships Management (CRM), and Supply Chain Management (SCM) as the elements of information systems architecture in a company (Figure 8). Nowadays, SMEs and even microenterprises apply ERP systems, although open source or open access solutions through Clouds. The requirement is that the data that is produced by ERP systems is to be stored and archived for later data analytics. Generally, in various industry sectors, Data Warehouses play a significant role in administering data.

Figure 8.

Enterprise resource planning (ERP) system and Data Warehouse (DWH) and Data Lake architecture for data analytics in agriculture.

The Data Warehouse along with Data Lake in agriculture make it possible to support decision-making, and ground in durable, reliable, and trustable data analytics for the huge size of data both in real time and batch processing (see [63, 64] in other sectors of the economy). The primary role of Data Lake in an enterprise environment is to yield the chance for data integration and reconciliation; the decisive role of Data Warehouse is to provide the opportunity to integrate data in a structured manner and format; moreover, it stores persistently and efficiently the original data in the fact table from the disparate modules of ERP.

Data Analytics and Business Intelligence tools can use DW as the major source of structured data and provide insights [65, 66, 67]. The structured organization of DW serves as a sound foundation to acquire a holistic perspective of business process performance by senior managers, business workgroups, and data scientists. Generally, the Key Performance Indicators (KPI) can be monitored and tracked. The application of dashboards gives a cleaner, precise, reliable, trustable, and easy-to-access picture of the actual state of the enterprise. The dashboard lays the foundation for effective decision-making. The data feeding or ingestion framework should be built up in the case of Data Lake. In a Data Lake, structured and unstructured data should be handled in a secure environment, considering the data protection requirements. Adequate libraries and workbenches are needed for data analytics tools and machine learning. The data provenance and metadata management can be realized through an advanced machine learning tool set and data catalog in the Data Lake (Figure 8). The disciplined and strict access rights should be implemented through sophisticated single sign-on and multi-factor authorization and authentication methods.

Data Quality should be achieved through the application of combined tools in both cases, in DW and Data Lake. The processes of data quality should deal with the erroneous data and pass it to the data quality team for human interaction whether what format can be accepted and inputted.

Integrating Data is an essential function of DW and the operational teams. The goal is to transform the data into the standard structure and format and to create consistent, interpretable, and meaningful data. This process is realized by a data transportation tool that conveys the data from different systems into DW and into the appropriate zone of the Data Lake. This process applies heterogeneous standardization and data cleaning methods that are related to the domain.

5. Conclusion

After showcasing research in agriculture for precision farming in Chapters 2, and 3, we have proposed the application of contemporary, modern information architecture that is capable to process data efficiently and effectively. The focus point is to support decision-making for enterprises including large, SMEs, and even microenterprises. Since there are available solutions in the IT market that can be scaled to the size of the enterprise. Especially, the Cloud-based solutions can be regarded as viable for SMEs and microenterprises.

Acknowledgments

This research was partially supported by the project of “Application Domain Specific Highly Reliable IT Solutions” that has been implemented with the support provided by the National Research, Development, and Innovation Fund of Hungary, financed under the Thematic Excellence Programme TKP2020-NKA-06, TKP2021-NVA-29 (National Challenges Subprogramme) funding scheme.

Conflict of interest

“The authors declare no conflict of interest.”