Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
The advent of machine learning, of which artificial neural networks (ANN) are a component, has provided an opportunity for improved rainfall forecasts, which is of value for water infrastructure management, agriculture, mining and other industries. In this chapter, ANNs are shown to provide more skillful monthly rainfall forecasts for locations in south-eastern Queensland, Australia, for lead-times of 3–12 months. The skill of the forecasts from the ANNs is highest when the models are individually optimized for each month, and when longer-duration series are used as input. The ANN technique has application where there is temperature and rainfall data extending back at least 50 years. Such datasets exist for much of Europe and North America, though a review of the available literature indicates most research into the application of ANN has focused on China, India and Australia.
chapter and author info
University of Tasmania, Tasmania, Australia
Institute of Public Affairs, Melbourne, Victoria, Australia
*Address all correspondence to: email@example.com
Until relatively recently, simple statistical models were used by meteorological agencies around the world to forecast seasonal and monthly rainfall. Typically, these models use relationships between large scale climate indices, such as the Southern Oscillation Index, and rainfall at some future time, generally utilizing a small number of input variables, perhaps only one or two. For example, until May 2013 the Australian Bureau of Meteorology (BOM) generated seasonal rainfall forecasts based on a statistical scheme using an El Niño Southern Oscillation (ENSO) index as a primary predictor in a relatively simple statistical model [1, 2].
These traditional statistical models are limited in the number of input variables that can be effectively combined, while advances in machine learning has now significantly expanded this potential . Machine learning has close relationships with artificial intelligence, pattern recognition and data mining. Data mining focuses on the discovery of previously unknown properties embedded in data , whereas machine learning focuses on prediction based on known properties learned from exposure to data sets during a process known as “training”. A principle objective of the learning process is to construct a model that can generalize from experience . Subsequently, the performance of the trained model can be tested using data not utilized in the training set. Performance of the model in testing gives confidence, but not certainty, that the model would provide reliable forecasts if deployed operationally.
Artificial neural networks (ANNs), a form or machine learning, provide several important advantages over simple statistical models. ANNs can accommodate non-linear relationships, and test multiple inputs, and this is particularly important when the influence of climate indices may vary geographically and temporarily in poorly understood ways .
Rather than progressing from simple statistical models to artificial neural networks, however, meteorological agencies around the world now tend to rely almost exclusively on general circulation modeling for rainfall forecasting. These are physical models, that attempt to simulate real-world oceanic and atmospheric circulation patterns. For example, the Australian Bureau of Meteorology uses the Predictive Oceanic Atmospheric Model for Australia (POAMA) as its operational model for forecasting daily rainfall, and also as the basis of its monthly and seasonal forecasts.
The skill of the forecasts from ANNs can be compared with POAMA (and other general circulation models) through a comparison of root mean square errors (RMSE), mean absolute errors (MAE) and correlation coefficients – with such comparisons a focus of this chapter. RMSE is commonly applied to compare skill between different rainfall forecasting models, as it gives a simple, transparent, quantitative measure of difference between input and target and is easily understood across disciplines . RMSE is more sensitive than MAE to the occasional large error (the squaring process gives higher proportionate weight to the large errors), and is therefore arguably more useful when skill at forecasting floods is particularly relevant.
Given the importance of skillful monthly rainfall forecasts for most inhabited regions, for activities as diverse as crop harvesting, mine scheduling, and dam management, it is surprising that there are so few comparative studies and so little discussion about how advances in machine learning might aid medium-term rainfall forecasting – including the forecasting of flood events.
Queensland is a state in the north-east of Australia, facing the South Pacific. Brisbane is the fast-growing capital of Queensland, and is located in the south-east. Brisbane has a long history of flooding and the Wivenhoe dam was built specifically for flood mitigation following a flood in 1974.
The flooding of Brisbane in January 2011 has been termed a “dam release flood”  with the sudden release of water from the Wivenhoe storage a principal cause of flooding. The extent of the rainfall in 2010/2011 was not unprecedented relative to historical records that extend back to 1864; but the heavy rainfall was not forecast . Because the heavy rain in December 2010 and January 2011 followed a long period of drought, and was not forecast, the Wivenhoe dam was not properly managed for flood mitigation – the purpose for which it was originally built. Because of the extent of the economic losses the flood event has led to a class action lawsuit against the dam operator SEQEB and the State of Queensland, with the trial expected to commence in late 2017 .
2. Data, method, and a first study
Monthly data were obtained from the Bureau’s Climate Data Online. Data was downloaded for all 62 sites in south eastern Queensland that are considered in this chapter. The sites were chosen on the basis of their geographic spread and also the quality of the data: that is the desirability of long data series with few missing values.
South-eastern Queensland is defined very broadly in this chapter, and does not correspond with the administrative region of South East Queensland.
This chapter is a review of various studies undertaken since 2012 focused on this general area, with specific information on data and methodology in the published technical papers that are referenced [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]. However, in this first section, the method used in an early study  is provided in more detail, by way of background into how an ANN can be practically deployed to generate a rainfall forecast.
Many neural network applications incorporate multilayer perceptrons (MLPs) as fundamental processing elements (PEs) trained with a standard backpropagation algorithm. These neural networks can perform well in solving static problems but are limited in solving temporal problems, ones where the previous value of the input affects the current output. Recurrent networks, such as Jordan networks, extend the basic MLP architecture by also including context units, PEs that remember past activity. In the Jordan network, the output of the network is copied to the context unit.
In addition, the context units are locally recurrent, that is, they feedback onto themselves. The local recurrence decreases the values by a multiplicative time constant (τ) as they are fed back. This constant determines the memory depth, that is, how long a given value fed to the context unit will be “remembered”. The context unit acts as a simple lowpass filter, creating an output (y(n)), calculated as a weighted average value of some of its more recent past inputs. In the case of the Jordan context unit, the output is obtained by summing the past values multiplied by the scalar τn, where:
Genetic optimization provides an efficient way of selecting those inputs that are significant in determining target rainfall and eliminating those inputs with very low information content. Essentially, genetic optimization enables elimination of inputs that carry mainly noise rather than useful signal, so that the number of input considered in the optimized model might typically be reduced from over 40 to less than 10.
In the early study  forecasts were made for Lowood and two other sites within the Brisbane catchments (upstream of the Wivenhoe dam) using the Jordan network with optimization. The initial forecasts were made using only lagged input parameters – that is any variable measured in the past. The initial focus was on:
Benchmarking output from the ANN against POAMA (the operational model used by the Bureau);
Evaluating the impact of 1-, 2- and 3-month lags, and accordingly the extent to which a rainfall forecast potentially loses skill moving from forecasting 1, 2 and 3 months in advance;
Determining the capacity of the ANN to deal with unary, binary and ternary data sets, that is, with an increasing number of input variables;
Determining if the ANN model could forecast the extreme rainfall of December 2010 and January 2011 for these sites; and
A ‘best forecast’ using only lagged values was constructed to explore the potential to improve forecasts through post-processing, as shown in Figure 1.
Figure 1 shows the skill of this forecast, as an orange line, relative to observed rainfall, blue line. Clearly the ANN was able to forecast that December 2011 was likely to be unusually wet at Lowood. The rainfall of over 600 mm for the month of December contributed to the flooding of Brisbane in early January 2011.
In this early study , several climate indices were also successfully forecast and then inputted as lead variables. A ‘lead’, also known as a forecast value, are un-measurable from the reference point of the current period, but potentially predictable in a forecast model.
Variations in rainfall in many parts of the world, including south-eastern Queensland, are associated with large-scale climate phenomena which can be described by climate indices typically measuring changes in temperatures and pressures across oceans [22, 23, 24]. ENSO, a Pacific Ocean phenomenon, can be represented by both the Southern Oscillation Index (SOI) and a combination of four different Niño values (Niño 4, Niño 3.4, Niño 3, Niño 1.2). The Interdecadal Pacific Oscillation (IPO) also measures pressure and temperature changes in the Pacific Ocean. The Indian Ocean Dipole measured by the Dipole Mode Index (DMI), is a measure of pressure and temperature changes in the Indian Ocean.
These large-scale climate indices were used as input data, together with local data for each site specifically rainfall, and temperatures. This data was always divided into training (75%), evaluation (15%) and testing sets (10%) – with absolutely no overlap.
3. Progressing to automation and single-month optimization
A wide range of ANN architectures have been applied in forecasting rainfall [11, 13, 25, 26, 27]. The selection of an ANN architecture is commonly achieved through a trial and error process [11, 12, 13]. This can, however, be very time-consuming: that is choosing network topology, and architecture based on trial and error – with the selected model with lowest error score, subsequently applied with all the data input sets.
In contrast, with the Neurosolutions Infinity software that we used since 2015 [19, 20, 21] the selection of network architecture and configuration was automated. This offered a great advantage in terms of arriving at an optimum forecast model for each data set of interest without prohibitive time outlay. The Infinity program uses a pre-set formula incorporating RMSE, MAE and correlation coefficient values to evaluate the accuracy for each ANN model and a corresponding set of selected inputs. Based on this formula, the program determines which ANN model and set of inputs is optimal.
In the early study using the Jordan ANN model (Lowood example), and also subsequently with the Infinity software automating the process of selection of network architecture, the default was for optimization of “all-months” within each calendar year at the same time. That is, it was assumed that the same climate indices would be important for each of the months of the year. This did not prove valid. Indeed, it is well known within the climate science literature, that the magnitude of the linear correlation between SOI and annual rainfall are highly variable both temporarily and geographically particularly for Australia [24, 28] has shown that other ‘remote drivers’ of Australian rainfall dominant at different times of the year. Furthermore, this temporal variability has been a reason why forecasting beyond autumn in the southern hemisphere, and spring in the northern Hemisphere spring has been considered particularly problematic [29, 30, 31].
In a study published in 2015  we showed for the first time that the prominent rainfall peak in December 2010 for the location of Harrisville, which is near Lowood, could be much more skillfully forecast using a single-month optimization technique. Further, the very heavy rainfall could be forecast at the long-lead times of at least 12 months, as shown in Table 1. We have since extended the method of single month optimization to show its relevance to locations along the Queensland coast .
Skill parameters for monthly rainfall forecasting for Gatton and Harrisville on an annual basis for the test period July 2004 to August 2011 .
The process is much more time consuming as 12 optimizations are carried out to produce monthly rainfall forecasts for the entire year. Improved skill, however, is generally achieved as demonstrated by reduced MAE and RMSE and increased correlation coefficient r.
While single-month optimization clearly gives a lower RMSE and MAE than all-month optimization and therefore is a preferred forecast method, it should be noted that both methods of ANN optimization provided a superior forecast to climatology and POAMA as shown in Table 1.
The results shown in Table 1 where achieved from a general regression neural network (GRNN). This same ANN architecture was used to generate forecasts for Bingera, which is in a coastal catchment to the north of Brisbane. Both single-month and all-month optimization methods gave skillful forecasts for Bingera, as shown in Figure 2.
4. Best forecasts from longer datasets
Our very first paper on forecasting rainfall using ANNs  focused on 17 sites, chosen specifically because they had the longest continuous rainfall records for anywhere in Queensland. In that paper, we used a temperature series from the relatively far away destination of Sydney – with the comment that there is no comparable temperature dataset available for any Queensland location. We have since used local temperature series, even if this requires the infilling of missing values through interpolation and/or linear regression .
In our early work, we also limited the selection of climate indices to those of ‘long duration’ – specifically ENSO (SOI and the four Ninos), IPO and DMI. These climate indices were downloaded from the Royal Netherlands Meteorological Institute (KNMI) Climate Explorer.
There are many more climate indices available, but most are of much shorter published duration, often only beginning in the 1950s. These ‘short’ duration indices are listed in Table 2, specifically the south-eastern Indian Ocean index (SEIO), Western Indian Ocean Index (WIO), The Southern Annular Mode (SAM) and the Quasi-Biannual Oscillation (QBO). Data for these indices were variously sourced from the Bureau, KNMI Climate Explorer, and also the UK Met Office for the recent publication  in which we tested the trade-off between more climate indices and shorter series as input.
Attributes used as input for ANN models.
The skill of an ANN is dependent on finding patterns or relationships in data, and we intuitively thought that input series of less than 100 years would be unlikely to provide adequate information. In a recent study for a region in central Queensland it was apparent that the skill of the forecast depended on the particular month for which the forecast was being made, as shown in Figure 3. Nevertheless, it was apparent that of more importance than inputting long-duration series, was single-month optimization. When single month optimization was applied to both long and short duration series, the long duration series gave superior forecasts, as shown in Figure 3. More information on the definition of, and methodology used, to calculation the skill score is provided in the technical paper  and under the following section: Deterministic versus Probabilistic Forecasts, and Skill Scores.
5. Regional rainfall forecasts
The rainfall forecasts discussed so far correspond to specific locations within a region. The limitation in selection of those sites depends on the availability of sufficient historical data for adequately training and testing. In some cases, shorter data sets may be adequate, and shorter data duration may be offset by inclusion of additional climate indices .
Provided there are sufficient well-distributed sites within a geographical region, it may be possible to construct a regional rainfall forecast map, as shown in Figures 4 and 5. This enables forecasts to be generated for specific locations within the geographical region that do not correspond to sites with rainfall data. It also enables rainfall forecasts to be made corresponding to a defined geographical area within the region, such as a water catchment area.
As an example, Figure 4, is a specific sub-region of south-eastern Queensland defined by longitudes 151.0°E and 153.5°E, and latitudes 24.4°S and 28.5°S based on rainfall data from 54 sites. The region extends about 300 km southwards along the Queensland Coast from Bundaberg to the Gold Coast, and extended inland from the coast to include the towns of Dalby, Inglewood and Mundubbera, a distance of about 200 km. This sub-region therefore approximates the same area as a single grid forecast area used by POAMA (250 km × 250 km).
This example is provided to show that it is possible to develop an isohyet map for such a region derived from the individual series to predict rainfall for a period of intense rainfall, as occurred in December 2010, and also a relatively dry month such as December 2005. The isohyet map was constructed using Teraplot software – and the forecast are 12 months in advance using single-month optimization .
In Figures 4 and 5, it is evident that rainfall varies with topography, and was highest at the places of some altitude in both the relatively dry December of 2005, and also the flood month of December 2010. It is also evident that the ANN correctly predicted December 2010 as experiencing relatively higher rainfall across the sub-region, peaking at over 700 mm.
6. Deterministic versus probabilistic rainfall forecasts, and skill scores
In this chapter, we have shown that forecasts generated through application of an ANN are deterministic, that is numerical values are produced corresponding to the expected rainfall for a specific month. In contrast, medium-term rainfall forecasts issued to the public by the Australian Bureau of Meteorology are in the form of probabilities relative to the median seasonal rainfall, and do not differentiate between an anticipated rainfall slightly above the median and an extreme rainfall event, such as occurred in Queensland during the period December 2010 and January 2011 .
This is illustrated in Figure 6 showing the seasonal forecast issued by the BOM for the period December 2010 to February 2011 for the entire continent of Australia (Bureau of Meteorology, Archive of rainfall forecasts). Although the forecast indicates that there is a probability in the range of 60–70% above median rainfall for the south-east Queensland region, there was no warning of the magnitude of the impending heavy rainfall. Furthermore, the extensive flooding that affected much of costal Queensland beyond the south-eastern region where the forecast is a 50% probability of above median. This contrasts with the specific quantities forecast for the isohyet maps shown in the previous section.
Not only is the information content of probabilistic forecasts less than corresponding deterministic forecasts, where actual numerical values of predicted rainfall are provided, studies have demonstrated that the general public and specific classes of end-users, such as farmers from south-east Queensland, often have difficulty interpreting the meaning of the probabilistic forecasts [33, 34]. The distinction between median and average was often not understood. In addition, the statement that there was a 30% probability of above the median rainfall was often misinterpreted to mean a forecast of a specific quantity of rainfall 30% higher than the median. There is confusion between a probabilistic and deterministic forecast that can be understood by the public as the expectation of higher rather than lower rainfall as intended by the forecaster.
The Bureau bases its probabilistic forecasts on the simulation of actual physical climatic processes through general circulation models, specifically POAMA [36, 37, 38]. For the period from 2002 through until 2011, seasonal rainfall predictions were made using POAMA 1.5.
In July 2011, the Bureau provided us with monthly forecasts for the period to March 2011 for 17 individual sites as simple bilinear interpolations of surrounding grid points which were calculated from the ensemble mean which in turn had been calculated from many runs of this general circulation model, POAMA 1.5. Figure 7 shows the difference between the observed monthly rainfall, together with forecasts generated by POAMA 1.5 for the site of Miles. The forecasts generated by POAMA show little variation between the 6-year period 2004–2009 and the occurrence of heavy rainfall in 2010: each showing a maximum of approximately 100 mm. In contrast, the forecast generated by the ANN clearly show much higher rainfall of about 350 mm, closely corresponding to the observed value, Figure 7 (Table 3).
Comparison of rainfall forecast skill for Miles using different forecasting methods at leads of 3, 6 and 9 months for the period July 2004 to March 2011.
Evaluations of forecast skill were also made by comparisons of skill scores calculated relative to climatology from Eq. (1). This is analogous to the method used by the Bureau where a skill score is calculated for forecasts using POAMA .
When the calculated value of the RMSE from climatology and for a particular model are equal, the forecast skill score will be zero. For a perfect forecast, the RMSE for the model will be zero, and the calculated skill score 100%. Negative values calculated from Eq. (2) indicate a forecast skill score worse than climatology. Table 4 shows skill scores for each individual month using an ANN with a lead time of 12 months and POAMA with a lead time of 8 months. In all cases the ANN skill scores are positive and lie in the range 23–69%. In contrast, the skills cores for POAMA forecasts are negative, that is worse than climatology.
Forecast skill as a percentage for monthly rainfall forecasts for Miles with reference to climatology for the composite ANN model and POAMA.
Hawthorne et al.  used the output from POAMA to produce monthly rainfall forecasts with up to 8 months lead time for 250 km × 250 km grid areas over continental Australia. The skill of their monthly rainfall forecasts was described as being generally low. Skill scores fell between −20% and 20%, including grid locations in coastal south-east Queensland. For lead times between 3, 4, 5, 6, 7 and 8 months, with approximately 60% of the forecasts give skill levels relative to climatology below 0%. For southeast Queensland only about 20% of the forecasts had a skill level in the 15–20% range  Hawthorne et al.  described the skill of their monthly rainfall forecasts as low, and concluded that monthly rainfall forecasting with POAMA remained a challenge.
7. Other ANN studies
The comparative data we have shown for specific locations, and also the isohyet for the sub-region, indicate that ANNs can give more skillful forecasts than the general circulation models currently relied upon by the Australian Bureau of Meteorology.
We have also published a series of articles describing the application of ANNs to monthly rainfall forecasting in other parts of Australia including northern and central Queensland [11, 12, 13, 15, 16, 17] the Murray Darling Basin  and Western Australia . The forecasts for these regions also use a combination of large-scale climate indices and the local variables temperature and rainfall with lead times between 1 and 18 months.
The island state of Tasmania lies to the south of mainland Australia, and has an area similar to that of the region of south-eastern Queensland considered above. Tasmania is primarily reliant on hydro-electric power, and therefore the state is highly dependent on rainfall for electricity generation. Until 2005, Tasmania was self-reliant for electricity when the Basslink interconnector to mainland Victoria was completed. This link was intended to both provide energy security, in case of drought; and provide renewable energy to Victoria, which relies heavily on coal for power generation. Hydro Tasmania water storage levels decreased from over 60% capacity in 2013 to below 15% in early 2016. The state experienced an energy crisis in 2016 following 2 years of high volumes of energy export to Victoria, followed by low rainfall and a fault in the link to the mainland.
This situation emphasized the need to incorporate more accurate rainfall forecasts in planning decisions relating to hydro-electricity generation. The skill of monthly forecasts for this region of Australia at 2–3 months lead time generated by the Australian Bureau of Meteorology for this region are in the range −10 to 10%, averaging close to climatology . Tasmania has at least 30 sites with long rainfall records extending back about a century or longer, distributed over the island. It should therefore be possible to construct regional or catchment forecasts for medium-term rainfall. One example is shown for Hobart, the capital of Tasmania in Figure 8. This shows forecasts of monthly rainfall at a 3 month lead time for a test period of 10 years from January 2004 to December 2013. Table 5 gives the corresponding skill scores for each month, which lie in the range 34–71%. This is significant improvement on that currently achievable using general circulation models.
Forecast skill as a percentage for monthly rainfall forecasts for Hobart with reference to climatology for the composite ANN model and POAMA.
Other studies, including by Bagirov et al.  predicted monthly rainfall for 8 sites in Victoria with 5 input meteorological variables, using data over the period 1889–2014. Montazerolghaem et al.  made monthly rainfall forecasts from 136 weather stations in south-eastern Australia in the states of Victoria, NSW, South Australia and southern Queensland with 1 month lead time. Using large scale climate indices, He et al.  made forecasts of monthly rainfall at 255 stations across Australia with training data between 1959 and 1889 test data between 1999 and 2008 with large-scale climate indices as inputs.
ANNs have been applied to rainfall forecasting in other parts of the world [25, 26, 43], with Table 6 showing a survey of 36 articles published between 2014 and 2017 where ANNs have been used to forecast monthly or seasonal rainfall. The majority of these investigations were from Asia, particularly India and China.
Survey of publications on rainfall forecasting 2014-2017.
ANNs have also been applied to rainfall forecasting at both shorter forecast periods, such as hourly  and daily  as well as longer forecast periods including annual . ANNs have also been used to forecasting other climatic and meteorological variables including temperatures [47, 48, 49, 50] and wind speed [51, 52]. Other machine learning methods have also been applied to rainfall forecasting; including support vector machines (SVM)  and adaptive neuro-fuzzy inference systems (ANFIS) [54, 55, 56].
The survey results detailed in Table 6 indicate that there have been very few studies of ANNs and their application to rainfall forecasting in North America and Europe. These are regions that are likely to have the longest rainfall and temperature series and therefore are potentially ideally suited to such studies.
While general circulation models are used by meteorological agencies around the world for rainfall forecasting, they do not generally perform well at forecasting medium-term rainfall, despite substantial efforts to enhance performance over many years [36, 81, 82, 83]. These are the same models used by the Intergovernmental Panel on Climate Change (IPCC) to forecast climate change over decades. Though recent studies suggest ANNs have considerable application here, including to evaluate natural versus climate change over millennia, and also to better understand equilibrium climate sensitivity .
While machine learning is now a well-established discipline, and artificial neural networks a well understood subcomponent, this technology is only beginning to be applied to rainfall forecasting, so far with most of this effort concentrated in China, India and Australia as shown in Table 6. The technology is likely to have application to many more regions, particularly North America and Europe where longer time series of rainfall and temperature is likely to exist.
In this chapter, we demonstrated the application of ANNs for rainfall forecasting for specific locations and also a sub-region within south eastern Queensland, Australia. We demonstrated that the ANN could forecast the extreme flood event of December 2010 that resulted in the catastrophic flooding of Brisbane in January 2011. We show that the skill of the ANN forecast can be improved through single-month optimization of the ANN model, and also by using datasets that extend further back in time. We compare forecasts from the ANN with forecasts from a general circulation model, POAMA, and show that both the deterministic nature of the forecasts, and also their actual skill, indicates that ANNs have much to offer by way of improved monthly rainfall forecasts at least 1 year in advance.
This work was supported by the B. Macfie Family Foundation.
John Abbot and Jennifer Marohasy (December 27th 2017). Forecasting of Medium-term Rainfall Using Artificial Neural Networks: Case Studies from Eastern Australia, Engineering and Mathematical Topics in Rainfall, Theodore V Hromadka II and Prasada Rao, IntechOpen, DOI: 10.5772/intechopen.72619. Available from:
Strategies for Testing the Impact of Natural Flood Risk Management Measures
By Barry Hankin, Peter Metcalfe, David Johnson, Nick A. Chappell, Trevor Page, Iain Craigen, Rob Lamb and Keith Beven
We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.