The 15 items identified.
In the literature, a lack of strong consensus or well-known theoretical research framework exists to defining and evaluating e-commerce success among small to medium enterprises (SMEs). Exploring more effective methods to describe and evaluate e-commerce success becomes a challenging task. This research seeks to help fill the gap by proposing a new model to evaluate e-commerce success from a business perspective. This measure has been termed e-commerce business satisfaction (EBS). A total of 2401 surveys were successfully sent to SMEs. The usable response rate for the surveys was 7.54%. Principal component analysis with varimax rotation method was then adopted within the factor analysis. Using the 15 critical success factors (CSFs) obtained from previous research as a foundation, an EBS management model was finally simply developed to assist SMEs business managers in effectively adopting e-commerce systems or evaluating e-commerce success, which was categorised into five components including Marketing, Management Support and Customer Acceptance, Website Effectiveness and Cost, Managing Change and Knowledge and Skills. Further research is needed to determine the weighting of each CSF so that a yardstick measurement method might be further developed to assist SMEs in adopting e-commerce successfully.
- E-commerce satisfaction
- E-commerce success
- management model
Electronic commerce (e-commerce) is not a new concept, but it has had increasing and unpredictable developments . The fact is that the number of small to medium enterprises (SMEs) adopting e-commerce has increased significantly in recent years. It has been proved that successful e-commerce/IS use can create net benefits concerning financial and operational performance for SMEs, even in developing countries . A significant number of SMEs have, however, failed in adopting e-commerce while many businesses are not satisfied with their e-commerce systems. The research stated, therefore, that quantifying the value contribution of e-commerce has become an issue for managers seeking to justify the enormous expenditures involved in new IT investment .
Among the research methods, the evaluation has been one of the five top research areas on the adoption of e-commerce along with trust, technology acceptance and technology application, e-commerce task-related application, and e-markets, which resulted from the analysis of a total of 1064 e-commerce-related articles and 33,173 references published in leading e-commerce journals between 2006 and 2010 . Researchers have also enunciated the need for evaluating e-commerce success as avoiding failure again, learning from experience, indicating actual business benefits, the requirement for adoption guidelines, and for further improvement and development .
Investigation to date does not clearly show how to evaluate the success of e-commerce systems . In the investigation of e-commerce, many research studies use the IS success model to evaluate e-commerce systems . For example, the original or updated DeLone and McLean model have been widely used for evaluating the degree of IS/e-commerce success . The Technology Acceptance Model (TAM) and its extensions have also been used as reliable and robust models for predicting the user acceptance of e-commerce .
IS success approach may not, however, be methodologically and theoretically feasible in e-commerce among SMEs . The literature has noted that the difficulties existed in using such IS models and its extensions . The main difficulty has been highlighted that the determinants of e-commerce might be dissimilar to the concepts in IS success studies [5, 10]. Other difficulties could be focused on the involvement of top management, beyond Internet technology and lack of experiences .
In the literature, no strong consensus or well-known comprehensive and integrated theoretical framework currently exists [10, 11, 12]. Research frameworks also lack a theoretical approach to defining and evaluating e-commerce success among SMEs [10, 11, 12]. Exploring more effective methods to describe and evaluate e-commerce success, thereby, becomes a challenging task [13, 14]. This research seeks to help fill the gap by proposing a new model to evaluate e-commerce success from a business perspective.
2. CSFs and E-commerce success
2.1. Using business satisfaction for evaluating e-commerce success
In the literature, satisfaction is a very important element for a successful long-term relationship with e-commerce adoption/success . Research highlighted that satisfaction rather than system use was adopted as an appropriate measure of e-commerce success . Any unsatisfactory performance regarding this criterion could not be compensated for by better performance concerning one or more other criteria . Satisfaction study is, therefore, being used widely for a better understanding of e-commerce success.
This research proposed that the term e-commerce business satisfaction (EBS) would be a better measure of e-commerce success where e-commerce satisfaction was discussed from a business perspective, which has been previously defined as ‘the overall satisfaction that a business has with an e-commerce system meeting its requirements and expectations’ .
2.2. Using CSFs for understanding EBS to evaluate e-commerce success
The concept and approach of CSFs have been studied and applied successfully in a broad range of contexts in many areas of IS/IT research including identifying system needs [11, 18]. Research states that satisfaction with e-commerce is significantly affected by organisational determinant critical success factors (CSFs) .
Using a set of CSFs identified as one of the key implementation strategies employed in the context of performance analysis and strategic management, SMEs may focus on and manage the few key areas in the implementation process to measure their e-commerce systems, judge the efficacy of e-commerce systems, improve their e-commerce systems, and predict e-commerce usage, which could contribute to the success of these experiential-driven initiatives and help achieve a successful implementation outcome of e-commerce [14, 18, 19, 20, 21, 22]. Identifying CSFs that impact the adoption of e-commerce will also make it possible to assess its future growth .
Over the past decade, therefore, a large number of investigations have focused on determining what CSFs affected the adoption of e-commerce successfully [23, 24]. Despite the increasing work on e-commerce, however, the investigation indicated that there were still very few studies that attempted to investigate the effect of proposed CSFs on the implementation of e-commerce  and also lack of an integrated approach for the development of a well-established e-commerce CSFs model . This research adopted the CSFs identified in previous work for understanding business satisfaction associated with e-commerce success.
3. Research methodology and the survey results
A blend of research methods consisting of focus group studies, pilot tests, and surveys have been used and discussed.
3.1. Focus group studies
A focus group study involves a formalised process of bringing a small group of people together for an interactive and spontaneous discussion or interview of one particular topic or concept . With origins in sociology, focus group study became widely used in market research during the 1980s and was used for more diverse research applications in the 1990s . Focus group studies might help prepare for a survey by providing sufficient information about the survey objective, by defining and improving indicators and about preventing possible errors .
Most experts agree that the optimal number of participants in any type of focus group interview is from six to 12 members in nondirective interactive communications facilitated by a moderator who prepares and uses a loosely constructed set of relevant questions [27, 28, 29]. In this research, a target of 18 SMEs business managers (nine for each in Australia and China) were formed to define and improve indicators preventing possible errors.
3.2. Pilot tests
A pilot test is an important and essential step in checking the rigour of the survey instrument and the need for any final modification before conducting the survey proper [28, 30, 31, 32]. The objectives of this step were to examine the validity of each item in the survey and to avoid any misleading cultural differences due to inaccurate translation , as it is extremely difficult even for experienced social scientists to write a questionnaire .
Research advises that a pilot test of 20–50 cases is usually sufficient to discover the major flaws in a questionnaire before they damage the main study . Research further suggested that researchers use open questions in pilot tests and then develop closed-question responses from the answers given to the open questions for large-scale surveys .
Twenty businesses were involved in pilot tests (10 for each in Australia and China) in this research, which were carried out with open questions to modify the proposed questionnaires and any errors.
3.3. Surveys and survey results
The survey samples were selected first. Since most computer programs use standard error algorithms based on the assumption of simple random samples, the standard errors reported in the literature often underestimate sampling error . The goal of the sampling is to collect data representative of a population within the limits of random error, which the researcher then uses to generalise findings from a drawn sample back to a population . It is critical that the chosen respondents are representative of the study population . The random sampling method was chosen in this research to select samples  as follows:
Stage 1: random sampling of big clusters.
Stage 2: random sampling of small clusters within each selected big clusters.
Stage 3: the sampling of elements from within the sampled small clusters.
Research recommends a mailing sequence for sending the survey questionnaire followed by a reminder sent about 1 week later . Two follow-up reminder letters should then be sent to those not responding while the first should arrive about 1 week after sending the questionnaire and the second a week later . If the higher response rate is necessary, phone calls can be made to the non-respondents about 2 weeks after two reminders . Follow-up notices with personally requesting non-responders’ participation may increase response rates to some extent [36, 37]. This research adopted the following data collection sequence represented based on research advice [32, 36, 37, 38]:
Step 1: sending survey forms with an invitation letter.
Step 2: 2 weeks later, sending the first e-mail reminder to non-respondents.
Step 3: another 2 weeks later, sending a second e-mail reminder to non-respondents.
Step 4: followed 2 weeks later, making phone calls to remaining non-respondents.
In this research, a total of 2401 surveys were successfully sent to SMEs in Australia and China including Australian SMEs (1528) and Chinese SMEs (873). The usable response rate for the surveys was 7.54% (181 out of 2401) including Australian SMEs (69) and Chinese SMEs (112) . Consequently, a total of 15 items from item 1 to item 15 were finally identified (Table 1).
|Item 1||CEO IT/e-commerce/e-commerce marketing knowledge|
|Item 2||Senior staff IT/e-commerce knowledge|
|Item 3||Regular staff training in the appropriate or relevant IT skills|
|Item 4||Flexibility of e-commerce systems to change depending on the business process|
|Item 5||Ability to keep up with the rate of technology change (externally)|
|Item 6||The response time effectiveness/performance of an e-commerce website|
|Item 7||Trust in the interface design and information displayed on a website|
|Item 8||Support from top management/decision-maker|
|Item 9||Support from senior management|
|Item 10||Customer pressure/acceptance/ interest|
|Item 11||Cost associated with keeping up to date or upgrade of e-commerce system|
|Item 12||Decision-maker’s effective e-commerce marketing plan|
|Item 13||Effective e-commerce marketing strategy|
|Item 14||Adoption of different e-commerce marketing strategies based on different business requirements/ needs|
|Item 15||The consistency of graphics and backgrounds with business culture used on a website|
4. Data analysis
During the data analysis procedures, this research conducted an initial reliability analysis and validity analysis first. Principal component analysis with varimax rotation method was then adopted within the factor analysis.
4.1. Reliability analysis
Reliability is the extent to which a question yields the same responses over time, or a scale produces consistent results when repeated measurements are made [36, 40]. Any summated scale should be first analysed for reliability to ensure its appropriateness before proceeding to an assessment of its validity . In this research, reliability was assessed using internal consistency analysis . The earliest and simplest measure of the internal consistency of a set of data items is the split-half reliability of the scale [40, 41]. On assessing split-half reliability, the total set of items is divided into two equivalent halves—if the two scales correlate highly, they should produce similar scores [42, 43]. The total scores for the two halves are then correlated, and this is taken as the measure of the reliability of the survey [42, 43].
In practice, the approach for assessing internal consistency is the coefficient alpha (a.k.a., ‘the reliability coefficient’) or Cronbach’s alpha popularised in a 1951 article by Cronbach based on the work in the 1940s by Guttman and others and is the most common measure of internal consistency of items in a scale [32, 36, 42, 44, 45], which is the most commonly applied estimate of a survey’s reliability [40, 43]. It provides a summary measure of the inter-correlations that exist among a set of items [40, 42, 45, 46].
At the initial reliability analysis stage, Cronbach’s alpha should be considered as the critical characteristic. Cronbach’s alpha varies from 0 to 1 [40, 46]. The higher the correlations among the items is the greater the Cronbach’s alpha values which imply that high scores on one question are associated with high scores on other questions . If the value of Cronbach’s alpha is low and if the item pool is sufficiently large, this suggests that some items do not equally share in the common core and should be eliminated . Research also indicates that a value is considered to have very good reliability between 0.80 and 0.95, good reliability between 0.70 and 0.80, fair reliability between 0.60 and 0.70 and poor reliability below 0.60 .
In the test of reliability in this research, Cronbach’s alpha is 0.837. The results showed strong evidence of meeting the reliability standards of exploratory research and are considered to have very good reliability.
4.2. Validity analysis
The validity of a survey instrument may be defined as the extent to which it accurately measures what it is supposed to measure [36, 40]. There are three basic approaches to establishing validity namely content validity, criterion validity, and construct validity [36, 40, 43].
4.2.1. Content validity
Content validity, sometimes called face validity, measures the extent to which a survey’s content logically appears to reflect what was intended to be measured . It typically involves a systematic review of the survey’s contents to ensure that it includes everything it should and nothing that it should not . Although there does not yet exist a scientific measure of the content validity for a survey instrument , content validity is often assessed practically by approaches such as focus groups, and/or pilot test studies [41, 46, 47].
In this research, content validity was assessed subjectively but systematically to establish the appropriateness of the variables used—items not considered appropriate were rejected by two focus group studies (one for each in Australia and China) and 20 pilot tests (10 for each in Australia and China).
4.2.2. Criterion validity
Criterion validity addresses the ability of a measure to correlate with other standard measures of similar established criteria . Criterion validity may be classified as either concurrent or predictive depending on the time sequence in which the new measurement scale and the criterion measure are correlated [40, 43] as follows:
Concurrent criterion validity—if there is an existing instrument that one can compare with a newly devised questionnaire [40, 46], or if the new measure is taken at the same time as the criterion measure and is shown to be valid, then it has concurrent validity .
However, no method for assessing criterion validity is foolproof while none can conclusively show if a concept is truly measuring what it should . As the concern is more about the validity of the use of the survey instrument than its own inherent validity , most researchers appear to more commonly use construct validity as discussed below. In this research, criterion validity analysis was not conducted, as no similar established surveys were available with which to compare it, and the measure is not being used to predict a future event.
4.2.3. Construct validity
Convergent validity is the extent to which the survey correlates positively with other measures of the same construct . It assesses the extent to which different data collection methods produce similar results . Convergent validity is also another way of expressing internal consistency as highly reliable scales contain convergent validity .
Discriminant validity is the extent to which a survey result does not correlate with other constructs from which it is supposed to differ and represents a measure of its distinctiveness [40, 43]. When two item values are correlated above 0.75, discriminant validity between items may be questioned and the items rejected .
In this research, convergent validity was being assessed actually through reliability analysis. Technically, discriminant validity was being indirectly established through the following factor analysis. Nomological validity was not analysed as no similar established relationships appeared to exist in the literature.
4.3. Factor analysis
The basic rationale of factor analysis is that the variables are correlated because they share one or more common components, and if they did not correlate, there would be no need to perform factor analysis, which operates on the correlation matrix of the variables to be factored . For the ease of interpretation, principal component analysis with varimax rotation method was conducted within the factor analysis. In practice, principal component analysis can be conducted using SPSS software through factor analysis.
Table 2 shows that the criteria for evaluating a matrix of the factor loading for each variable onto each component are quite good that the 15 items are grouped into five components with the suppression of loadings not less than 0.4. Items with factor loadings less than 0.4 have not been displayed for clarity .
|Extraction Method: Principal Component Analysis.|
|Rotation Method: Varimax with Kaiser Normalization.|
|a. Rotation converged in 7 iterations.|
Any components with a variance (represented by the eigenvalue) less than 1.0 were rejected as they contribute less than other factors to the model . Table 3 shows that all eigenvalues are over 1.0 of component 1, component 2, component 3, component 4, and component 5. According to research, the components accepted should account for at least 60% of the cumulative variance [36, 40]. In this research, cumulative percentage of the variance for the four components accounts for 64.843% which satisfies the normally accepted measure (see Table 3).
|Component||Initial eigenvalues||Extraction sums of squared loadings||Rotation Sums of Squared Loadings|
|Total||% of Variance||Cumulative %||Total||% of Variance||Cumulative %||Total||% of Variance||Cumulative %|
5. Results and discussion
Based on the above analysis, the 15 CSFs were categorised into five components by the strength of relationship:
Component 1: for the three items grouped including F15, F14 and F13, there is a single focus on Marketing.
Component 2: for the three items grouped including F12, F11 and F10, the centre points are Management Support and Customer Acceptance.
Component 3: for the four items grouped including F9, F8, F7 and F6, the focuses are Website Effectiveness and Cost.
Component 4: for the two items grouped including F5 and F4, the common factor is Managing Change.
Component 5: for the three items grouped including F3, F2 and F1, the common focus is on Knowledge and Skills.
By superimposing the 15 CSFs onto the above EBS management model, SMEs Business Managers might effectively adopt or evaluate e-commerce satisfaction and success from a business perspective (see Figure 2).
6. Conclusions, limitation and further research
This research seeks to explore a new method to describe and evaluate e-commerce success from a business perspective. An EBS management model with 15 CSFs as a foundation was finally simply developed to assist SMEs business managers in effectively adopting e-commerce systems or evaluating e-commerce success, which was categorised into five components including Marketing, Management Support and Customer Acceptance, Website Effectiveness and Cost, Managing Change and Knowledge and Skills.
The major limitation is that the investigation just focuses on SMEs in Australia and China. This implies to conduct further research in other counties. Further research is also needed to determine the weighting of each CSF so that this EBS management model might be further updated and extended as a yardstick measurement method to assist SMEs in adopting e-commerce successfully.
This research is sponsored by the Fund for Shanxi ‘1331 Project’ Collaborative Innovation Center, China.