Methodological and Psychometric Characteristics of the Research Instrument: Retest

Abstract

This chapter describes testing of the key research instrument, by identifying the individual steps, comprised of adaptation and testing of the questionnaire, by using statistical analysis. The selected sample of the research by surveying the employees of the groups of companies operating in Eastern Europe is not only minimal, but also sufficient (error is not greater than 1%), allowing to make sound conclusions of the research. After adaptation of the questionnaire for the larger and multilingual sample, high psychometric characteristics, which confirmed the reliability of the instrument and its suitability for the research, have been found. This shows that the developed questionnaire is appropriate, and it can also be used to measure management culture and corporate social responsibility not only in this research.

1. Introduction

Relevance of the research and the level of problem exploration. Quite different opinions about the construction of new instruments, their tests and methodological, psychometric characteristics are found. According to the authors of this book, very high requirements are raised to the quantitative research instrument to make it suitable in case of various samples. During the exploratory research, 159 employees of one of the regional municipalities of the country (public sector) have been surveyed [1] and 1717 respondents, representing two industrial clusters (private sector) have been surveyed [2, 3] during the retest. It should be emphasized that our aim was to develop as universal instrument as possible so that it could be used by other scientists as well to carry out the studies of similar nature.

Problem of research: the problem of the research is raised by the question: what are methodological and psychometric characteristics of the questionnaire in case of a larger sample and how they changed after the retest compared to the results of the exploratory research?

Object of the research: methodological and psychometric characteristics of the research instrument.

Purpose of the research: to check the methodological and psychometric characteristics of the questionnaire with respect to this sample.

Objectives of the research: (1) to provide methodological quality characteristics of the management culture subscales; (2) to carry out factoring of management culture scales and subscales; (3) to provide methodological quality characteristics of the subscales of behaviour of a socially responsible organization; (4) to carry out factoring of the scales and subscales of behaviour of a socially responsible organization and a socially responsible employee.

Methods of the research: In order to achieve the aim, the quantitative research method was selected—a written survey which was carried out by using a proven, statistically reliable questionnaire ‘Management culture level determination aiming to implement corporate social responsibility’. The data were processed by SPSS programme (version 21). Explained dissemination, Cronbach’s alpha and Spearman-Brown coefficients, factorial weight (L), correlation of the unit as a whole (r/itt) have been calculated and factor analysis has been carried out as well.

2. The research sample

The research involved employees from 12 industrial organizations. Ten organizations form one group of companies (hereinafter the first (1) group of companies), the remaining two organizations also form a group of companies (hereinafter the second (2) group of companies). In total, 1915 employees worked in the two industrial groups in general during the researched period. There were 1030 employees in the first group and 885 employees in the second group. The total number of participants in the survey is 1717 employees (911 employees in the first group of companies, 806 employees in the second group of companies), representing 89.6% of all employees. The research sample was calculated from the total number of employees of all 12 organizations on the basis of Paniotto’s formula as in Eq. (1) [4]:

$n=\frac{1}{{\Delta }^2+\frac{1}{N}}$E1

where n represents the sample size; ∆ represents the sample error size (= 0.05); and N represents the general size of the whole.

$n=\frac{1}{{\Delta }^2+\frac{1}{2}}=\frac{1}{{0.05}^2+\frac{1}{1915}}=331$E2

When the probability is 95% and the error is 5%, the necessary sample size is 331 employees as shown in Eq. (2). However, when the probability is 99% and the error is 1%, calculating on the 12 organizations’ scale, the necessary sample size is 1607 employees as shown in Eq. (3). When preparing the research plan, it is essential to determine the minimum number of the researched. This is necessary in order to draw statistically valid conclusions that are in line with the characteristics of the general set. In our research, this estimated sample is considered not only to be minimal, but sufficient to find out reasonable research conclusions.

$n=\frac{1}{{\Delta }^2+\frac{1}{2}}=\frac{1}{{0.01}^2+\frac{1}{1915}}=1607$E3

3. The research organization

Two Lithuanian groups of companies operating in Eastern Europe were selected for the research whose main activity is production. The companies are mainly based in Lithuania, but the activities also cover other countries such as the Ukraine, Russia, Estonia, Latvia and Romania, where branches of groups of companies were established. The activities, size and other indicators of both groups of companies are more or less similar. It is important to mention that both groups of companies seek for the status of corporate social responsibility.

The highest level managers of the group of companies, who were involved in coordination of the questionnaire content and survey process, were interested in the research performance and the results. The survey was organized in Lithuanian and English. There were 1915 questionnaires distributed in the companies; 198 questionnaires were removed from the research, because the questionnaires were filled incorrectly and/or incompletely. A total of 1717 filled in questionnaires were recognized valid, which fully meets the sample size when the probability is 99%.

4. Questionnaire reliability

As the pilot research was carried out by using a Lithuanian version of the questionnaire, before the start of the research in multilingual sample, an adaptation of the questionnaire was conducted in English [5, 6, 7]. The adaptation process consisted of six conditional stages. The first stage involved questionnaire translation into English that was carried out by two professional translators whose native language is English (and who speak Lithuanian well). In the second stage, translation versions of both translators were evaluated and together with translators the questionnaire authors formed the primary English version of the questionnaire. In the third stage, the questionnaire was given to the translator whose native language is Lithuanian and who has good knowledge of English. In the fourth stage, the translator (native Lithuanian) conducted the translation of the initial English version of the questionnaire into the Lithuanian language (‘back-translation’). In the fifth stage, after the translation of the questionnaire was finished, the discussion of every statement was performed. There were chosen the most appropriate options, expressions, words that would be acceptable and understandable to both Lithuanians and foreigners and would not change the meaning of the statements formulated in the questionnaire. In the sixth stage, the primary survey—a questionnaire testing (10 people) was conducted. Testing is necessary for clarity of the questions, intelligibility and suitability to assess in the linguistic and cultural aspects. The goal of the primary survey is to identify words, statements, questions that cause doubt or uncertainty; to determine the cause and make suggestions how to reformulate obscure terms. It should be emphasized that the questionnaire name and structure of the questions, the number of the questions and answers were not changed, i.e., only the wordings of the statements and questions were corrected. Each of the 10 respondents was interviewed individually. The respondent completed the questionnaire and then together with researchers looked at each statement and question. The aim of this review was to determine whether the respondents, when carrying out the research in the future, will not have any doubts about the questions and answers. During the review, the researchers suggested that the respondents who participated in the primary survey should provide more understandable versions of the statements and questions. After adapting the questionnaire in English, the survey was conducted.

Having conducted the survey again, with new research results, with respect to bigger and multilingual sample questionnaire, methodological and psychometric characteristics reliability determination was carried out. Table 1 presents the methodological quality characteristics of four subscales making up the scale of management staff culture. Cronbach’s alpha coefficient values range from 0.74 to 0.86. The closer the Cronbach’s alpha value is to 1, the higher internal consistency of the questionnaire (the greater accuracy of the questionnaire measurement) is shown by the coefficient. As it can be seen, the explained dissemination percentage in this scale ranges from 39.16 to 53.77, which indicates that such percentage of the survey respondents agree with isolated factors.

Subscales	Number of statements in subscales	Explained dissemination, %	Cronbach’s alpha	Spearman-Brown	Factorial weight (L)			Correlation of the unit as a whole (r/itt)
					Mean	Min	Max	Mean	Min	Max
Management staff general culture level	7	53.77	0.86	0.83	0.73	0.63	0.77	0.53	0.34	0.77
Management science knowledge level	5	41.56	0.74	0.61	0.64	0.47	0.71	0.39	0.11	0.68
Managers’ personal and professional characteristics	5	51.01	0.76	0.66	0.71	0.64	0.77	0.50	0.25	0.76
The level of the ability to manage	9	39.16	0.80	0.77	0.62	0.47	0.72	0.37	0.13	0.70

Table 1.

Methodological quality characteristics of management staff culture subscales.

Source: compiled by the authors.

As the explained factor dissemination is bigger than the allowable lowest 10% limit, this means that this scale does not contain statements that reduce dissemination. The minimum factorial weight (L) may not be lower than 0.3. If it is less than 0.3, it indicates that an inappropriate statement in the subscale was found. The analysis of the factorial weight minimum values in management staff culture scale showed that the lowest weight, i.e. 0.47, was recorded in only one subscale. In management staff culture subscales, the average of the minimum unit correlation (r/itt) is from 0.37 to 0.53. So, it is not less than 0.2, which confirms that there are no inappropriate statements in the subscales.

Methodological quality characteristics of culture of managerial processes in an organization subscales are shown in Table 2. Cronbach’s alpha coefficient values on this scale are high, i.e. ranging from 0.72 to 0.82. The percentage expression of the explained dissemination in the analysed scale falls into the interval from 42.21 to 49.29, indicating a relatively high level of approval. On this scale, the minimum factorial weight is found in only one subscale, i.e. in the subscale of optimal regulation of managerial processes, its value is 0.36. However, even the lowest factorial weight exceeds the indicated minimum limit of 0.3. The correlation of the unit as a whole indicates that the questionnaire statements correlate with the isolated subscale as r/itt average is 0.40–0.47.

Subscales	Number of statements in subscale	Explained dissemination, %	Cronbach’s alpha	Spearman-Brown	Factorial weight (L)			Correlation of the unit as a whole (r/itt)
					Mean	Min	Max	Mean	Min	Max
Optimal managerial processes regulation	7	49.29	0.82	0.75	0.69	0.36	0.78	0.47	0.13	0.77
Rational organization of management work	5	48.60	0.73	0.70	0.69	0.55	0.75	0.47	0.22	0.75
Modern computerization level of managerial processes	5	49.64	0.72	0.68	0.70	0.40	0.81	0.46	0.09	0.76
Culture of visitors’ reception, conducting meetings, phone calls	7	42.21	0.77	0.76	0.64	0.48	0.73	0.40	0.17	0.71

Table 2.

Methodological quality characteristics of managerial processes organization culture subscales.

Source: compiled by the authors.

The methodological quality characteristics of management working conditions culture subscales presented in Table 3 show that the Cronbach’s alpha coefficient values range from 0.66 to 0.84. The explained dissemination lowest percentage of 37.61 is above the established 10% limit. Here, the minimum factorial weight is 0.50, so it may be claimed that the statements of the subscales on this scale are quite closely related to each other. The correlation of the unit as a whole on this scale shows that the lowest mean is 0.35, the highest—0.42, which confirms that the statements in the questionnaire correlate with isolated subscales.

Subscales	Number of statements in subscale	Explained dissemination, %	Cronbach’s alpha	Spearman-Brown	Factorial weight (L)			Correlation of the unit as a whole (r/itt)
					Mean	Min	Max	Mean	Min	Max
Working environment level (interior, lighting, temperature, cleanness, etc.)	9	43.51	0.84	0.80	0.66	0.57	0.76	0.42	0.24	0.73
Level of organizing working places	5	53.29	0.78	0.72	0.73	0.68	0.78	0.52	0.32	0.77
Work and rest regime, relaxation options	6	46.54	0.77	0.76	0.68	0.51	0.76	0.45	0.21	0.76
Work security, socio-psychological microclimate	6	37.61	0.66	0.62	0.61	0.50	0.71	0.35	0.13	0.67

Table 3.

Methodological quality characteristics of management working conditions culture subscales.

Source: compiled by the authors.

The methodological quality characteristics of the documentation system culture subscales are shown in Table 4. Psychometric characteristics of this subscale show that the strongest approval of the respondents was seen with respect to culture of official registration of documentation, that is, the percentage of the explained dissemination (46.58) as well as Cronbach’s alpha (0.77) coefficient values are quite high. Although in the subscale of rational use of modern information technologies Cronbach’s alpha coefficient value is higher (0.80), the percentage of explained dissemination in this case, although not significantly, is lower (41.75), comparing these two subscales with each other. The minimum factorial weight on this scale ranges from 0.47 to 0.66, while the average of unit as a whole correlation is from 0.37 to 0.45. Thus, it can be said that discussed indicators of this subscale meet the necessary conditions for the questionnaire reliability.

Subscales	Number of statements in subscale	Explained dissemination, %	Cronbach’s alpha	Spearman-Brown	Factorial weight (L)			Correlation of the unit as a whole (r/itt)
					Mean	Min	Max	Mean	Min	Max
Culture of official registration of documentation	6	46.58	0.77	0.66	0.68	0.62	0.72	0.45	0.23	0.71
Optimal document search and access system	5	48.33	0.73	0.72	0.69	0.66	0.74	0.47	0.28	0.73
Rational use of modern information technologies	8	41.75	0.80	0.74	0.64	0.54	0.69	0.40	0.19	0.69
Rational storage system of archival documents	6	39.59	0.69	0.63	0.62	0.47	0.74	0.37	0.12	0.70

Table 4.

Methodological quality characteristics of documentation system culture subscales.

Source: compiled by the authors.

Traditionally, when methodological quality characteristics of questionnaire subscales have already been stated, their secondary factorization must be carried out. Primary and secondary factorizations are required when there are questionnaires of very large-scale. Subscales that make up the scale must be similar in content and logic. During primary factorization, the whole complex of criteria is deducted, while during the secondary factorization, these criteria are combined to scales. Table 5 gives the general factorization results of management culture scales and subscales.

Questionnaire scales and subscales	Principal components	Alpha factoring
Management staff culture
The level of the ability to manage	0.87	0.84
Managers’ personal and professional characteristics	0.87	0.82
Management staff general culture level	0.86	0.82
Management science knowledge level	0.74	0.62
Explained dissemination:	70.38%	61.27%
Managerial processes organization culture
Optimal managerial processes regulation	0.90	0.89
Rational organization of management work	0.87	0.82
Culture of visitor reception, conducting meetings and phone calls	0.87	0.82
Modern computerization level of managerial processes	0.84	0.76
Explained dissemination:	75.74%	67.85%
Management working conditions culture
Working environment level (interior, lighting, temperature, cleanness, etc.)	0.86	0.84
Level of organizing working places	0.80	0.67
Work security, socio-psychological microclimate	0.77	0.72
Work and rest regime, relaxation options	0.69	0.56
Explained dissemination:	61.28%	49.40%
Documentation system culture
Rational use of modern information technologies	0.88	0.85
Optimal document search and access system	0.87	0.83
Culture of official registration of documentation	0.84	0.77
Rational storage system of archival documents	0.80	0.71
Explained dissemination:	71.70%	62.54%

Table 5.

Factorization results of management culture scales and subscales.

Source: compiled by the authors.

Management staff culture secondary factorization results indicate that factorial weights in the subscales of this scale range from 0.74 to 0.87 (by principal components method) and from 0.62 to 0.84 (by alpha factoring method). The subscales including the scale of culture of organization of managerial processes reflect such factorization results: minimum weight—0.84, maximum—0.90 (by principal components method) and the minimum—0.76, and maximum—0.89 (by alpha factoring method). On the scale of management working conditions culture, the indicators of these subscales are established: 0.69–0.86 and by the second method 0.56–0.84. Documentation system culture secondary factorization shows that factorial weights range from 0.80 to 0.88 (by principal components method) and from 0.71 to 0.85 (by alpha factoring method).

The results of secondary factorization indicate that factorial weights are high, therefore, the scales are reliable, the made up questionnaire is suitable for the measurement of the set whole of signs. In that case, the explained dissemination, revealing how strongly the respondents agree with this criterion, is also high, i.e. in the general context of management culture scales it ranges from 61.28 to 75.74% (by principal components method) and from 49.40 to 67.85% (by alpha factoring method). Factorial weights range from 0.48 to 0.84 (by principal components method) and from 0.40 to 0.83 (by alpha factoring method). With the help of secondary factorization method, it was found that factorial weights are high, so the scales are reliable; the made up questionnaire is suitable for the measurement of the set whole of signs (Table 6).

Questionnaire scales and subscales	Primary factorization	Secondary factorization
Behaviour of a socially responsible organization
Market responsibility (consumer information, health and safety)	0.84	0.81
Environment protection responsibility	0.82	0.78
Responsibility in relations with society	0.82	0.77
Market responsibility (services and their quality)	0.80	0.73
Responsibility in relations with employees	0.75	0.66
Explained dissemination:	65.18%	56.71%
Behaviour of a socially responsible employee
Intentions to leave work	0.83	0.83
Uncertainty and lack of information at work	0.81	0.60
General physical and psychological condition of the employee	0.81	0.74
Social responsibility criticism: staff attitude	0.73	0.84
Corruption, nepotism, favouritism	0.78	0.72
The employee‘s opinion about the organization	0.48	0.40
Explained dissemination:	53.63%	47.82%

Table 6.

Factorization of behaviour of socially responsible organization and socially responsible employee.

Source: compiled by the authors.

Below verification results of four subscales (market responsibility subscale is divided into two parts), forming of the scale of a socially responsible organization behaviour is presented. Subscales range from 5 to 7 statements (total number of statements on a scale is 31). The general percentage of explained dissemination on the scale of socially responsible organization behaviour ranges from 43.36 to 51.20. Meanwhile, the Cronbach’s alpha coefficient ranges from 0.74 to 0.81. Market responsibility (with the respect to services and their quality) subscale‘s high level of reliability is indicated as a percentage of explained dissemination expression (51.20%) as well as Cronbach’s alpha coefficient value—0.81 (respectively high and high sensitivity, i.e. Spearman-Brown’s coefficient indicator—0.78). The lowest Cronbach’s alpha coefficient value (0.74) and the lowest percentage of explained dissemination (43.36) on the scale of socially responsible organization behaviour are recorded in the statements of the subscale of responsibility in relations with society. Regardless of the fact that these figures are lower if compared with other presented indicators, they are considered to be quite high in statistics. The explained factor dissemination, as has been already said, is a factor affecting the result which cannot be less than 10%. In the presence of 1717 respondents participating in the survey (100%), the highest explained factor dissemination is 51.20%, and this means that 51.20% of respondents approve the isolated factor (Table 7).

Subscales	Number of statements in subscale	Explained dissemination, %	Cronbach’s alpha	Spearman-Brown	Factorial weight (L)			Correlation of the unit as a whole (r/itt)
					Mean	Min	Max	Mean	Min	Max
Market responsibility (services and their quality)	6	51.20	0.81	0.78	0.71	0.65	0.79	0.50	0.30	0.78
Market responsibility (consumer information, health and safety)	5	50.26	0.75	0.67	0.71	0.66	0.76	0.49	0.29	0.76
Environment protection responsibility	7	44.40	0.79	0.72	0.66	0.59	0.72	0.43	0.19	0.71
Responsibility in relations with employees	7	44.57	0.79	0.74	0.66	0.55	0.73	0.43	0.23	0.73
Responsibility in relations with society	6	43.36	0.74	0.64	0.66	0.55	0.75	0.41	0.18	0.73

Table 7.

Methodological quality characteristics of behaviour of socially responsible organization subscales.

Source: compiled by the authors.

When analysing the methodological quality characteristics on employee’s social behaviour scale, it is necessary to emphasize that this scale has six subscales, including 41 statements in total. The number of statements in the subscales is spread fairly unevenly, but the results are not obviously affected by this, except the situation in the subscale ‘The employee‘s opinion about the organization’. Cronbach’s alpha coefficient is relatively low in the subscale ‘The employee‘s opinion about the organization’, i.e. 0.62, and it is impossible to calculate Spearman-Brown’s coefficient when the number of statements is less than 5 (Table 8).

Subscales	Number of statements in subscale	Explained dissemination, %	Cronbach’s alpha	Spearman-Brown	Factorial weight (L)			Correlation of the unit as a whole (r/itt)
					Mean	Min	Max	Mean	Min	Max
Intentions to leave work	6	59.59	0.86	0.84	0.77	0.70	0.80	0.59	0.42	0.80
Uncertainty and lack of information at work	6	49.26	0.79	0.72	0.70	0.64	0.75	0.48	0.30	0.74
General physical and psychological condition of the employee	5	58.03	0.82	0.77	0.76	0.68	0.80	0.57	0.32	0.79
The employee‘s opinion about the organization	4	43.55	0.62	–	0.65	0.38	0.86	0.52	0.22	0.77
Corruption, nepotism, favouritism	10	36.61	0.80	0.74	0.59	0.36	0.72	0.34	0.05	0.70
Social responsibility criticism: staff attitude	10	43.27	0.85	0.79	0.66	0.57	0.72	0.42	0.23	0.71

Table 8.

Methodological quality characteristics of behaviour of socially responsible employee subscales.

Source: compiled by the authors.

Comparing methodological quality characteristics of behaviour of socially responsible organization and behaviour of socially responsible employee subscales, it is seen that coefficient values of behaviour of socially responsible organization scale are slightly higher, but the difference is quite insignificant.

The percentage of the explained dissemination of the factor in both scales is above the lowest limit for at least three times, so it is clear that the respondents’ approval of isolated factors is high. The resulting high Cronbach’s alpha values suggest that the statements of the subscales included in the questionnaire in the scales of social responsibility are closely interlinked, because if they are lower than 0.3, it indicates that an inappropriate statement was found in the subscale. Comparing the minimum factorial weight values on both scales, it is seen that the lowest weight, i.e. 0.36, was recorded in only one subscale. The unit as a whole correlation r/itt shows how the questionnaire statements correlate with an isolated subscale. In behaviour of socially responsible organization subscales, the unit correlation average ranges from 0.41 to 0.50, and in behaviour of socially responsible employee subscales, it ranges from 0.34 to 0.59. This indicates that the statements in the questionnaire correlate with isolated subscales.