Organizational research methods
Organizational Research Methods: A Guide for Students and Researchers
Criteria for selecting a method
?Appropriate to your research objective;
?Able to elicit a form of data appropriate to testing your hypothesis/hypotheses or addressing your research question(s);
?Feasible given time, resource and organizational constraints and requirements; Adequately piloted; Ethically sound; Agreed and accepted by the organization;
?Used appropriately, in the context of its original formulation and development;
?One you feel comfortable with, being confident and well rehearsed in its use before you use it ‘for real’.
Data collection methods
?Interviews (including repertory grid and Q-sort approaches);
?Focus groups;
?Psychometrics;
?Observation;
?Surveys and questionnaires;
?Diary methods.
Interviews
Interviewing is an extremely flexible research tool (Breakwell, 1995). It can be used at any stage of the research process: during initial phases to identify areas for more detailed exploration and/or to generate hypotheses; as part of the piloting or validation of other instruments; as the main mechanism for data collection; and as a ‘sanity check’ by referring back to original members of a sample to ensure that interpretations made from the data are representative and accurate. Interviews can also be readily combined with other approaches in a multi-method design which may incorporate, for example, questionnaire measures or observation. Interviewing, as with all research methods, is also open to a number of biases and shortcomings, the most critical of which is the difficulty of achieving reliable and valid results. Quantification and objectification of interview-derived data are the most powerful ways to remedy this, and highlight the importance of the researcher maintaining an objective stance throughout the research process.
Types of interview
Structured interviews: prescribed set of questions in fixed order. Pros: Ensures rapid data coding and analysis, easy quantification of data and comparability of responses. Cons: Constrains interviewees and does not allow for exploration or probing.
Unstructured interviews: Questions and their order are not fixed. Open-ended, rather than forced-choice, questions. Cons: the depth of exploration may be the same as structured, since both depend on the knowledge and skill of the researcher.
Semi-structured interviews: elements of both quantifiable, fixed-choice responding and the facility to explore, and probe. Pros: easy to analyse, quantify, compare, while allowing interviewees to explain their responses. Cons: danger of losing control to the interviewee.
Ethnographic interviews: Ethnographic interviews amount to unstructured interviews but within the context of the target research area, and extending beyond the restrictions of an unstructured interview (which still imposes the researcher’s perceptions and pre-conceptions) by allowing interviewees (often termed ‘informants’ in this field of research) to develop their responses in their own way, using their own frame of reference.
Guidelines for the interview process
?Design for consistency
?Obtain as much background information as possible
?Prepare and pilot the interview in advance
?Ensure privacy and avoid interruptions
?Put the interviewee at ease
?Establish rapport
?Maintain control
?Avoid bias as far as possible
?Be objective
?Obtain the maximum response from each question
?Be sensitive
?Give yourself a chance
Analyzing interview data
?Keep focused
?Select an analytical approach
?Estimate inter-rater reliability
?Authenticate interpretations-by verifying with a sample of interviewees.
?Report on the analysis
Advantages and disadvantages of interviewing
Advantages:
?Rich data
?Flexibility
?Interviewer availability
?Ensures cooperation
?Rapport and confidence-building
Disadvantages
?Cost
?Time-consuming
?Accessibility
?Open to bias
?Poor reliability
Techniques for use in interview contexts
The repertory grid technique (RGT)involves three stages: the elicitation of ‘elements’ (i.e. the entities in the area of meaning to be investigated), the elicitation of ‘constructs’ (in relation to the elements), and the construction of a grid matrix of elements and constructs. Elements can be people, objects, events, or in fact anything with reference to which an individual constructs meaning. It is essential that whatever the class of element, this remains consistent within each particular grid application (e.g. all people, all events).
The Q-sort methodology was devised by Stephenson (1936). It can be employed to assess, in a highly structured and systematic way, people’s understandings of an issue (Millward, 1995), from their own unique standpoint, and has the potential to integrate qualitative and quantitative information.
Individuals are invited to arrange a set of items along a scale anchored by, say +5 (e.g. most happy with) through to –5 (e.g. least happy with) within a prescribed distribution (i.e. the normal distribution). The exact number of items distributed at each point of the scale will depend on how many items are to be sorted. Thus, in a 54-item set, the distribution along which cards are to be sorted would be: 2, 4, 6, 9, 12, 9, 6, 4, 2 (see Figure 6.2). The sorting criteria are always self-referent, and domain-specific, selected depending on the aim of the Q-sort (i.e. what evidence, and with respect to what, it is designed to elicit).
Focus group method
The focus group is described by Millward (1995) as a discussion-based interview that produces a particular type of qualitative data. It involves the simultaneous use of multiple respondents to generate data and it is the ‘focused’ (i.e. on an external stimulus) and relatively staged (i.e. by a
‘moderator’) nature of the focu s group method that separates it from other types of group interviewing strategy.
?Focus group logistics
?Clarifying aims
?Sampling – random sampling not necessary, but should be systematic.
?Recruitment strategies –The time and energy invested with ‘local’ p eople and making personal contact with potential participants at the outset can facilitate group rapport and
contribute substantially to this.
?Focal stimuli
?Group size and number – average of 9, range of 6 to 12.
?Location – neutral grounds
?Duration – 1-2 hours.
?Moderator style and skills – preferable for the moderator to be someone directly involved in the project who is sensitive to the research issues and the need for methodological rigour
even if their group management skills are not especially polished.
?Introducing the session – moderator explains his role, the purpose, who will see the results (anonymity), general rules.
?Data extraction – qualitative in the form of transcripts produced from audio-tape or video-tape.
?Field problems:
?Facilitator as ‘organizational therapist’ – groups treat the facilitator as someone to tell problems to. Can present a problem.
Psychometric instruments
Psychometric instruments are widely used throughout social science and applied organizational research. At both individual and organizational levels of analysis, they afford the researcher an objective, consistent and comparable method of measuring people or groups of people in various settings and to various ends. Psychometric instruments, according to Hammond’s (1995) categorization, can include self-report questionnaires, objective tests (of ability, preference or personality), and normative, criterion-referenced and idiographic tests.
Reliability and validity
‘Reliability’ refers to the internal and temporal consistency of an instrument, whereas validity may be defined as the degree to which an instrument actually represents what it purports to represent.
Criterion-related validity As mentioned above, this type of validity pertains to the relationship between scores on a psychometric tool and external criteria which the tool is intended to predict. Criterion-related validity is usually sub-divided into ‘concurrent’ and ‘predictive’ validity. Concurrent validity involves the criterion and the test behavior being measured simultaneously, e.g. a test of managerial competence being administered at the same time as actual current job performance data are obtained in order to determine how closely the two measures are related. Predictive validity involves the criterion and the test behavior being temporally separated. Examples Construct validity Construct validity is very closely related to theory development and testing, with an instrument being assessed while, simultaneously, its underlying theoretical concepts are being rigorously investigated.
?Clear specification
?Relationship between instrument dimensions or scales
?Relationships with different constructs
Advantages:
?Cost
?Comparability
?Power
Disadvantages
?Restrictive
?Reductionist
?Reliant on understanding of population
?Open to temporary bias
?Norm-based
?Existence of many poor psychometric tools
Observation
Participant and non-participant
Observation is a highly skilled activity which should not be considered lightly. There are two broad types of observational activity: participant and nonparticipant. With participant observation, the researcher immerses him or herself in a situation alongside target participants ‘in the field’. The aim here is to become an accepted member of the participant community. The observation process is unstructured (i.e. without preconceived ideas or codes or foci of investigation). The researcher observes in toto, writing up observations as soon as possible after they have been made. The observations may be of interest ‘in the raw’ or as the basis on which to formulate hypotheses. Participant observation is also often termed ethnography which means capturing the ‘native point of view’ and is mainly anthropological in thrust (see Schwartzman, 1993). In non-participant observation, the researcher stands back from the situation and observes at a distance (either in situ or using video material). The observation process may be guided by a set of analytical codes or checklists, or in the case of checklist development, may be completely unstructured in the first instance. In participant observation, the focus of observation is usually at the macro-level (e.g. group dynamics), whereas the non-participant kind is usual directed at the micro-level (e.g. interaction processes).
Technical issues
?Sampling
?Method of recording
?Location of observer
Problems/issues
Observation requires skill, time and experience at a level that is very demanding, particularly observation of the participant kind.
The issue arising from the unstructured observation situation pertains more to concerns about validity and reliability, insofar as there is scope for many alternative interpretations. Who is to say that your interpretation is appropriate?
Survey and questionnaire design
Questionnaire and survey measures are probably the most widely used research tools within the social sciences (Fife-Schaw, 1995). Their low cost, minimal resource requirements and potentially large sample-capturing abilities make them an attractive research method for academics and practitioners alike.
Questions relating to ethnic background should be treated with caution: questions such as ‘Why are they necessary to the research?’, ‘What will they be used to achieve?’, ‘What impact might they have on accuracy of response and on sample size obtained?’ should be considered by the researcher.
Behavioral information: Fife-Schaw (1995) discusses issues associated with behavioural self-report questions in some depth, highlighting the major problems of memory and social desirability as potential biases in these types of question.
Attitudinal information: There are a wide variety of techniques that can be used by researchers to tap into respondents’ attitudes. Among the most commonly used are 5or 7-point attitude scales, but others include open-ended questions, semantic differential scales, diagrammatic rating, or forced-choice questions. Five or 7-point Likert-type scales typically involve the respondent being presented with a statement to which s/he is asked to report how s/he feels about the accuracy or otherwise of the statement, ranging between the extremes of ‘Strongly agree’ through ‘Neither agree nor disagree’ to ‘Strongly disagree’.
Wording and other problems
?Unfamiliar words
?Ambiguous or imprecise words or concepts
?Complicated wording
?Double barreled questions
?Arithmetic concepts
?Double-negatives
?Leading questions
?Questions with hidden assumptions
?Context effects
?Sensitive questions
Problem of social desirability
Any questionnaire should be customized to the level and specific characteristics of the target population. It is extremely difficult to ask questions of one sample which will not in some way condescend to, or ask too much of, another. However, the researcher should try to achieve generalizability of questions as far as is possible in order that findings are as comparable as they can be.
Firstly, as noted earlier, people answer questions using similar response sets (some peppering the mid-point of a scale, others using only extreme points, and yet others using the full scope of a scale) and this will influence the statistical properties of information obtained using a survey measure. Reversing items or breaking long sections of questions with a different type of response scale can
ameliorate this situation. In addition, ‘method co-variance’ refers to the use of more than one measure within a survey battery: due to the similarity of methods used to collect data from each measure and the response sets of individuals, correlations between the measures are likely to be inflated; that is, their shared variance will be unrealistically high. It is only by using alternative methods in combination with questionnaires, e.g. interviews, observation, etc., that such effects may be offset. This effect is discussed in greater depth later in this section.
Aesthetic issues
?Respondent instructions and covering letter
?Questionnaire length
?Question order
?Spelling/grammar
?Font type and size
?Layout density
?Provision of sufficient space for responses
As noted by Podsakoff and Organ (1986), one of the principal problems that may arise from the use of self-report questionnaire measures within organizational research is that of common method variance (Campbell and Fiske, 1959). This refers to the conflation of response-response correlations when all data derive from the same source, i.e. from each questionnaire respondent. Podsakoff and Organ (1986) note a number of post hoc statistical procedures which may be used to address this problem following data collection, including Harman’s (1967) one-factor test, social desirability partialling (Edward s, 1970) and scale ‘trimming’ (Birnbaum et al., 1986). However, each of these procedures contains its own drawbacks and none has gained overwhelming popularity within the research community. However, there remains the possibility that the self-report researcher may introduce mediating measures at the design stage.
Diary methods
Burgess (1982) suggests that a diary provides a first-hand account of a situation to which a researcher may not have direct access. Any form of personal log of events, generically de scribed as a diary, then, may provide the researcher with an insight into issues and events which would not be accessible via other means, such as interview, questionnaire or observation methods. At their heart, diary methods draw on the subjective interpretation of events by individuals in particular situations to contextualize reported experience, and place this within a continuous temporal framework.
Method
?Select an appropriate medium
?Provide clear instructions
?Ensure usability of the diary format
?Encentivize the research
?Remain available
?Keep it short initially
Advantages:
?Familiarity
?Cost-effective information sampling
?Sequencing data
?Sensitive information
Disadvantages:
?Control of content (participants misremember information)
?Mortality (high dropout rate)
?Verification (can’t be verified)
?Reactance (the act of journaling may alter perceptions, attitudes and behaviors)