Challenges
Beyond the structural limitations and criticisms of this methodology listed previously, there are several challenges or pitfalls for researchers using it. This methodology is powerful, and well suited to new understandings, for example Casey and Evans’ (2011) study incorporating “links to chaos and complexity theories and to fractal patterns as it reports” (p. 1). Below is a summary of some of these challenges which researchers should be mindful of.
Not to Look for Preconceived Categories
As mentioned in the discussion of expert languages, researchers will usually have a minimum of a familiarity with the topic, field, or group being studied, but more often will have more than that. Researchers may be members of the group themselves, whether present or former, or they may be closely connected by family, profession, etc.; beyond this, they may also have personal and possibly even emotional connections to not only the research, but also their own situation in the research, whether their funding, timeline or even pride. Given these undercurrents, it is difficult not to sway, whether consciously or unconsciously, toward biased interpretations and analysis. Again, allowing for explicit, subject-generated definitions, cross-analysis to verify internal reliability, and conscious, reflexive analysis by the researcher are all proactive corrective steps (Tonkiss, 2012, p. 377).
Reconciling Conflicting Ideas and Data
A danger when trying to categorize or cluster data is how, or more likely if, data is to be grouped. A natural inclination may be to push for ‘neat’ data showing clear delineation and trends, but, especially when looked at qualitatively, this is not a requirement. “Human perception obeys the rule of biological equilibrium; in other words, contradictions are not eliminated, but are maintained in a state of tension and simultaneously counterbalanced.” (Psychology, 1974, p. 2). This tension of competing analyses may be uncomfortable for the goal-driven researcher, but remembering that this methodology values more the understanding of the perspectives and processes that lead to the phenomenon being studied allows for these competing ideas to remain and be seen as benefits, increasing possibilities of both causes and solutions. Moreover, these conflicts may themselves show an overarching 'pattern of variation', or consistency in conflicts, that may be as valuable as the data themselves (Tonkiss, 2012, p. 377).
Considering Sufficient Omissions and Silences
Another difficult area for the researcher focused on ‘getting the data’ is to change their perspective to focus on what they didn’t, and possibly couldn’t, get. This is challenging, as most researchers will have investigated as deeply and considerately as possible, but what is required is almost to ‘know what you don’t know’. Beyond omissions in methodology, simple omissions and silences in reported text can also be important sources of data. What is unsaid, whether by accident or purposely, can have meaning both in terms of what is unsaid, as well as that and why it was unsaid. Of course, attributing meaning to this involves construction, which must be tempered with the logic and likelihood of the analysis. Follow-up interviews, whether unstructured or focused on the omissions, can also lend backing to assumptions of meaning and interpretation (Tonkiss, 2012, p. 379).
Internal Validity
As above, logical, likely and complete analytical claims must be balanced with the co-construction of meaning between subject and researcher. This is a difficult balance to achieve, as any extrapolation or logical conjecture has a very sensitive dependence on initial input, such as the interpretations of meaning discussed above. Choices of textual reference, even before these interpretations, can from the very start slant the pretext and perspective of the analysis, possibly to the point of guiding or misleading, as mentioned in the discussion above on looking for preconceived categories. Beyond the help mentioned for that issue, extra-textual support for terminology, such as the agreed definitions of expert language or jargon, or conversely a push for the usage of ‘plain language’ may aid consistency, and therefore internal validity. Overlap or cross-reference of several examples from within the text may also serve to help this goal (Tonkiss, 2012, p. 380).
Reflexivity Creating Another Variable
Though the solutions to many of the above challenges involve rigorous reflection and reflexivity on the part of the researcher, the researcher’s questioning of their own assumptions may confound assumptions of subjects. If both the researcher and the subject are questioning, evolving, and challenging their understandings of phenomena, terminologies and their meanings, a sort of relativity can exist where the research can be ungrounded, or worse, lost with no direction. The act of reflection, though labyrinthine, at least makes explicit assumptions, delimitations and perspectives. Again, the goal of this methodology is to understand these, not necessarily the solutions that lay beyond (Tonkiss, 2012, p. 380).
Not to Look for Preconceived Categories
As mentioned in the discussion of expert languages, researchers will usually have a minimum of a familiarity with the topic, field, or group being studied, but more often will have more than that. Researchers may be members of the group themselves, whether present or former, or they may be closely connected by family, profession, etc.; beyond this, they may also have personal and possibly even emotional connections to not only the research, but also their own situation in the research, whether their funding, timeline or even pride. Given these undercurrents, it is difficult not to sway, whether consciously or unconsciously, toward biased interpretations and analysis. Again, allowing for explicit, subject-generated definitions, cross-analysis to verify internal reliability, and conscious, reflexive analysis by the researcher are all proactive corrective steps (Tonkiss, 2012, p. 377).
Reconciling Conflicting Ideas and Data
A danger when trying to categorize or cluster data is how, or more likely if, data is to be grouped. A natural inclination may be to push for ‘neat’ data showing clear delineation and trends, but, especially when looked at qualitatively, this is not a requirement. “Human perception obeys the rule of biological equilibrium; in other words, contradictions are not eliminated, but are maintained in a state of tension and simultaneously counterbalanced.” (Psychology, 1974, p. 2). This tension of competing analyses may be uncomfortable for the goal-driven researcher, but remembering that this methodology values more the understanding of the perspectives and processes that lead to the phenomenon being studied allows for these competing ideas to remain and be seen as benefits, increasing possibilities of both causes and solutions. Moreover, these conflicts may themselves show an overarching 'pattern of variation', or consistency in conflicts, that may be as valuable as the data themselves (Tonkiss, 2012, p. 377).
Considering Sufficient Omissions and Silences
Another difficult area for the researcher focused on ‘getting the data’ is to change their perspective to focus on what they didn’t, and possibly couldn’t, get. This is challenging, as most researchers will have investigated as deeply and considerately as possible, but what is required is almost to ‘know what you don’t know’. Beyond omissions in methodology, simple omissions and silences in reported text can also be important sources of data. What is unsaid, whether by accident or purposely, can have meaning both in terms of what is unsaid, as well as that and why it was unsaid. Of course, attributing meaning to this involves construction, which must be tempered with the logic and likelihood of the analysis. Follow-up interviews, whether unstructured or focused on the omissions, can also lend backing to assumptions of meaning and interpretation (Tonkiss, 2012, p. 379).
Internal Validity
As above, logical, likely and complete analytical claims must be balanced with the co-construction of meaning between subject and researcher. This is a difficult balance to achieve, as any extrapolation or logical conjecture has a very sensitive dependence on initial input, such as the interpretations of meaning discussed above. Choices of textual reference, even before these interpretations, can from the very start slant the pretext and perspective of the analysis, possibly to the point of guiding or misleading, as mentioned in the discussion above on looking for preconceived categories. Beyond the help mentioned for that issue, extra-textual support for terminology, such as the agreed definitions of expert language or jargon, or conversely a push for the usage of ‘plain language’ may aid consistency, and therefore internal validity. Overlap or cross-reference of several examples from within the text may also serve to help this goal (Tonkiss, 2012, p. 380).
Reflexivity Creating Another Variable
Though the solutions to many of the above challenges involve rigorous reflection and reflexivity on the part of the researcher, the researcher’s questioning of their own assumptions may confound assumptions of subjects. If both the researcher and the subject are questioning, evolving, and challenging their understandings of phenomena, terminologies and their meanings, a sort of relativity can exist where the research can be ungrounded, or worse, lost with no direction. The act of reflection, though labyrinthine, at least makes explicit assumptions, delimitations and perspectives. Again, the goal of this methodology is to understand these, not necessarily the solutions that lay beyond (Tonkiss, 2012, p. 380).