Friese S. Qualitative Data Analysis with ATLAS.ti 3ed 2019

266 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI some projects, especially larger ones, are situated in research contexts where the standards of quantitative research prevail – and in those contexts, analyzing inter-coder agreement also has a political component. She mentions another argument worth pondering: ‘But being reliable (to use the adjective) beats being unreliable. If a category is used in different ways, you will be unable to rely on it to bring you all the relevant data. Hence, you may wish to ensure that you yourself are reliably interpreting a code the same way across time, or that you can rely on your colleagues to use it in the same way’ (2009: 108). Another goal of analyzing inter-coder agreement could be to gain a better understanding about your codes and how you have applied them, when discussing disagreements with colleagues. It is not a ‘must’ to parse inter-coder agreement just because ATLAS.ti offers this functionality. ATLAS.ti has not suddenly become a tool that tries to quantify everything and forces you to code in a certain way. It only wants to offer a set of tools that serve different methodological approaches and epistemologies. Which tools you use is ultimately your decision. If you decide that the inter-coder agreement analysis fits your approach and study, you must follow some rules, otherwise the numbers you receive are hard to interpret or meaningless. If you continue reading to determine if inter-coder agreement analysis is for you, please consider this. In the following, when I write about inter-coder agreement, I assume the epistemological attitude of a researcher for whom this kind of analysis is appropriate and desirable. Why reliability matters According to Krippendorff (2018), the purpose of data collection and analysis is to give researchers answers to research questions that underlie a research project. Thus, the data are the trusted foundation from which to derive and discuss results and draw conclusions. Therefore, it is important that the researchers ensure that the data are generated without bias and that they mean the same thing for anyone who uses them. Reliability establishes this trust empirically by the agreement achieved among independently working coders. There are two ways to operationalize reliability: one is routed in measurement theory, which is less relevant for the type of data that ATLAS.ti users have; the second is an interpretivist conception of reliability. When collecting any type of interview data or observations, the phenomenon of interest usually disappears right after it has been recorded or observed. Therefore, the analyst’s ability to examine the phenomenon relies heavily on a consensual reading and use of the data that represent it. Researchers need to presume that their data can be trusted to mean the same to all their users. This means ‘that the reading of textual data as well as of the research results is replicable elsewhere, that researchers demonstrably agree on what they are talking about. Here, then, reliability is the degree in which members of a designated community agree on the readings, interpretations, responses to, or uses of given texts or data. […] Researchers need to demonstrate the trustworthiness of their data by measuring their reliability’ (Krippendorff, 2018: 277). Thus, reliability is the ability to rely on something for achieving something else. In this case, it is a reason to trust that data resulting from the coding of observed phenomena in fact reflect the phenomena, textual qualities or meanings of analytical interests.

TEAMWORK 267 One source of making data unreliable is intra-coder inconsistencies – that is, you as the coder apply the same codes differently over time or to different documents. Intercoder disagreements due to unequal backgrounds, prejudices and different literary abilities to interpret written coding instructions are another source. Both can be detected when replicating the coding process with different coders who apply identical coding instructions to the same set of textual phenomena. The motivation for measuring the degree to which a diversity of coders agree is: 1 that any disagreement observed among coders reduces the possibility that an analysis of the data yields valid conclusions, 2 the analysts of unreliable data can no longer be sure of what they are analyzing, 3 published results may not be what they claim to have studied, 4 stakeholders in the research results cannot be sure that they are talking of the same phenomena the data are claimed to be about and 5 critics who cannot replicate the findings have good reasons to reject them as empirically unsupported. Given this understanding, ensuring data reliability is a first step. Only after it has been determined that the reliability is sufficiently high does it make sense for the researchers to proceed with the analysis of the data. If there are significant doubts about what the data mean for different people, it will be difficult to justify further analysis and the results of that analysis. Why call it inter-coder agreement rather than inter-coder reliability? Reliability cannot be measured directly. What can be measured is the extent to which there is agreement or disagreement in the coding of two or more coders. Based on the agreement measures, reliability of the coding can be inferred. Therefore, the proper term is inter-coder agreement rather than inter-coder reliability. Agreement is what we measure; reliability is what we wish to infer from it. Reliability and validity Whereas reliability offers the certainty that research findings can be duplicated and that no or only limited external ‘noise’ has contaminated the data or the results, validity assures that the assertions made by the research reflect the reality it claims to represent. Validity concerns truths. Reliability relates to validity in the following ways: the more unreliable the data, the less likely it is that researchers can draw valid conclusions from the data. In terms of coding

268 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI data, this means that researchers need to identify valid accounts in the data to a degree better than by chance. If the agreement of two or more coders is not better than the agreement by chance, then reliability is low, and you cannot infer that a common understanding of the data exists. Thus, unreliability limits the chance of validity. On the other hand, reliability does not necessarily guarantee validity. If two coders share the same world view and have the same prejudices, they may well agree on what they see but could objectively be wrong. Or if two researchers have a unique perspective based on their academic discipline but their reading is not shared by many people outside their own scholarly community, the reliability might be high, but the outcome of the research has little chance of being substantiated by evidence of the reality that is implied. As Krippendorff (2004: 213) states: ‘Even perfectly dependable mechanical instruments, such as computers, can be wrong – reliably.’ A third aspect is the dilemma between high reliability and validity. Interesting interpretations might not be reproducible, or interesting data may not occur often enough to establish reliability. Highly reliable data might be boring and oversimplified to establish a high reliability in the first place. Requirements for coding As explained above, sources for unreliable data are intra-coder inconsistencies and intercoder disagreements. To detect these, we need to replicate the coding process. Replicability can be assured when several independently working coders agree on the use of the written coding instruction, by highlighting the same textual segments to which the coding instructions apply, identifying the same semantic domains that would describe them, and code them using the same codes for each semantic domain. Development of semantic domains A semantic domain is defined as a space of distinct concepts that share common meanings. Examples of semantic domains are emotional states, a set of strategies mentioned to deal with something, motivations for achieving something, consequences of an action, a set of different values, conditions or contexts. Each semantic domain embraces mutually exclusive concepts indicated by a code. A semantic domain can be viewed as equivalent to a category with subcategories, as described in Chapter 5. In developing the inter-coder agreement function, the team at ATLAS.ti decided to use a special name to make it clear that this type of coding is related to an inter-coder agreement analysis. Although I generally recommend such a structure for the reasons given in Chapter 5, not all users want to code in that way. And ATLAS.ti does not want to demand a specific way of coding. Therefore, it is also possible that every single code represents a semantic domain. That means it is not a must that a semantic domain consists of several codes.

TEAMWORK 269 At the development stage of a coding instruction a semantic domain is open-ended. For example, one may start with the semantic domain ‘benefits of friendship’ by distinguishing ‘caring for each other’ and ‘supporting each other’ but soon discover that there are also other aspects like ‘improving health and longevity’ and ‘learning from each other’. So, a code for these aspects needs to be added to the semantic domain ‘benefits of friendship’. Once a coding instruction is written and used in testing its reliability, one can no longer expand a semantic domain. All semantic domains are context-dependent. The gender of nouns has little to do with the gender of individuals. The color of a ballpoint pen has little to do with the color of an ethnic group, the state of a drunk or a political party. Semantic domains may use abstract names, but without acknowledgements of their contexts coding becomes unreliable. Semantic domains are logically and conceptually independent of each other, hence freely combinable as appropriate. A person may have a gender, an age, an education, a hobby, a marital status, and be a mother, a taxi driver, a socialist, etc. One must be careful not to confuse freely combinable semantic domains with the mutual exclusiveness of their codes. A single segment of text can be characterized in terms of several contextually linked semantic domains. For example, many political arguments concern a ‘proposal’ that has ‘proponents’ and ‘adversaries’, ‘benefits’ and ‘harms’. The words in quotation marks spell out the names of semantic domains. An example of how to represent semantic domains in ATLAS.ti is shown in Figure 9.28. Figure 9.28 Examples of semantic domains in ATLAS.ti There are two requirements for developing and working with semantic domains: exhaustiveness and mutual exclusiveness. Exhaustiveness means that the codes of the code system cover the variability in the data and that no aspect that is relevant for the research question is left out. On the domain

270 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI level, this means that all main topics are covered. On the sub-code level, this means that the codes of a semantic domain cover all aspects of the domain and the data can be sorted in the available codes without forcing them. An easy way out is to include a catch-all ‘miscellaneous’ code for each domain into which coders can add all data that they think does not fit anywhere else. However, keep in mind that such catch-all codes will contribute little to answering the research questions. Mutual exclusiveness affects two areas. One is the meaning of the sub codes: each of the sub codes within each domain needs to be different, and this needs to be clearly specified in the code definitions and coding rules. There should be no doubt when to apply sub code 1, 2 or 3. The second is the application of the sub codes: you can only apply one of the sub codes of a semantic domain to a quotation or to overlapping quotations. Using the same code color for all codes of a semantic domain will help you to detect possible errors. If you find that you have coded a quotation with two codes from the same semantic domain, you can fix it by splitting the quotation. This means you change the length of the original quotation and create one new quotation, so you can apply the two codes to two distinct quotations. Figure 9.29 How to correct a violation of mutual exclusiveness If codes within a semantic domain are not applied in a mutually exclusive manner, the inter-coder agreement coefficient is inflated. In the result table, ATLAS.ti will mark such coefficients in red. Multi-valued coding As shown in Figure 9.29 and 9.30, you can apply multiple codes of different semantic domains to the same quotation. This is referred to as multi-valued coding. For instance, a respondent expresses an opinion about a recent event. The semantic domain in this example is the RECENT EVENT and the various ‘opinions’ about the recent event are the codes of this domain. Within the same section or in an overlapping section, the respondent also talks about the people involved in the event and the consequences it had for them. The PEOPLE INVOLVED and the CONSEQUENCES are two further semantic domains. If your coding should cover all these different aspects that are mentioned in the sentence or paragraph, you need to apply codes from all three domains.

TEAMWORK 271 Figure 9.30 Multi-valued coding Coding with codes from multiple semantic domains will allow you to see how the various semantic domains are related – for example, what kind of people are involved in what kinds of events and how they experience the event. For this you can use the code co-occurrence tools at a later stage in the analysis (see Chapter 6). Common mistakes Commonly, it is assumed that consensus is better than individual judgment. Rather than working independently, coders are encouraged to discuss what they code and to reach their decision by compromise. However, data generated in this way does not ensure reproducibility, nor does it reveal the extent of it. When coders are able to communicate and agree on how to code a document, they are no longer independent. This means any observed agreement cannot be interpreted as indicative of reliability. Further, the results might also reflect the social structure of the group, the most prestigious member of the group dominating the outcome. Reproducibility, however, requires at least two independent coders. Another common procedure is to allow coders to consult each other if problems arise. For example, they do not understand a coding instruction, or they have problems applying some of the codes to the data given to them. This also compromises reliability. Ideally, the coding instructions should be clear and easy to understand. If this is not the case, the coders involved in the discussion create their own interpretation of what the codes mean. This is difficult to communicate to others and, therefore, jeopardizes reproducibility. In addition, the process loses stability as the data coded in the early phases were coded based on a different understanding of the codes. If coders do not work independently and discuss problems during the process of coding, the reliability coefficient might be higher in the end,

272 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI but this is partly illusory. If the data were to be given to different coders without the same insights, reliability would likely be lower again. As it is common that code systems evolve, and code definitions and coding rules need to be refined, you can ask coders to write down all the issues they see with the current instructions and definitions. Based on their notes, the coding system can be refined and tested again but with different coders. Another common mistake is to assume that colleagues with a long history of involvement in the subject of the research or close friends and associates serve as coders. Those coders are likely to agree, but not because they carefully follow the coding instructions, but because they know each other and the purpose of the research. This also results in higher measures of reliability but also does not serve reproducibility. Measuring inter-coder agreement Suppose two coders are given a text of a certain length and do what the coding instructions tell them to do. The simplest agreement coefficient |cuα assesses the degree of agreement among these coders to decide what is relevant to the current research project defined as being coded with any of the codes of the semantic domains. They may give you the following record of their coding – in this case, they have only selected segments of text they consider to be relevant: Figure 9.31 Coders code on a textual continuum (can be text, audio or video) Evidently, the first pair of segments agree in length and location on the continuum. The second pair agree in length but not in location. They have an intersection in common. In the third case, coder A finds a segment relevant, which is not recognized by coder B. The largest disagreement is observed in the last pair of segments: coder A takes a narrower view than coder B. In terms of your ATLAS.ti coding, you need to think of your document as a continuum. Each character of your text is a unit of analysis for inter-coder agreement, starting at character 1 and ending, for instance, at character 17,500. For audio and video documents, the unit of analysis is a second. Images can currently not be used in an inter-coder agreement analysis. The quotation itself does not go into the analysis, only the characters or seconds that have been coded. If you add multiple documents to the analysis, the continuum is extended like pearls strung on a chain. Thus, every character or second where coders agree goes into the calculation of the inter-coder agreement coefficient.

TEAMWORK 273 According to Krippendorff (2018), the simplest inter-coder agreement coefficient is: fi ff ffl 1 D observed disagreement D expected disagreement o e When coders pay no attention to the text, which means they just apply the codes arbitrarily and their coding is unrelated to what the text is about, then the observed disagreement Do equals the expected disagreement De, then α = 1–1 = 0.000. On the other hand, if agreement is perfect, which means disagreement Do = 0, then α = 1– 0 = 1.000. To calculate α for the example shown in Figure 9.31, we first must calculate the observed and the expected disagreement. This requires that we create a coincidence matrix. If you feel overwhelmed reading the following information, do not worry. You do not have to do the calculations yourself. ATLAS.ti does them for you. However, some mathematically interested readers might want to know how the agreement coefficients are computed. You can also skip the formulas and continue with Figure 9.34. Figure 9.32 Coincidence matrix Based on the document continuum shown in Figure 9.31, summing the intersections of A–B + B–A yields the following account: The sum of coincidences where coders agree to be relevant (labeled +): l++= 2(10 + 12 + 4) = 52 The sum of coincidences where coders agree to be irrelevant (labeled φ): lφφ= 2(14 + 11 + 8 + 15 + 12) = 120 The sum of coincidences where coders disagree on their relevance (φ ≠ +): lφ+= 2 + 2 + 4 + 6 = 14 = l+φ Based on these values, we can now construct the contingency tables for perfect agreement, observed and expected agreement/disagreement. The units of agreement are listed on the diagonal from the top left to the bottom right. The matrix on the right shows the state of complete disagreement. See Figure 9.33.

274 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Figure 9.33 Relation between perfect agreement, observed and expected agreement and disagreement, and no agreement We can now calculate the values for observed and expected disagreement. The observed disagreement would be: |cu Do l l l = + + .. = 14 + 14 200 = 0.140 φ φ The expected disagreement would be: |cu De l l l l l l = . .+ + . +. .. .. 1 = 134.66 + 66.134 200 200 1 =0.44 fi fi ffl ffi ff ff ffl ffi 4 And the inter-coder agreement would be: |cu |cu D o |cu D e l l l l fi =1 =1 .. 1 . . + = =1 0.140 0.444 =1 200 1 ff ff ff fflffi ffl ff ff ff fl  fl  14 134.66 =0.685 When users employ alternative codes with more options than the binary |cuα, the coincidence matrices are larger, but the logic stays the same. For example: Figure 9.34 Contingency table for a more complex example

TEAMWORK 275 The matrices with perfect and expected agreement/disagreement serve only as a benchmark. In this simple example, you could have worked it out in your head, but in a real example in ATLAS.ti this is no longer possible. The numbers are much higher and often also odd, since the calculation is based on the number of coded/non-coded characters/ seconds. Therefore, it is good to see the perfect and expected agreements as a point of reference. In the example above, the tables represent a semantic domain with five codes. Here two coders were employed to code the data. The matrix on the left for perfect agreement shows the percentages of how the five codes of the domain were applied under the condition of perfect agreement: code 1 to 20%, code 2 to 40%, code 3 to 10%, code 4 to 20% and code 5 to 10% of all data coded with codes of this semantic domain. The two matrices on the right show the two possible distributions for expected agreement by chance, if the coders had applied the codes arbitrarily. In the contingency table for observed agreement, we can see that there is perfect agreement for code 4. There is some confusion about the application of code 2 and some more for code 5. For code 5, for instance, you need to ask why it was confused with codes 1 and 3? Are there any overlaps in the definition? Why was it understood in different ways? For all codes where there is confusion, the code definition needs to be revisited. Krippendorff’s family of alpha coefficients – from the general to the specific Inter-coder agreement can be measured at different levels. At the most general level, you get a value for relevance. That is, have the coders identified the areas relevant to the research question in a consensus manner? With reference to ATLAS.ti, this means agreement or disagreement at the level of the quotation, regardless of what was coded there. Figure 9.35 Krippendorff’s alpha family – from the general to the specific

276 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI At the next level, you get a coefficient for all semantic domains that are included in the analysis. This value takes into account the multi-valued coding of codes from different semantic domains and is not just the average of the coefficients for all the semantic domains used in the analysis. This is the overall metric you would report in your thesis or journal article. For this and the two subsequent coefficients only the coded matter is considered. Next, you can see if coders could identify the same semantic domain and how well they can distinguish between codes within the semantic domain. • |cuα indicates the extent to which coders agree on the relevance of texts for the research project. • Suα indicates the extent to which coders agree on the presence or absence of sematic domains. • (s)uα indicates the degree to which coders identify a particular semantic domain. • csuα indicates the agreement on coding within a semantic domain. For calculating csuα, it is important that codes of a semantic domain have been applied in a mutually exclusive manner. If not, the measure is only an approximation of what agreement might become if the actual confusion were corrected. It should not to be used to infer replicability or be published. It serves mainly as an invitation to examine the lack of mutual exclusiveness and to correct it. As remedial action you can either change the coding or revise the coding instructions. If ATLAS.ti identifies non-mutually exclusive coding, the alpha-value is marked in red in the result table. Table 9.6 below shows how the coefficients are presented in ATLAS.ti. You need to read the table from left to right, from the general to the specific. By clicking on a table cell in ATLAS.ti, you will see the coding on the document continuum and the matrices that I have explained above. See Skills training 9.2 for further information. Table 9.6 Result table for all coefficients Relevance All semantic domains Domain name Identification Coding |cuα Suα Name 1 (s)uα csuα Name 2 (s)uα csuα Name 3 (s)uα csuα Other methods for analyzing inter-coder agreement In addition to the Krippendorff alpha coefficients, you can also calculate simple percent agreement and the Holsti index. These coefficients have been included because they are widely used even though they are inferior and not recommended for scientific reporting. I will use them for educational purposes only. Percent agreement is easy to demonstrate – much easier than the calculations I walked you through above – so, as a first step, it is easy to

TEAMWORK 277 understand for students. Based on this, one can introduce the notion of chance agreement and thereby show why both percent agreement and the Holsti index are not good measures for reproducibility. A third coefficient that is often used is Cohen’s kappa. It also has some flaws. I will explain this in more detail below. Percent agreement Percent agreement is a simple measure, but as Krippendorff (2018: 306) puts it: ‘it is simply uninterpretable as a measure of reliability’. It is calculated as follows: Percentage of agreement = Number of units of coding on which the coders agree Total number of units coded ×100 Consider the following example: three coders, Tom, Ani and Lena, were asked to rate 12 newspaper articles to see if the politician featured in the article was rated positive (= yes) or not (= no). This is how they coded the articles: Table 9.7 Coding by three coders Articles 1 2 3 4 5 6 7 8 9 10 11 12 Tom no yes no yes yes no no no yes yes yes yes Ani no yes no yes no no no no yes yes no yes Lena no yes yes no no yes no no yes no no no The data matrix tells us that Tom and Ani agree 10 times out of 12, or 83%. Ani and Lena agree seven out of 12 times, or 58%, and Tom and Lena agree five out of 12 times, or 42%. The general agreement is not so obvious. What we cannot derive from this calculation is whether Tom, Ani and Lena read the articles at all but simply gave their ratings randomly. If random coincidence (or chance agreement) is considered, the inter-coder agreement coefficient is only α = 0.234. You will find a step-by-step calculation for this example in Krippendorff (2018: 305–6). This shows that percent agreement as measure of inter-coder agreement inflates and distorts the results. It shifts its meaning with the number of items coded, is not capable of accounting for units of unequal length and its zero value equals the condition of maximum disagreement, which is far from being a chance event. In order to draw valid conclusions, you would not want to rely on the coding in the above example.

278 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Holsti index Holsti’s method (Holsti, 1969) is a variation of percent agreement, as percent agreement cannot be used if the coders have not all coded the same quotations. When coders set their own quotations instead of coding predefined quotations, the Holsti index needs to be used. As the Holsti index does not consider chance agreement, it has the same drawbacks as percent agreement: Holsti index (for 2 coders) = 2 number of agreements number o × f units coded by coder 1 +number of units coded by coder 2 Percent agreement and the Holsti index are equal when all coders code the same quotations. Cohen’s kappa Cohen’s kappa (Cohen, 1960) is another coefficient used in the calculation of inter-coder agreement, offered by other CAQDAS programs. I mention it here because I’ve been asked by users why ATLAS.ti does not offer it. Kappa is a modification of Scott’s P (Scott, 1955) and, according to Krippendorff (2018) and Zwick (1988), a rather unfortunate modification. There is a conceptual flaw in Cohen’s calculation, as kappa counts disagreements between observer preferences for available categories as agreements. In addition, both Cohen’s kappa and Scott’s P assume an infinite sample size. Krippendorff’s alpha coefficient is sensitive to different sample sizes and can also be used on small samples. Decision rules In addition to the coding rules, further considerations must be made. To obtain a meaningful coefficient, you need to decide how much data needs to be coded and how many instances of any given code. In addition, you need some decision aids to evaluate when a coefficient is satisfactory. Sample size The data you use for measuring the inter-coder agreement, from which you can infer reliability of the data you wish to analyze, needs to be representative of the total amount of data you want to analyze. Furthermore, the number of codings per code needs to be sufficiently high. As a rule of thumb, the codings per code should at least yield five agreements by chance. Bloch and Kraemer’s formula 3.7 (1989: 276) can, for instance, be used to obtain the required sample size:

TEAMWORK 279 N c Z p min min min P c P c min fi ff ffl ffl ffl 2 * ffl 2 1 3 4 1 1 , ffi ffi ffi ffi fl fl  fl  fl          Z for a p-value of 0.05 is 1.65 (see the standard normal distribution table if you use a different p-value). Pc: proportion of a code within a semantic domain. If there are four codes in a semantic domain, the proportion is 1/4=0,25=25%. The formula is applicable to two coders working with pre-defined quotations. If you work with three coders, multiply z2 with 3 instead of 2, and so on. Krippendorff (2018: 354) provides a table that lists the sample size for the three smallest acceptable reliabilities αmin: 0.667/0.800/0.900; for four levels of statistical significance: 0.100/0.050/0.010/0.005; and for semantic domains up to ten codes (probability pc). See Figure 9.36 below. The values in the table are calculated for two coders. Figure 9.36 Determination of sample size If you have a semantic domain with four codes and each of the codes are equally distributed, you need a minimum of 139 codings for this semantic domain if the minimum alpha should be 0.800 at a 0.05 level of statistical significance. This is after merging the projects of the two coders. For a lower alpha of 0.667, you need a minimum of 81 codings at the same level of statistical significance. If the frequencies across the subcategories per domain are not equally distributed, you need to increase the sample size. For four codes in a semantic domain, the estimated proportion pc of all values c in the population is .250 (¼). If the distribution is unequal, you need to make a new estimate for the sample size by using a pc in the above formula that is correspondingly less than ¼. Acceptable level of reliability The last question we need to discuss is when to accept or reject coded data based on the inter-coder agreement coefficient. Krippendorff’s (2018) recommendations are: • Strive to reach α = 1,000, even if this is just an ideal. • Do not accept data with reliability values below α = 0.667.

280 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI • In most cases, you can consider the data to be reliable if α = 0.800. • Select at least 0.05 as the statistical level of significance to minimize the risk of a Type 1 error, which is to accept data as reliable when they are not. You can choose another level of statistical significance depending on the risk of failure you can tolerate. Which cut-off point you choose always depends on the validity requirements that are placed on the research results. If the result of your analysis affects or even jeopardizes someone’s life, you should use more stringent criteria. A cut-off point of α = 0.800 means that 80% of the data are coded to a degree better than chance. If you are satisfied with α = 0.500, this means 50% of your data is coded to a degree better than chance. The latter sounds to me a bit like tossing a coin and playing heads or tails. SKILLS TRAINING 9.2 ANALYZING INTER-CODER AGREEMENT Project set-up As principal investigator you have developed a coding framework with semantic domains and their codes. Currently, there is no function to define a semantic domain formally. As was shown in Figure 9.28, an indirect way of creating semantic domains is to add the same prefix to all codes belonging to the same semantic domain and to create code groups. See Chapter 5 for further information. Next, you want to check if other coders (two or more) identify the same data segments to be relevant and code the data in the same way. Figure 9.37 Preparing your project for the inter-coder agreement analysis • Open your project, take a snapshot and rename the snapshot to make it clear that this is the version for testing inter-coder agreement – for example, Project X_sub project for inter-coder agreement. Close the original project. Open the snapshot and continue to use the snapshot project. Based on the estimated required sample, remove the appropriate number of documents from the project: • Open the Document Manager. Select all documents that should not be coded by the other coders. Click on the Delete document(s) button in the ribbon. Option A: coders get a project with predefined quotations • Highlight all documents, click on Tools in the contextual tab for documents and select Remove Codings (Figure 9.38).

TEAMWORK 281 Figure 9.38 Remove codings to set up a project with predefined quotations The project now holds the documents, quotations and all codes and code groups with definitions. If you have written memos, you must decide whether or which ones are relevant for the other coders. Remove those that you do not want to share. If you remove only the codings, the other coders do not need to decide what is relevant matter in relation to the research question. They only need to apply the codes to the existing quotations. This means the inter-coder agreement coefficient for relevance |Cuα will be 1; but, as a matter of fact, it cannot be determined for this data. Option B: coders get a project that only contains the documents to be coded and the codes • Open the Quotation Manager. Select all quotations (Ctrl+A). Click on the Delete Quotation(s) button in the ribbon. • Save the project and export it. • Send or give the project bundle file to the other coders. To facilitate access to each semantic domain, create a code group for each semantic domain. I recommend using capital letters for code groups of semantic domains as in the inter-coder agreement analysis they become titles. The other coders import the project bundle file and rename the project during the process of importing by adding their name or initials. They code the data by either creating their own quotations or by applying the codes of the code system to the already existing quotations. If they have questions about the coding system, they may create a memo and write all questions, concerns, needs for clarifications, etc. into the memo. If a question concerns a specific quotation, it can be written into a quotation comment. When they are done coding, they should check for non-mutually exclusive codings. Currently, ATLAS.ti only automatically checks for them when running an inter-coder agreement analysis. Coders can, however, check for it manually using the Code Co-occurrence Explorer. This is how it is done:

282 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI • Open the Code Co-occurrence Explorer – for example, in the navigation panel on the lefthand side (click on the button Navigator and select the option from the drop-down menu). • Open the code tree and then the sub branches for each code. As shown in Figure 9.39, there is no problem with the semantic domain BENEFITS. There are some co-occurrences, but all with codes from other domains. Figure 9.39 Checking for non-mutually exclusive coding There is a problem with the semantic domain CHARACTERISTICS. The two sub codes ‘understanding’ and ‘trust’ were not coded as mutually exclusive. However, the overlap is just a blank space and, thus, an unintentional error. It can be corrected quickly. Figure 9.40 Detecting and correcting non-mutually exclusive coding After each coder has checked their coding, they save the project, export it and send the project bundle file back to the principal investigator.

TEAMWORK 283 Performing inter-coder agreement analysis The principal investigator imports the project from the other coders and checks their work, enables inter-coder mode and merges the projects. If you do not have your own project yet but want to practice, you can download an example project from the companion website. Three projects are available: two sub projects of coders Sabine and Susanne (they have each coded three documents) and one merged project based on three coders. The latter project is used in the following; it has only one document and is for illustration purposes only. Figure 9.41 Merging sub projects as preparation for inter-coder agreement analysis • Import the projects of the other coders and review them. • Close all but one project as you cannot merge an opened project. • Enable inter-coder mode: select the Analyze tab and click on Enable Inter-coder Mode. Figure 9.42 Enabling the Inter-coder Mode • Merge the projects of coders 1 and 2. If there are more than two coders, repeat the process, (see Skills training 9.1). • Save the merged project. • To see the authors of the various codings, change the view settings for icons to Show User in the View tab of the contextual document ribbon (Figure 9.20):

284 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Figure 9.43 Display of the codings of different users Each coder is indicated by a different color. If you hover over the user icon, the name of the user is shown. • To start analyzing inter-coder agreement, click on the Inter-coder Agreement button in the Analyze tab. The following instruction applies to version 8.5 of ATLAS.ti and higher. At the time of writing this version of the tool was not completly finished. The released version might look sligtly different. Figure 9.44 Opening screen for the inter-coder agreement analysis A coding is the link between a code and a quotation. In Show User mode, you can see which coder has applied which code(s) to a quotation (see Figure 9.43).

TEAMWORK 285 If you add documents first, ATLAS.ti will automatically recognize all available coders. Each coder is assigned a color by ATLAS.ti. If you use version 8.4 or earlier, you first need to add coders. • Drag and drop document(s) from the Project Explorer to the Add Document field. Figure 9.45 Adding document(s) and coders Next, you need to add semantic domains: The easiest way is to drag and drop one or more code groups from the Project Explorer into the field Add Semantic Domain. This option is not available in version 8.4 or earlier. You need to highlight all the codes of a code group and drag them into the semantic domain field. As mentioned, a single code can also stand for a semantic domain, but it is rare that there is no variability within a domain or at least an indication of presence and absence. • Click on the Add Semantic Domain button and add one or more codes to the domain. Alternatively, you can drag and drop codes from the Project Explorer or the code list into the semantic domain field: Figure 9.46 Adding semantic domains

286 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Figure 9.47 Adding codes to a semantic domain Figure 9.48 Renaming a semantic domain If you drag code groups into the semantic domain field, the name of the code group will be used as the name for the semantic domain. ATLAS.ti assigns a color to each semantic domain so that they can be distinguished in the further analysis steps when you check for semantic domain identification. Each code within each semantic domain is also assigned a color. This is needed for the analysis of codings within each domain. The colors are randomly assigned and are unrelated to the colors you have assigned to each code. You can rename or delete a semantic domain by hovering over the name field and clicking on the rename button or the X. The X for removing an item also shows if you hover over coder names, codes and documents.

TEAMWORK 287 When you have added document(s), coders and semantic domains, the agreement matrix is displayed. To run the analysis, you need to select an agreement measure. • Click on the Agreement Measure button in the ribbon and from the drop-down menu select Krippendorff’s Alpha. This adds the alpha coefficients for the various analysis levels into the matrix. Figure 9.49 Agreement matrix with alpha coefficients Interpreting results In the following, only a small example project will be used for reasons of simplicity and space. The results in Figure 9.49 show that the alpha for relevance is high: |Cuα = 0.968. The three coders consider mostly the same data segments to be relevant. The agreement on the presence or absence of a sematic domain is lower, but still acceptable: Suα = 0.781. On the next level, we can see the degree to which coders were able to identify a semantic domain. Agreement for the domain ACTIVITIES ((s)uα = 0.988) is highest, followed by BENEFITS ((s)uα = 0.713) and CHARACTERISTICS ((s)uα = 0.692). The coefficients in the last column, Coding, show us how good the agreement is within a semantic domain. As you can see from Figure 9.49, the agreement within a semantic domain is independent of the value that tells you how well coders were able to identify a domain. The alpha for csuα can be higher or lower than (s)uα. The alpha for the domain CHARACTERISTICS is displayed in red and has a note attached to it. This means within the domain there is a violation of non-mutually exclusive coding (see below). Let’s look at the results more closely by opening the document continuum and matrices for each alpha value.

288 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Relevance • Click on the alpha value for relevance: |Cuα. If the matrix does not open right away, click on the matrix button on the right-hand side at the end of the document continuum. Figure 9.50 Document continuum and agreement/disagreement matrix for relevance In Figure 9.50, I have marked the two areas where the coders did not agree. In the first box, you can see that the second coder didn’t consider a segment relevant that the other two coders did. In the second box, you can see that coder 3 disagrees with the other two coders. The first matrix on the left shows perfect agreement: 20.737% of the material would have been coded by the three selected domains, while 79,263% would not have been coded. The matrix in the middle shows what was actually coded and not coded (20.182%/78.708%). The value of 0.555% tells us that there is very little disagreement. The matrix on the righthand side shows how much the coders would have agreed or disagreed if they had left it up to chance. The matrix of Perfect Agreement show how coders should have coded the documents to get a score of 1 (one), indicating perfect reliability. This matrix should be compared to the matrix of Observed Agreement/Disagreement to identify the sources of unreliability. The matrix of Observed Agreement/Disagreement shows how coders actually coded the documents. All disagreements reduce the reliability of the data. The occurrence serves as an invitation to do better, whether by improving the clarity of the coding instructions or calling for coders to be more attentive to the differences to be coded. The matrix of Expected Agreement/Disagreement shows the hypothetical condition under which coding is unrelated to apparent qualities of the documents. This is expected when coders paid no attention to coding instructions and to the content of the documents to which they were to be applied. Under this condition, alpha measures 0 (zero).

TEAMWORK 289 Agreement on the presence or absence of semantic domains Next, let’s look at the matrix for all semantic domains. I replaced the different colored diamonds with different shapes so that you can see the domains in a non-colored print. Figure 9.51 Matrices for all semantic domains The first thing to note is that we are now looking at a subset of all document units, only those that have been considered relevant by the coders for the three selected semantic domains. The 100% basis for calculation is 37.807 document units. Under perfect agreement, 19.933% would have been attributed to the domain ACTIVITIES, 45.296% to the domain BENEFITS, 34.417% to the domain CHARACTERISTICS and the remainder to a combination of domain codes from different domains. To evaluate how well the coders agreed, you can use these numbers as a baseline and compare them with the second matrix that shows the observed instances. For the domain ACTIVITIES, there is no difference between perfect and observed agreement. All coders identified the same segments to be about activities. There is some confusion related to the two other domains. At times, the coders could not map the domains BENEFITS and CHARACTERISTICS. 7.007% of the time they chose benefits rather than characteristics and vice versa. To look up which codes were confused, you can access the coding in context. • Look at the document continuum and find an area where you see disagreement. In ATLAS.ti this is indicated by the use of different colors. In Figure 9.53, I marked such an area with a black box. • Select the document continuum of one coder. • Click on the Show Quotations button in the ribbon. Figure 9.52 Show Quotations

290 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI This brings up all quotations of the selected coder for all selected semantic domains. Click on the area where you see disagreement. This highlights the corresponding quotation in the quotation list. Figure 9.53 Looking for the source of confusion • Double-click and view the quotation in context to inspect which codes were applied. Figure 9.54 Checking which codes were confused This will allow you to see which codes were confused, and you can review, for instance, the code definitions. Maybe the code definition was not clear enough so that a characteristic of friendship has been mixed up with the benefit of helping each other. By checking the confusion cases, you can identify code-system vulnerabilities and improve on them. In the worst case, it allows you to recognize that your coders have not taken their task seriously and coded without paying much attention to code definitions. Domain identification If you are interested in how well the coders identified the semantic domains, click on the coefficient (s)u alpha. Figure 9.55 shows that the observed agreement is nearly perfect. The coders have assigned 19.445% of all coded segments to a code from the ACTIVITIES domain. With perfect agreement it would have been 19.933%. The basis for the calculation is all coded data of the three selected domains (37,807 units). The other two domains took up 79.579% of the coding.

TEAMWORK 291 Figure 9.55 Reliability for domain identification Agreement on coding within a semantic domain On the fourth level you can check how well the coders differentiated among codes within a semantic domain. We are now looking only at the data segments where at least one of the coders assigned a code from the selected semantic domain. As above, click on the corresponding alpha coefficient: csuα. Figure 9.56 Agreement on coding within a semantic domain By now you should have an idea of how to read the matrices. Even though the coders were good in identifying the overall theme (i.e., all data segments that related to activities), within the semantic domain there is quite a bit of confusion: 12.915% of the data segments within the domain should have been coded with the sub code ‘going for a meal’, but there was only agreement on 5.776% of the data units. 1.624% of the data units were coded with the code ‘going for coffee’, 2.424% with the code ‘going out’, 1.505% with the code ‘miscellaneous’ and 1.587% with the code ‘sports’. The agreement for this and the other codes in this domain is not much better than chance agreement. This means you need to take a closer look at the code definitions and talk to the coders about why there is so much confusion. Note that the numbers above or below the diagonal in the matrix of expected agreement/disagreement show only one of many possibilities.

292 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Here is a summary of the general procedure for reading the matrices. First, look at the diagonal of the first two table and compare what the coding would look like under the condition of perfect agreement and what it actually looks like (observed agreement). If you compare the numbers, you can quickly see where the deviations are. Next, look at the numbers in the cells either above or below the diagonal; the table is symmetric. They are red in the ATLAS.ti interface. These numbers show you where the confusion is – in other words, which codes the coders applied in different ways. All numbers in the table add up to 100%. If you look across a row or a column, you can see which codes were used instead. Figure 9.57 shows a simplified example for the semantic domain BENEFITS. The domain sub code ‘caring for each other’ was confused with two other codes: ‘learning from each other’ and ‘supporting each other’. The code ‘helping each other’ was confused a few times with ‘supporting each other’, and the code ‘supporting each other’ mainly with ‘caring for each other’. Figure 9.57 Simplified example Therefore, you must check the code definitions to see if they are unclear and do not sufficiently differentiate the codes. You can also look at the quotations that have been coded, as shown above, and discuss the disagreements with the coders to give you an indication of why coders have confused these codes. After revising the code system, a new group of coders need to code the data. Afterwards, you run the inter-coder agreement analysis once more. You repeat this as often as necessary to achieve sufficient reliability. Violation of mutually exclusive coding As you may have noticed in Figure 9.49, within the domain CHARACTERISTICS there is an issue with one or more non-mutually exclusive codings. ATLAS.ti will show you where those instances are if you open the coding matrix:

TEAMWORK 293 Figure 9.58 Matrixes for perfect and observed agreement in the case of non-mutually exclusive coding. (The way non-mutually exclusive codings is displayed in the matrix may be different in the released version.) There will be a row/column in the matrix for each set of codes that have not been coded in a non-mutually exclusive manner. In Figure 9.58, we see that the two sub codes ‘trust’ and ‘understanding’ have not been applied in a mutually exclusive manner. Take a look at the document continuum. All areas where non-mutually exclusive coding occurs are underlined. Select the option Show Quotations and click on the area: the quotation with non-mutually exclusive coding is highlighted. Double-click to view it in context. Figure 9.59 Finding non-mutually exclusive coding This is the same area we looked at earlier when I explained how each coder can use the Code Co-occurrence Explorer to ensure that all encodings are mutually exclusive. The overlapping area is just a blank space. This can be corrected by the principal investigator without hesitation. However, if two codes of the same domain were applied to the same quotation, the principal investigator cannot decide which code to change. This would bias the intercoder agreement analysis. In this case, keep in mind that the estimated alpha value increases the real value; this means that the data is less reliable than it seems.

294 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI Qualitative comparison of the codings of different coders You can also use the inter-coder agreement tool for a qualitative comparison of the codings of different coders. • You just start, as above, by adding a document and semantic domains. I recommend going through one document at a time; otherwise, the codings will be difficult to see on the document continuum. • Activate the view option Show Quotations. • Look for disagreements in the document continuum and activate the corresponding quotations. • Reading the quotation preview and looking at the codes that have been applied may already give you enough information. If you need more detail, you can look up the quotation in the document context. Make sure to change the margin view to see the coders in the margin area. • You can use the quotation comment for writing nodes while you discuss the codings. These will help you later in improving the code system. Figure 9.60 Using the inter-coder agreement tool for a qualitative comparison of codings from different coders REVIEW QUESTIONS 1 What are the two most important things that you need to know when setting up a team project? 2 How would you set up a team project when all team members analyze the same set of data? 3 How would you set up a team project when all team members analyze different data? 4 How would you proceed to develop a code system with a team? 5 If you want to teach ATLAS.ti to your students, which type of classroom project would you choose and why? Develop a plan for how you would set up the projects for ATLAS.ti. What type of data will you use? What should be the student tasks? What do you want the students to learn? What should be the outcome? Even though not all of the following are explicitly discussed in this chapter, if you are a project manager for your team, you should be able to answer the following questions:

TEAMWORK 295 6 How do you change the library? When is it necessary to work with a different library? 7 How do you import a list of codes with definitions? 8 What is a snapshot project? 9 How do you merge projects? 10 What is inter-coder agreement analysis? What is the purpose and when is it useful? 11 If you set up a project for inter-coder agreement analysis, what do you need to pay attention to? 12 What is the minimum number of people required for an inter-coder agreement analysis? 13 What is a semantic domain and how do you create one in ATLAS.ti? 14 What is multi-valued coding? 15 What is mutually exclusive coding? 16 Which rules should you observe when preparing for and running an inter-coder agreement analysis? 18 How can inter-coder agreement be measured? 19 Which of the three methods offered by ATLAS.ti for inter-coder agreement is recommended? Why? 20 Describe Krippendorff’s alpha family of agreement coefficients. 21 How do you run an inter-coder agreement analysis in ATLAS.ti? 22 How do you interpret the results? FURTHER READING Teaching Kalpokaite, Neringa and Radivojevic, Ivana (2016). Integrating ATLAS.ti into an undergraduate qualitative research course: evaluating students’ experiences, in Friese, Susanne and Ringmayr, Thomas (eds) ATLAS.ti User Conference 2015: Qualitative Data Analysis and Beyond. Conference proceedings. Universitätsverlag TU Berlin, August.http://dx.doi.org/10.14279/ depositonce-5158. Paulus, Trena and Bennet, Ann (2014). Teaching qualitative methods with ATLAS.ti: Beyond data analysis, in Friese, Susanne and Ringmayr, Thomas (eds) ATLAS.ti User Conference 2013: Fostering Dialog on Qualitative Methods. Conference proceedings. Universitätsverlag TU Berlin, May. http://dx.doi.org/10.14279/depositonce-4837. Inter-coder agreement Guba, Egon G. and Lincoln, Yvonna S. (2005). Competing paradigms in qualitative research, in Denzin, Norman and Lincoln, Yvonna S. (eds) The SAGE Handbook of Qualitative Research, 3rd edn, 191–225. London: Sage. Harris, Judith, Pryor, Jeffry and Adams, Sharon (2006). The challenge of intercoder agreement in qualitative inquiry. Working paper.https://www.researchgate.net/publication/228490436_ The_challenge_of_intercoder_agreement_in_qualitative_inquiry. Krippendorff, Klaus (2018). Content Analysis: An Introduction to Its Methodology, 4th edn. Thousand Oaks, CA: Sage.

296 QUALITATIVE DATA ANALYSIS WITH ATLAS.TI MacPhail, Catherine, Khoza, Nomhle, Abler, Laurie and Ranganathan, Meghna (2015). Process guidelines for establishing Intercoder Reliability in qualitative studies. Qualitative Research, 16(2), 198–212. Richards, Lyn (2009). Handling Qualitative Data: A Practical Guide, 2nd edn. London: Sage. Rolfe, Gary (2006). Validity, trustworthiness and rigour: Quality and the idea of qualitative research. Methodological Issues in Nursing Research, 53(3), 304–10. http://garyrolfe.net/ documents/validitytrustworthiness.pdf. Schreier, Margrit (2012). Qualitative Content Analysis in Practice. London: Sage.

Epilogue As you can imagine, like every author, I was happy when the manuscript for this edition was done. Writing about software, however, is like football (soccer): after the game is before the game. When this book is published, we will already be in the middle of planning for version 9 of ATLAS.ti. Version 9 will offer new features, but the difference from version 8 will not be as great as between versions 7 and 8. I’m pretty sure you will be able to use this book with version 9 as well. Nevertheless, I would like to invite you to be exploratory. If an instruction in the book does not work, the button may have a new icon, been renamed or been moved to another ribbon or menu. Software development never stops, because it has become very easy to distribute updated versions over the Internet. On the positive side, this allows for adding new features and improving existing ones. Things may look good on the drawing board but may not work so well in practical daily research work. Instead of waiting for a new major release, improvements can be released with a new service pack. An online version of the book, which can be updated continuously, could be a solution. But maybe some of you, no matter how old you are, like me, still prefer a printed version of the book for the bookshelf or to have a copy of the book to lay next to you on the desk when working on the computer. Even if you are reading this now in the e-book version on your computer, the e-book is also a static version of the book. Therefore, I would like to repeat the invitation to just try out buttons and options in the software. After you have worked through the book, you know everything you need to find your own way. Do not be afraid to click a button or open a menu you have not seen yet. The worst thing that can happen is that the software crashes. Do not worry, ATLAS.ti saves all your steps and will at restart offer the last version it has saved. Nevertheless, I would like to remind you, as your teacher, that you should still save frequently AND export the project after each work session and save it as an external backup copy. Until we meet again in the fourth issue of the book, the world will change and with it the software. I am looking forward to an exciting future in general and in the field of computeraided qualitative data analysis. Yours faithfully

References Alexa, Melina and Zuell, Cornelia (2000). Text analysis software: Commonalities, differences and limitations. The results of a review. Quality & Quantity, 34(3), 299–321. Araujo, Luis (1995). Designing and refining hierarchical coding frames, in Kelle, Udo. (ed.) Computer-aided Qualitative Data Analysis, chapter 7. London: Sage. Barbour, Rosaline S. (2001). Checklists for improving rigour in qualitative research: A case of the tail wagging the dog? BMJ, 322(7311), 1115–17. Barry, Christine A. (1998). Choosing qualitative data analysis software: Atlas/ti and NUD*IST compared. Sociological Research Online, 3(3), www.socresonline.org.uk/socresonline/3/3/4. html. Bazeley, Pat (2007). Qualitative Data Analysis with NVivo. London: Sage. Bazeley, Pat (2013). Qualitative Data Analysis: Practical Strategies. London: Sage. Bazeley, Pat and Richards, Lyn (2000). The NVivo Qualitative Project Book. London: Sage. Bergman, Manfred Max and Coxon, Anthony P.M. (2005). The quality in qualitative methods. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 6(2), Art. 34, http://nbn-resolving.de/urn:nbn:de:0114-fqs0502344. Bernard, H. Russell and Ryan, Gery W. (2010). Analyzing Qualitative Data: Systematic Approaches. London: Sage. Birks, Melanie, Chapman, Ysanne and Francis, Karin (2008). Memoing in qualitative research: Probing data and processes. Journal of Research in Nursing, January, 13(1), 68–75. Blanchette, Oliva (2003). Philosophy of Being: A Reconstructive Essay in Metaphysics. Washington, DC: The Catholic University of America Press. Bloch, Daniel A. and Kraemer, Helena Chmura (1989). 2 x 2 kappa coefficients: Measures of agreement or association. Biometrics, 45(1), 269–87. Blumer, Herbert (1969). Symbolic Interactionism: Perspective and Method. Englewood Cliffs, NJ: Prentice Hall. Bogdan, Robert C. and Biklen, Sari Knopp (2007). Qualitative Research for Education: An Introduction to Theories and Methods, 5th edn. Boston: Pearson Education. Bruner, Jerome S., Goodnow, Jacqueline J. and Austin, George A. (1956). A Study of Thinking. New York: John Wiley & Sons. Cameron, Roslyn (2011). An analysis of quality criteria for qualitative research. 25th ANZAM Conference, Track: Research Methods, https://www.anzam.org/wp-content/uploads/pdfmanager/595_ANZAM2011-375.pdf. Charmaz, Kathy (2002). Qualitative interviewing and grounded theory analysis, in Gubrium, Jaber F. and Holstein, James A. (eds) Handbook of Interview Research: Context and Method, 675–94. Thousand Oaks, CA: Sage. Charmaz, Kathy (2006/2014). Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis. London: Sage.

REFERENCES 299 Cohen, Jacob (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. Corbin, Juliet and Strauss, Anselm (2008/2015). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 3rd and 4th edns. Thousand Oaks, CA: Sage. Cortazzi, Martin (1993). Narrative Analysis. London: Falmer Press. Dey, Ian (1993). Qualitative Data Analysis: A User-friendly Guide for Social Scientists. London: Routledge. Di Gregorio, Silvana and Davidson, Judith (2008). Qualitative Research Design for Software Users. Maidenhead: Open University Press/McGraw-Hill. Fielding, Nigel G. and Lee, Raymond M., (1998). Computer Analysis and Qualitative Research. London: Sage. Forrester, Michael A. (ed.) (2010). Doing Qualitative Research in Psychology: A Practical Guide. London: Sage. Freeman, Melissa (2017). Modes of Thinking for Qualitative Data Analysis. New York: Routledge. Friese, Susanne (2000). Self-concept and Identity in a Consumer Society: Aspects of Symbolic Product Meaning. Marburg: Tectum. Friese, Susanne (2011). Using ATLAS.ti for analyzing the financial crisis data [67 paragraphs]. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 12(1), Art. 39, http:// nbn-resolving.de/urn:nbn:de:0114-fqs1101397. Friese, Susanne (2016a). Qualitative data analysis software: The state of the art. Special Issue: Qualitative Research in the Digital Humanities, Bosch, Reinoud (ed.) KWALON, 21(1), 34–45. Friese, Susanne (2016b). CAQDAS and grounded theory analysis. MMG Working Paper 16-07, October 2016. Friese, Susanne (2018). Computergestützte Analyse: Das Kodieren narrativer Interviews, in Pentzold, Christian, Bischof, Andreas and Heise, Nele (eds) Praxis Grounded Theory. Theoriegenerierendes empirisches Forschen in medienbezogenen Lebenswelten. Ein Lehr- und Arbeitsbuch, 277–309. Wiesbaden: Springer VS. Friese, Susanne (2019). Grounded theory analysis and CAQDAS: A happy pairing or remodeling GT to QDA?, in Bryant, Tony and Charmaz, Kathy (eds) The SAGE Handbook of Grounded Theory, chapter 11. London: Sage. Friese, Susanne and Ringmayr, Thomas (2014). ATLAS.ti User Conference 2013: Fostering Dialog on Qualitative Methods. Conference proceedings. Universitätsverlag TU Berlin. Friese, Susanne and Ringmayr, Thomas (2016). ATLAS.ti User Conference 2015: Qualitative Data Analysis and Beyond. Conference proceedings. Universitätsverlag TU Berlin, August. Friese, Susanne, Soratto, Jacks and Pires, Denise (2018). Carrying out a computer-aided thematic content analysis with ATLAS.ti. MMG Working Paper 18-02, www.mmg.mpg.de/ fileadmin/user_upload/documents/wp/WP_18-02_Friese-Soratto-Pires_AtlasTI.pdf. Gable, Robert K. and Wolf, Marian B. (1993). Instrument Development in the Affective Domain: Meaning Attitudes and Values in Corporate and School Settings, 2nd edn. Boston: Kluwer Academic. Gibbs, Graham (2005). Writing as analysis. Online QDA: http://onlineqda.hud.ac.uk./Intro_ QDA/writing_analysis.php. Glaser, Barney G. (1978). Theoretical Sensitivity: Advances in the Methodology of Grounded Theory. Mill Valley, CA: Sociology Press.

300 REFERENCES Golafshani, Nahid (2003). Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597–606, https://nsuworks.nova.edu/tqr/vol8/iss4/6. Goldstone, Robert L. and Kersten, Alan (2003). Concepts and categorization, in Healy, Alice F. and Proctor, Robert W. (eds) Comprehensive Handbook of Psychology, Volume 4: Experimental Psychology, 599–621. Hoboken, NJ: Wiley. Goleman, Daniel (1995). Emotional Intelligence. New York: Bantam Books. Guba, Egon G. and Lincoln, Yvonna S. (2005). Paradigmatic controversies, contradictions, and emerging confluences, in Denzin, Norman and Lincoln, Yvonna S. (eds) The SAGE Handbook of Qualitative Research, 3rd edn, 191–225. London: Sage. Guest, Greg, MacQueen, Kathleen M. and Namey, Emily E. (2012). Applied Thematic Analysis. Thousand Oaks, CA: Sage. Hartmann, Eddie (2011). Strategien des Gegenhandelns: Zur Soziodynamik symbolischer Kämpfe um Zugehörigkeit. Konstanz: UVK. Hinchliffe, S.J., Crang, M.A., Reimer, S.M. and Hudson, A.C. (1997). Software for qualitative research: 2. Some thoughts on ‘aiding’ analysis. Environment and Planning A, 29(6), 1109–24. Holsti, Ole R. (1969). Content Analysis for the Social Sciences and Humanities. Reading, MA: Addison-Wesley. Holton, Judith A. (2007). The coding process and its challenges, in Bryant, A. and Charmaz, K. (eds) The SAGE Handbook of Grounded Theory, 265–90. London: Sage. Hug, Theo (2001). Editorial zur Reihe: ‘Wie kommt die Wissenschaft zu Wissen?’, in Hug, Theo (ed.) Einführung in die Forschungsmethodik und Forschungspraxis, Band 2, 3–10. Baltmannsweiler: Schneider Verlag. Kelle, Udo und Kluge, Susann (2010). Vom Einzelfall zum Typus: Fallvergleich und Fallkontrastierung in der qualitativen Sozialforschung. Wiesbaden, VS Verlag. Khateb, Asaid, Pegna, Alan J., Michel, Christoph M., Landis, Theodor and Annoni, JeanMarie (2002). Dynamics of brain activation during an explicit word and image recognition task: An electrophysiological study. Brain Topography, Spring 14(3), 197–213. Konopásek, Zdeněk (2007). Making thinking visible with ATALS.ti: Computer assisted qualitative analysis as textual practices. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 9(2), Art. 12, http://nbn-resolving.de/urn:nbn:de:0114-fqs0802124. Krippendorff, Klaus (2004/2018). Content Analysis: An Introduction to Its Methodology, 2nd and 4th edns. Thousand Oaks, CA: Sage. Kuckartz, Udo (1995). Case-oriented quantification, in Kelle, Udo (ed.) Computer-aided Qualitative Data Analysis: Theory, Methods and Practice, 158–66. London: Sage. LeCompte, Margaret Diane and Preissle, Judith (1993). Ethnography and Qualitative Design in Educational Research, 2nd edn. San Diego: Academic Press. Mayring, Philipp (2015). Qualitative Inhaltsanalyse, 12th edn. Weinheim: Beltz. Miles, Matthew B. and Hubermann, A. Michael (1994). Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. Thousand Oaks, CA: Sage. Miles, Matthew B., Huberman, A. Michael and Saldaña, Johnny (2014). Qualitative Data Analysis, 3rd edn. Thousand Oaks, CA: Sage. Morrison, Moya and Moir, Jim (1998). The role of computer software in the analysis of qualitative data: Efficient clerk, research assistant or Trojan horse? Journal of Advanced Nursing, 28(1), 106–16. Muhr, Thomas (1997). ATLAS.ti: Visual qualitative data analysis, User’s Manual and Reference, Version 4.1. Scientific Software Development: Berlin.

REFERENCES 301 Novak, Joseph D. and Cañas, Alberto J. (2006). The theory underlying concept maps and how to construct and use them. Institute for Human and Machine Cognition, http://cmap. ihmc.us/docs/theory-of-concept-maps. Novak, Josef D. and Gowin, D. Bob (2002). Learning How to Learn. New York: Cambridge University Press. (First published in 1984.) Paulus, Trena, Woods, Megan, Atkins, David and Macklin, Rob (2014). Current reporting practices of ATLAS.ti users in published research studies, in Friese, Susanne and Ringmayr, Thomas (eds) ATLAS.ti User Conference 2013: Fostering Dialog on Qualitative Methods. Conference proceedings. Universitätsverlag TU Berlin. Poortman, C.L. and Schildkamp, K. (2012). Alternative quality standards in qualitative research? Quality & Quantity, 46(6), 1727–51. Powdthavee, Nattavudh. (2009). Think having children will make you happy? Think again, suggests Nattavudh Powdthavee you’re experiencing a focusing illusion. The Psychologist, 22(4), 308–11. Prus, Robert C. (1996). Symbolic Interaction and Ethnographic Research: Intersubjectivity and the Study of Human Lived Experience. Albany, NY: SUNY Press. Richards, Lyn (2009). Handling Qualitative Data: A Practical Guide, 2nd edn. London: Sage. Richards, Lyn and Morse, Janice M. (2013). Readme First for a User’s Guide to Qualitative Methods, 3rd edn. Thousand Oaks, CA: Sage. Richards, Lyn and Richards, Tom (1994). From filing cabinet to computer, in Bryman, Alan and Burgess, Robert G. (eds) Analysing Qualitative Data, 146–72. London: Routledge. Richards, Tom and Richards, Lyn (1995). Using hierarchical categories in qualitative data analysis, in Kelle, Udo (ed.) Computer-aided Qualitative Data Analysis, chapter 6. London: Sage. Riessman, Catherine K. (2008). Narrative Methods for the Human Sciences. Thousand Oaks, CA: Sage. Rodik, Petra and Primorac, Jaka (2015). To use or not to use: Computer-assisted qualitative data analysis software usage among early-career sociologists in Croatia. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 16(1), Art. 12, http://nbn-resolving.de/ urn:nbn:de:0114-fqs1501127. Rolfe, Gary (2006). Validity, trustworthiness and rigour: Quality and the idea of qualitative research. Methodological Issues in Nursing Research, 304–10, http://garyrolfe.net/ documents/validitytrustworthiness.pdf. Saillard, Elif Kus (2011). Systematic versus interpretive analysis with two CAQDAS packages: NVivo and MAXQDA. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 12(1), Art. 34, http://nbn-resolving.de/urn:nbn:de:0114-fqs1101345. Schreier, Margrit (2012). Qualitative Content Analysis in Practice. London: Sage. Scott, William A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–5. Seale, Clive (1999). The Quality of Qualitative Research. London: Sage. Seidel, John (1991). Methods and madness in the application of computer technology to qualitative data analysis, in Fielding, Nigel G. and Lee, Raymond M. (eds) Using Computers in Qualitative Research, 107–16. London: Sage. Silver, Christine and Lewins, Ann (2014). Using Software in Qualitative Research: A Step-by-step Guide, 2nd edn. London: Sage. Silverman, David (2000). Doing Qualitative Research: A Practical Handbook. London: Sage.

302 REFERENCES Strauss, Anselm L. and Corbin, Juliet (1998). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 2nd edn. London: Sage. Strübing, Jörg and Schnettler, Bernt (2004). Klassische Grundlagentexte zur Methodologie interpretativer Sozialforschung, in Strübing, Jorg and Schnettler, Bernt (eds) Methodologie interpretativer Sozialforschung: Klassische Grundlagentexte, 9–18. Konstanz: UVK. Tracy, Sarah J. (2010). Qualitative quality: Eight ‘big-tent’ criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837–51, https://www.researchgate.net/ publication/230557825_Qualitative_Quality_Eight_Big-Tent_Criteria_for_Excellent_ Qualitative_Research. Weller, Susan C. and Romney, A. Kimball (1988). Systematic Data Collection. Newbury Park, CA: Sage. Welsh, Elaine (2002). Dealing with data: Using NVivo in the qualitative data analysis process. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 3(2), Art. 26, http://nbn-resolving.de/urn:nbn:de:0114-fqs0202260. Winograd, Terry and Flores, Fernando (1987/1995). Understanding Computers and Cognition. Addison-Wesley. Wolcott, Harry F. (1994). Transforming Qualitative Data: Description, Analysis, and Interpretation. Thousand Oaks, CA: Sage. Wolcott, Harry F. (2009). Writing Up Qualitative Research. London: Sage. Woolf, Nick (2014). Analytic strategies and analytic tactics, in Friese, Susanne and Ringmayr, Thomas (eds) ATLAS.ti User Conference 2013: Fostering Dialog on Qualitative Methods. Conference proceedings. Universitätsverlag TU Berlin. Woolf, Nick H. and Silver, Christina (2017). Qualitative Analysis Using ATLAS.ti/MAXQDA/ NVivo: The Five Level QDA® Method. Oxford: Routledge. Zwick, Rebecca (1988). Another look at interrater agreement. Psychological Bulletin, 103(3), 347–87.

Index Note: Page locators in bold refer to figures and tables. abstract codes 118, 219, 220 adding documents 37, 39 adjacency operators and disjointed quotations 161 alpha co-efficients and inter-coder agreements 275, 275–6, 276, 287, 287 analytic process, the 4, 4–6, 83 and methodological approaches to 6–7 and recursive processes 5, 5 Analyze tab, the 13, 13 AppData\Roaming directory, the 35 appendix, the 234–5 ATLAS.ti mobile app 31 audio/video files 28, 29–30, 30, 31, 91, 95 creating quotations 93, 93–4, 94 displaying video data 91–2, 92, 93 and video quotations 94, 94, 95, 95–6, 96, 97 auto coding 51, 65 backdrop menu, the 14, 14 Basics of Qualitative Research (book) 59 Belkin blog, the 72, 109, 172, 185, 201 Big Bang Theory, The (TV show) 49 Blanchette, Olivia 69 Boolean operators 155–7, 156, 190 CAQDAS (Computer Assisted Qualitative Data Analysis) programs 70, 145, 278 case-based networks 219, 219–20 case patterns in data 206 cases in documents 38 categories and data analysis 6, 104–8, 105, 106 aggregation from descriptive labels 129–35, 130, 132, 133, 134, 143 and coding 69–70, 104–8, 106, 107 and defining code list categories 135, 135–8, 136, 137, 138 and developing subcategories 124–9, 125, 126, 127, 128, 129, 144, 144 and mutually exclusive categories 118–20, 119, 127 changing the default location 46–8, 47, 48 Children & Happiness sample project, the 9, 11, 22–3 circular layouts 212, 212 circular single-cycle layout 212, 213 cloud sharing services 2, 35, 46 clustering of coded multiple segments 168 Code Browser, the 15, 76, 77, 116 Code Co-occurrence Explorer, the 198, 199, 281–2, 293 Code Co-occurrence Table, the 51, 116, 166–8, 167, 178, 191, 235 and global filters 184–5, 185 and networks 196–8, 197, 198 code co-occurrence tools 155 code definitions 83–4, 127 Code Document Table, the 51, 116, 235 and comparing narratives and code frequencies across document groups 6, 169, 169–72, 170, 171, 172 and global filters 182, 186, 186 and querying data 147, 179 Code Forest, the 161, 217, 217 code frequency 117–18, 143, 144, 255 and the Code Document Table 169, 170, 170–1, 171, 172, 172 code groups 130, 242, 254–5, 263, 286 and categories 176–8, 187–8, 188 use as filters 114, 115, 128, 129 code lists 230, 263 advantages of 143, 143–5, 145 and defining categories 135, 135–8, 136, 137, 138 and document analysis in team projects 252, 252–3, 253 and exporting to Excel 86, 86, 235, 235–6, 236 filtering of 75–6, 76, 131, 132, 133, 146 and importing data 85, 85 and network links for 161, 162 and sorting of 115, 115 and structuring of 114, 114, 217 Code Manager, the 17, 18, 21–2, 51, 61–2, 72, 74, 76–7, 81–3 see also coding

304 INDEX code references 74–5 codes as entity browsers 15, 15, 22 coding 1–4, 49, 65, 115, 188–9 benefits of 21 and categorization 69–70, 104–8, 106, 107 coded document content 51, 51 collecting quotations under sub codes 124–5, 125 and color attributes 82, 82, 85, 128, 133 comparison between thematic and interpretive coding 139–46, 140, 143 creation of 81, 81, 112–13, 133 of data segments 70, 71, 72–5, 73, 74 deleting 82, 82 and development of the coding system within a project 120–1, 229–30, 230, 231 distinguishing between levels and types of 145–6, 146 and drag-and-drop coding 76, 76–7, 112, 121, 199, 199 of geo data 101, 101–2 of image documents 99 importing and exporting code lists 85, 85–6, 86 ‘in-vivo’ codes 80, 80–1 and labels (‘tags’) 2, 70, 84, 103, 113, 140 list coding 75–6, 76, 114, 114, 115, 115 of multiple codes 75, 75 and renaming codes 82, 133 and splitting of 125–6, 126, 127, 128, 255 and sub codes 124–5, 125, 126–7, 126, 134 and unlinking and removing 78 of video quotations 96, 97 see also inter-coder agreement coding frames 253–7, 254, 255 codings of project users 284, 284 Cohen’s kappa 278 coincidence matrix 273 color attributes to codes 82, 82, 113, 133, 134, 286 to global filters 184, 184 comments 164, 229 in codes 83, 83, 118 in merged projects 243, 243–4, 244 and protocols for documents 37, 38 and quotation comments 79, 79, 84, 96, 96 and use of 57–8, 58 computers and qualitative data analysis 1, 3 consensus and consultation among coders 271–2 context search of keywords 65, 65–6, 66, 67 co-occurrence operator in quotations 159–61, 160 and non-mutually exclusive codings 281–2, 282, 293, 292–4 Corbin, Juliet 59 core tabs 11, 12 creating codes 81, 81, 112–13 creating groups 39–42, 40, 42 creating memos 56, 56–7 creating queries 173, 173–5, 174, 175 creating smart codes 164, 164–5, 165, 175, 175 data content coding 103–4, 108, 108–16, 109, 110–11, 112, 117–18 and the categorical mode of data analysis 104–8, 106, 107, 118–20, 119 and hyperlinking repeated quotations 116–17, 117 data file formats 27–8, 27–32, 30, 90 data querying 154, 188 and creating queries 173, 173–5, 174, 175 and creating reports 174–5, 175 and operators for 155–62, 156–62, 176, 176 and quotations across overlapping categories 176–8, 177 and smart codes 163–6, 164, 166 using proximity operators 176, 176 data relationships and patterns 147 data retrieval 21 data segment coding 70, 71, 90, 124, 137 and quoting in reports 233–4 and reference to quotations and codes 72–3, 73, 139 with two or more codes 75, 75, 77 deactivating a global filter 184, 184 default location 46–8, 47, 48 deleting codes 82 deleting documents 43, 43 deleting groups 42 deleting projects 44, 244, 244 descriptive-level analysis and first-stage coding 2–3, 80, 112 and splinters 112–13, 118, 129 diagram mode of data analysis, the 6 docked and floated windows 17–19, 18, 19 document groups 39–44, 40, 41, 42, 43, 49, 53, 54 adding to 37, 39 and code queries 178–81, 179, 180 and creating groups 39–42, 40, 42 in team projects 241, 241 Document Manager, the 16, 17, 17, 91, 179, 229, 230 and importing data 50, 54 and renumbering documents 33, 43–4 documents as entity browsers 15, 15 duplicate codes in merged projects 246 editing smart codes 165–6, 166 entity browsers 14–16, 15 entity managers 16, 16–17, 17 entity types and the user interface 9–10, 11–14

INDEX 305 Excel files 28, 30, 49 and exporting code lists 86, 86 exhaustiveness and semantic domains 269–70 exporting of code lists 86, 86, 235, 235–6, 236 of memos for reports 232, 232 of project bundle files 44, 44–5 file size 31 filters 15, 40, 40, 129, 146, 164 and code lists 75–6, 76, 131, 132, 133, 146 see also global filters first and second-cycle coding 114, 120 first-stage coding and descriptive-level analysis 2–3, 80, 112 and splinters 112–13, 118, 129 focus group transcripts 32–3, 86–8, 87 coding of data 88, 88–90, 89 geo data 98–101, 99, 101 and geo quotations 100, 100–1 global filters 63, 63, 64, 65, 164, 182–4, 183 in the Code-Co-occurrence table 184–5, 185 in the Code-Document table 186, 186 and deactivating of 184, 184 in networks 197, 197, 201 groups as entity browsers 15, 15–16 Hartmann, Eddie 205 hierarchical layouts 212, 213, 217, 218 holistic manner of the analytic process 5–6 Holsti index, the 278 Home tab, the 12, 37, 39 hyperlinking and the network function 215, 220–6, 222–6 of repeated quotations 116–17, 117 idea memos 55 identifier patterns 86–8, 87, 89, 89 image files 27, 30, 31 Import & Export tab, the 12, 12–13 importing data 12, 12, 48–9, 50–1 and code lists 85, 85 and the Document Manager 50, 54 and merging of imported sub projects 242–5, 243, 244, 246, 254 and networks 200–2, 201 of project bundle files 247–8, 248, 251, 261 and reference manager data 52–4, 53, 54 Import Project Bundle dialog 10 inter-coder agreement 127, 239, 263, 264–8, 265, 271–2, 294 and measurement of through co-efficients 272–4, 272–8, 287, 287 and the merging of projects 260, 261, 262, 262, 282–3, 283 and multi-valued coding 270–1, 271 and non-mutually exclusive codings 281–2, 282, 293, 292–4 and percent agreement analysis 276–7, 277 and project preparation for analysis 280–5, 280–5 and results interpretation 287–90, 288–90 and sample size for codings per code 278–9, 279 and semantic domains 268–70, 270, 289, 289, 291, 291–2, 292 inter-coder disagreements 267, 268 interpretive approach to analysis, the 139 intra-coder inconsistencies 267, 268 key frames 29 Krippendorff alpha co-efficients 275, 275–6, 276, 287, 287 labels (tags) in the coding process 2, 70, 84, 103, 113, 140 language interfaces 31 layout options in networks 21, 212, 212–14 libraries 35, 46 links between codes and documents 196, 203, 221 between codes in open networks 206–7, 209, 209, 218, 219 and visualization options in networks 195, 200, 200 loading documents 19, 19–20 local filters 129, 184 see also global filters lumpers and data collection 124 Master project, the 239, 249–50, 253, 256, 257–9, 260–3 importing Master project bundle files 246–8, 247, 248, 251, 254 and merging of Import projects 242–5, 243, 244, 246, 254 memo function, the 4 memos as entity browsers 15, 15 Mendeley (software) 52, 54 merging of codes 84, 84, 115, 131, 132, 132–3, 134, 146 of sub projects 282–3, 283 see also Master project, the methodological approaches to computer-assisted analysis 6–7 Minecraft (computer game) 49 Modes of Thinking for Qualitative Data Analysis (book) 6 multi-valued coding 270–1, 271 mutual exclusiveness and semantic domains 270, 270

306 INDEX naming data files 33, 33–4 navigation panel, the 14, 14–15, 15 N-C-T (Noticing-Collecting-Thinking) method, the 7, 108, 109, 114 network function, the 4, 21, 21, 192–4, 206 case-based analysis and entity sensitive imports 200–2, 201 and case-based networks 219, 219–20 and code-code relations 207, 207, 208 and code co-occurrences 196–200, 197–200 and developing storylines 202, 203 as entity browser 15, 15 and exporting for reports 234, 234 and first and second-class relations 194–6, 195, 215 and hyperlinks 215, 220–6, 222–6 layout and routing options 212, 212–14 and linking codes in open networks 206–7, 209, 209, 218, 219 linking multiple nodes 209–10, 210 and nodes 210, 210–11, 211 and presentation and publication of research findings 203–5, 204–6 and the Relations Editor 215, 215–16 and transitive and asymmetric relations between code links 207–9, 208 and use for structural purposes 216–19, 217, 218 and view options 214, 214 network links for code lists 161, 162 normalizing data and code frequencies 171–2, 172 operators for query building 155–62, 156–62 organic layouts 212, 213 organic routing 212, 214 organizing project documents 38–9 orthogonal routing 212, 214 orthogonal tree layouts 212, 212 overlapping quotations 159, 159 partitioning the screen 20 PDF files 27, 28–9, 31, 74, 90 perpendicular layouts 212, 212 phenomenological research 120 polyline edge routing 212, 214 preparing survey data 49, 49–51, 50 Prezi (software) 204 printing options 236, 236 project bundle files exporting 44, 44–5 importing 247–8, 248, 251, 261 Project Explorer, the 14, 14 project management 34–8, 36, 38 see also Master project, the project manager, the 239, 241–2, 246–7 project memos 55 proximity operators 157–8, 158, 176, 176 QDC format, the 86 Query Tool, the 22, 22–3, 51, 173, 174, 179 see also data querying Quotation Manager, the 79, 96, 122, 123, 139, 165, 183, 183–4, 223 quotations 73–4, 74, 161, 188, 215 in audio/video files 93, 93–4, 94 and coding frame development 255, 256 coding of video quotations 96, 97 and co-occurrence 159–61, 160, 168 and creating queries 174, 174 and data segments 70, 71, 72–3, 73, 90, 124, 137, 139, 157, 233–4 and editing of 140, 140 embedded quotations 158, 158–9, 159 as entity browsers 15, 15 and export to Excel 123, 123 and filtering 182, 183, 183 and geo quotations 100, 100–1 and hyperlinks 215, 220, 221–5, 222–5 and image quotations 98, 98 and length and resizing of 77, 78, 116 and links from memos 150, 150, 196 and overlapping 159, 159, 168, 176–8, 177 and projects with predefined quotations 280–1, 281 and quotation comments 79, 79, 84, 96, 96 renaming of video quotations 94, 94, 95, 95, 101 repeated occurrences of 116–17, 117 and retrieval from code 122, 122–3, 123, 129, 160, 174 and semantic domains 289, 289–90, 290 in smart code 163, 163–4 splitting into subcategories 124–6, 125, 126, 127, 134, 137, 270 radial layouts 212, 213 random layouts 212, 213 Redundant Codings Analyzer, the 167 reference manager data 52–4, 53, 54 reliability of data 266–8, 279–80, 288 removing entities from a group 41, 41 renaming groups 43 renumbering documents 43–4 replacing code 77 report preparation and writing 228–37, 233, 235, 236 and exporting memos for 232, 232 and exporting networks for 234, 234 and research-question memos 231, 233 research diaries 55, 56, 57 researchers and inter-coder agreement 264–6, 265

INDEX 307 research-question memos 147–51, 148, 149, 231, 233 resizing quotations 77, 78, 116 ribbons 11 routing options 212, 212–14 Schwarzenegger, Arnold 204 Scope Tool, the 22, 179, 180, 180–1 Search Project tab, the 13, 13 second stage coding and research questions 3–4 semantic domains 268–70, 270, 276, 279, 281 and agreement on codings 291, 291–2, 292 and identification of 290–1, 291 and inter-coder agreement 268–70, 270, 285–6, 285–7, 289, 289, 291, 291–2, 292 semantic operators 161–2, 162 smart codes 155, 164–5, 165, 190–1 and editing of 165–6, 166 and quotations 163, 163–4 smart groups 155, 181, 181–2 snapshots of projects 45, 45, 131, 256 software 7, 52, 53, 54, 204 splinters and descriptive coding 112–13, 118, 129 Split Code tool 124–6, 125, 127, 129, 129 stop-word lists 60–1, 61 storylines for research reports 202, 203 straight routing 212, 214 students and team projects 257–9, 260–2, 262, 263–4 sub codes 124-5, 125, 126-7, 126, 134 sub trees 212, 213, 217 switch library process, the 46–8, 47, 48 symmetry in directed and undirected first-class network relations 195, 195, 208, 208 tab groups 20, 20, 223–4, 224 tagging see labels (tags) in the coding process teachers and team projects 257–9, 260–1, 262, 263, 263 team memos 55 team projects 238–40 and analysis of common documents 250–2, 251 for classroom projects 257–9, 257–64, 260, 262, 263 and coding frames 253–7, 254, 255 and the deletion of entities 244, 244 document analysis with common code lists 252, 252–3 and entities and comments 243, 243–4, 244 and importing project bundle files 247–8, 248, 251 and inter-coder agreement 264–5, 265, 288, 288–9 and merging projects 242–3, 243, 244–6, 245, 251 overview of tasks 249–50 and the project manager 239, 240–1, 249, 251, 256, 257 and user accounts 239–40, 240 Technical Report CTIT-02-34 54 text documents 27, 28, 31, 77 textual continuum 272 theory (literature) memos 56 timestamps 30, 55 to-do memos 55 Tools & Support tab, the 13, 13 transcription of data 31–3, 32, 34, 74, 161 transitive and asymmetric relations in code links 208, 208–9 transitive links 161, 161 tree-layouts 212, 213 unique IDs 34, 261 unlinking and removing codes 78–9 user accounts on team projects 239–40, 240 video files see audio/video files video snapshots 98, 98 word clouds 2, 59–64, 60–3, 65 writing memos 54–9, 56, 58, 79, 109, 147, 192 research-question memos 147–51, 148, 149 zooming the timeline 92, 93 Zotero 52, 53