5 Collecting Data in Your Classroom

As you develop an action plan for your action research project, you will be thinking about the primary task of conducting research, and probably contemplating the data you will collect. It is likely you have asked yourself questions related to the methods you will be using, how you will organize the data collection, and how each piece of data is related within the larger project. This chapter will help you think through these questions.

Data Collection

The data collection methods used in educational research have originated from a variety of disciplines (anthropology, history, psychology, sociology), which has resulted in a variety of research frameworks to draw upon. As discussed in the previous chapter, the challenge for educator-researchers is to develop a research plan and related activities that are focused and manageable to study. While human beings like structure and definitions, especially when we encounter new experiences, educators-as-researchers frequently disregard the accepted frameworks related to research and rely on their own subjective knowledge from their own pedagogical experiences when taking on the role of educator-researcher in educational settings. Relying on subjective knowledge enables teachers to engage more effectively as researchers in their educational context. Educator-researchers especially rely on this subjective knowledge in educational contexts to modify their data collection methodologies. Subjective knowledge negotiates the traditional research frameworks with the data collection possibilities of their practice, while also considering their unique educational context. This empowers educators as researchers, utilizing action research, to be powerful agents for change in educational contexts.

Thinking about Types of Data

Whether the research design is qualitative, quantitative or mixed-methods, it will determine the methods or ways you use to collect data. Qualitative research designs focus on collecting data that is relational, interpretive, subjective, and inductive; whereas a typical quantitative study, collects data that are deductive, statistical, and objective.

In contrast, qualitative data is often in the form of language, while quantitative data typically involves numbers. Quantitative researchers require large numbers of participants for validity, while qualitative researchers use a smaller number of participants, and can even use one (Hatch, 2002). In the past, quantitative and qualitative educational researchers rarely interacted, sometimes holding contempt for each other’s work; and even published articles in separate journals based on having distinct theoretical orientations in terms of data collection. Overall, there is a greater appreciation for both quantitative and qualitative approaches, with scholars finding distinct value in each approach, yet in many circles the debate continues over which approach is more beneficial for educational research and in educational contexts.

The goal of qualitative data collection is to build a complex and nuanced description of social or human problems from multiple perspectives. The flexibility and ability to use a variety of data collection techniques encompasses a distinct stance on research. Qualitative researchers are able to capture conversations and everyday language, as well as situational attitudes and beliefs. Qualitative data collection is able to be fitted to the study, with the goal of collecting the most authentic data, not necessarily the most objective. To researchers who strictly use quantitative methods, qualitative methods may seem wholly unstructured, eclectic, and idiosyncratic; however, for qualitative researchers these characteristics are advantageous to their purpose. Quantitative research depends upon structure and is bounded to find relationship among variables and units of measurement. Quantitative research helps make sense of large amounts of data. Both quantitative and qualitative research help us address education challenges by better identifying what is happening, with the goal of identifying why it is happening, and how we can address it.

Most educator-researchers who engage in research projects in schools and classrooms utilize qualitative methodologies for their data collection. Educator-researchers also use mixed methods that focus on qualitative methods, but also use quantitative methods, such as surveys, to provide a multidimensional approach to inquiring about their topic. While qualitative methods may feel more comfortable, there is a methodological rationale for using quantitative research.

Research methodologists use two distinct forms of logic to describe research: induction and deduction. Inductive approaches are focused on developing new or emerging theories, by explaining the accumulation of evidence that provides meaning to similar circumstances. Deductive approaches move in the opposite direction, and create meaning about a particular situation by reasoning from a general idea or theory about the particular circumstances. While qualitative approaches are inductive – observe and then generate theories, for example – qualitative researchers will typically initiate studies with some preconceived notions of potential theories to support their work.

Flexible Research Design

A researcher’s decisions about data collection and activities involve a personal choice, yet the choice of data sources must be responsive to the proposed project and topic. Logically, researchers will use whatever validated methods help them to address the issue they are researching and will develop a research plan around activities to implement those methods. While a research plan is important to conducting valid research in schools and classrooms, a research plan should also be flexible in design to allow data to emerge and find the best data to address research questions. In this way, a research plan is recommended, but data collection methods are not always known in advance. As you, the educator-researcher, interacts with participants, you may find it necessary to continue the research with additional data sources to better address the question at the center of your research. When educators are researchers and a participant in their study, it is especially important to keep an open mind to the wide range of research methodologies. All-in-all educator-researchers should understand that there are varied and multiple paths to move from research questions to addressing those questions.

Mixed Methods

As mentioned above, mixed methods is the use of both qualitative and quantitative methods. Researchers generally use mixed methods to clarify findings from the initial method of data collection. In mixed-methods research, the educator-researcher has increased flexibility in data collection. Mixed methods studies often result in a combination of precise measurements (e.g., grades, test scores, survey, etc.) along with in-depth qualitative data that provide meaningful detail to those measurements. The key advantage of using mixed methods is that quantitative details enhance qualitative data sources that involve conclusions and use terms such as usually, some, or most which can be substituted with a number or quantity, such as percentages or averages, or the mean, the median, and/or the mode. One challenge to educator-researchers is that mixed methods require more time and resources to complete the study, and more familiarity about both qualitative and quantitative data collection methods.

Mixed methods in educator research, even if quantitative methods are only used minimally, provide an opportunity to clarify findings, fill gaps in understanding, and cross-check data. For example, if you are looking at the use of math journals to better engage students and improve their math scores, it would be helpful to understand their abilities in math and reading before analyzing the math journals. Therefore, looking at their test scores might give you some nuanced understanding of why some students improved more than others after using the math journals. Pre- and post-surveys would also provide valuable information in terms of students’ attitudes and beliefs about math and writing. In line with thinking about pre- and post-surveys, some researchers suggest using either qualitative or quantitative approaches in different phases of the research process. In the previous example, pre- and post test scores may quantitatively demonstrate growth or improvement after implementing the math journal; however, the qualitative data would provide detailed evidence as to why the math journals contributed to growth or improvement in math. Quantitative methods can establish relationships among variables, while qualitative methods can explain factors underlying those same relationships.

I caution the reader at this point to not simply think of qualitative methodologies as anecdotal details to quantitative reports. I only highlight mixed methods to introduce the strength of such studies, and to aid in moving educational research methodology away from the binary thinking of quantitative vs. qualitative. In thinking about data collection, possible data sources include questionnaires or surveys, observations (video or written notes), collaboration (meetings, peer coaching), interviews, tests and records, pictures, diaries, transcripts of video and audio recordings, personal journals, student work samples, e-mail and online communication, and any other pertinent documents and reports. As you begin to think about data collection you will consider the available materials and think about aspects discussed in the previous chapter: who, what, where, when, and how. Specifically:

If you find you are having trouble identifying data sources that support your initial question, you may need to revise your research question – and make sure what you are asking is researchable or measurable. The research question can always change throughout the study, but it should only be in relation the data being collected.

Participant Data

As an educator, your possible participants selection pool is narrower than most researchers encounter – however, it is important to be clear about their role in the data design and collection. A study can involve one participant or multiple participants, and participants often serve as the primary source of data in the research process. Most studies by educator-researchers utilize purposeful sampling, or in other words, they select participants who will be able to provide the most relevant information to the study. Therefore, the study design relies upon the participants and the information they can provide. The following is a description of some data collection methods, which include: surveys or questionnaires, individual or group interviews, observations, field notes or diaries, narratives, documents, and elicitation.

Surveys

Surveys, or questionnaires, are a research instrument frequently used to receive data about participants’ feelings, beliefs, and attitudes in regard to the research topic or activities. Surveys are often used for large sample sizes with the intent of generalizing from a sample population to a larger population. Surveys are used with any number of participants and can be administered at different times during the study, such as pre-activity and post-activity, with the same participants to determine if changes have occurred over the course of the activity time, or simply change over time. Researchers like surveys and questionnaires as an instrument because they can be distributed and collected easily – especially with all of the recent online application possibilities (e.g., Google, Facebook, etc.). Surveys come in several forms, closed-ended, open-ended, or a mix of the two. Closed-ended surveys are typically multiple-choice questions or scales (e.g. 1-5, most likely–least likely) that allow participants to rate or select a response for each question. These responses can easily be tabulated into meaningful number representations, like percentages. For example, Likert scales are often used with a five-point range, with options such as strongly agree, agree, neutral, disagree, and strongly disagree. Open-ended surveys consist of prompts for participants to add their own perspectives in short answer or limited word responses. Open-ended surveys are not always as easy to tabulate, but can provide more detail and description.

Interviews and Focus Groups

Interviews are frequently used by researchers because they often produce some of the most worthwhile data. Interviews allow researchers to obtain candid verbal perspectives through structured or semi-structured questioning. Interview questions, either structured or semi-structured, are related to the research question or research activities to gauge the participants’ thoughts, feelings, motivations, and reflections. Some research relies on interviewing as the primary data source, but most often interviews are used to strengthen and support other data sources. Interviews can be time consuming, but interviews are worthwhile in that you can gather richer and more revealing information than other methods that could be utilized (Koshy, 2010). Lincoln and Guba (1985) identified five outcomes of interviewing:

As mentioned above, interviews typically take two forms: structured and semi-structured. In terms of interviews, structured means that the researcher identifies a certain number of questions, in a prescribed sequence, and the researcher asks each participant these questions in the same order. Structured interviews qualitatively resemble surveys and questionnaires because they are consistent, easy to administer, provide direct responses, and make tabulation and analysis more consistent. Structured interviews use an interview protocol to organize questions, and maintain consistency.

Semi-structured interviews have a prescribed set of questions and protocol, just like structured interviews, but the researcher does not have to follow those questions or order explicitly. The researcher should ask the same questions to each participant for comparison reasons, but semi-structured interviews allow the researcher to ask follow-up questions that stray from the protocol. The semi-structured interview is intended to allow for new, emerging topics to be obtained from participants. Semi-structured questions can be included in more structured protocols, which allows for the participant to add additional information beyond the formal questions and for the researcher to return to preplanned formal questions after the participant responds. Participants can be interviewed individually or collectively, and while individual interviews are time-consuming, they can provide more in-depth information.

When considering more than two participants for an interview, researchers will often use a focus group interview format. Focus group interviews typically involve three to ten participants and seek to gain socially dependent perspectives or organizational viewpoints. When using focus group interviews with students, researchers often find them beneficial because they allow student reflection and ideas to build off of each other. This is important because often times students feel shy or hesitant to share their ideas with adults, but once another student sparks or confirms their idea, belief, or opinion they are more willing to share. Focus group interviews are very effective as pre- and post-activity data sources. Researchers can use either a structured or semi-structured interview protocol for focus group interviews; however, with multiple participants it may be difficult to maintain the integrity of a structured protocol.

Observations

One of the simplest, and most natural, forms of data collection is to engage in formal observation. Observing humans in a setting provides us contextual understanding of the complexity of human behavior and interrelationships among groups in that setting. If a researcher wants to examine the ways teachers approach a particular area of pedagogical practice, then observation would be a viable data collection tool. Formal observations are truly unique and allow the researcher to collect data that cannot be obtained through other data sources. Ethnography is a qualitative research design that provides a descriptive account based on researchers’ observations and explorations to examine the social dynamics present in cultures and social systems – which includes classrooms and schools. Taken from anthropology, the ethnographer uses observations and detailed note taking, along with other forms of mapping or making sense of the context and relationships within. For Creswell (2007), several guidelines provide structure to an observation:

As an educator-researcher, you may take on a role that exceeds that of an observer and participate as a member of the research setting. In this case, the data sources would be called participant observation to clearly identify the degree of involvement you have in the study. In participant observation, the researcher embeds themselves in the actions of the participants. It is important to understand that participant observation will provide completely different data, in comparison to simply observing someone else. Ethnographies, or studies focused completely on observation as a data source, often extend longer than other data sources, ranging from several months to even years. Extended time provides the researcher the ability to obtain more detailed and accurate information, because it takes time to observe patterns and other details that are significant to the study. Self-study is another consideration for educators, if they want to use observation and be a participant observer. They can use video and audio recordings of their activities to use as data sources and use those as the source of observation.

Field Diaries and Notes

Utilizing a field dairy, or keeping field notes, can be a very effective and practical data collection method. In purpose, a field diary or notes keep a record of what happens during the research activities. It can be useful in tracking how and why your ideas and the research process evolved. Many educators keep daily notes about their classes, and in many ways, this is a more focused and narrower version of documenting the daily happenings of a class. A field diary or notes can also serve as an account of your reflections and commentary on your study, and can be a starting place for your data analysis and interpretations. A field diary or notes are typically valuable when researchers begin to write about their project because it allows them to draw upon their authentic voice. The reflective process that represents a diary can also serve as an additional layer of professional learning for researchers. The format and length of a field diary or notes will vary depending on the researching and the topic; however, the ultimate goal should be to facilitate data collection and analysis.

Narratives

Data narratives and stories are a fairly new form of formalized data. While researchers have collected bits and pieces of narratives in other forms of data, asking participants to compose a narrative (either written, spoken, or performed) as a whole allows researchers to examine how participants embrace the complexities of the context and social interactions. Humans are programmed to engage with and share narratives to develop meaningful and experiential knowledge. Educator autobiographies bring to life personal stories shaped by knowledge, values, and feelings that developed from their classroom experiences. Narrative data includes three primary areas: temporality, sociality, and place (Clandinin & Conolley, 2000). In terms of temporality, narratives have a past, present, and future because stories are time-based and transitional. Sociality highlights the social relationships in narratives as well as the personal and moral dispositions. Place includes the spaces where the narratives happen. Furthermore, bell hooks (1991) notes that narratives, or storytelling, as inquiry can be a powerful way to study how contexts are influenced by power structures, often linking and intersecting the structural dynamics of social class, race, and gender to highlight the struggle.

Documents

Documents provide a way to collect data that is unobtrusive to the participant. Documents are unobtrusive data because it is collected without modifying or distracting the research context when gathered. Educational settings maintain records on all sorts of activities in schools: content standards, state mandates, student discipline records, student attendance, student assessments, performance records, parental engagement, records of how teachers spend PTO money, etc. Documents often provide background and contextual material providing a snapshot of school policies, demographic information, ongoing records over a period of time, and contextual details from the site of the research study. Documents can be characterized similarly to historical research, as primary and secondary. Examples of primary materials are first-hand sources from someone in the educational context, such as minutes from a school board or faculty meeting, photographs, video recordings, and letters. Examples of secondary sources typically include analysis or interpretations of a primary source by others, such as texts, critiques, and reviews. Both types of sources are especially valuable in action research.

Elicitation Methods

We have talked about several methods of data collection that each have useful ways of documenting, inquiring, and thinking about the research question. However, how does a researcher engage participants in ways that allow them to demonstrate what they know, feel, think, or believe? Asking participants directly about their thinking, feeling, or beliefs will only take you so far depending on the comfort and rapport the participant has with the researcher. There are always a variety of hurdles in extracting participants’ knowledge. Even the manner in which questions are framed and the way researchers use materials in the research process are equally important in getting participants to provide reliable, comparable, and valid responses. Furthermore, all individuals who participate in research studies vary in their ability to recall and report what they know, and this affects the value of traditional data collection, especially structured and semi-structured interviewing. In particular, participants’ knowledge or other thinking of interest may be implicit and difficult for them to explicate in simple discussion.

Elicitation methods help researchers uncover unarticulated participant knowledge through a potential variety of activities. Researchers will employ elicitation methods and document the participants’ actions and typically the description of why they took those particular actions. Educators may be able to relate the process of elicitation methods to a “think aloud” activity in which the researcher wants to record or document the activity. Elicitation methods can take many forms. What follows are some basic ideas and formats for elicitation methods.

Brainstorming/Concept Map

Most educators are probably familiar with the process of brainstorming or creating a concept map. These can be very effective elicitation methods when the researcher asks the participant to create a concept map or representation of brainstorming, and then asks the participant to explain the connections between concepts or ideas on the brainstorming or concept map.

Sorting

Sorting provides an engaging way to gather data from your participants. Sorting, as you can imagine, involves participants sorting, grouping, or categorizing objects or photographs in meaningful ways. Once participants have sorted the objects or photographs, the researcher records or documents the participant explaining why they sorted or grouped the objects or photographs in the way that they did. As a former history teacher, I would often use sorting to assess my students’ understanding of related concepts and events in a world history class. I would use pictures too as the means for students to sort and demonstrate what they understood from the unit. For broader discussion of elicitation techniques in history education see Barton (2015).

Listing/ Ranking

Listing can be an effective way to examine participants’ thinking about a topic. Researchers can have participants construct a list in many different ways to fit the focus of the study and then have the participants explain their list. For example, if an educator was studying middle school student perceptions of careers, they could ask them to complete three lists: Careers in Most Demand; Careers with Most Education/Training; Careers of most Interest.

Then, once participants have filled out the lists, the most important part is documenting them explaining their thinking, and why they filled out the lists the way they did. As you may imagine, in this example, every participant would have a list that is different based on their personal interests.

Recall

Researchers can also elicit responses by simply giving participants a prompt, and then asking them to recall whatever they know about that prompt. Researchers will have the participants do this in some sort of demonstrative activity. For example, at the end of a world history course, I might ask students to explain what “culture” means to them and to explain their thinking.

Re-articulation (writing or drawing)

A unique way to engage participants in elicitation methods is to have them write about, rewrite, or draw visual representations of either life experiences or literature that they have read. For example, you could ask them to rewrite a part of the literature they did not like, add a part they thought should be there, or simply extend the ending. Participants can either write or draw these re-articulations. I find that drawing works just as well because, again, the goal is to have participant describe their thinking based on the activity. 

Scenario Decision-Making

Elicitation methods can also examine skills. Researchers can provide participants scenarios and ask them to make decisions. The researchers can document those decisions and analyze the extent to which the participant understands the skill.

 Document, Photograph, or Video Analysis

This is the most basic elicitation in which the researcher provides a document, photograph, or video for the participant to examine. Then, the researcher asks questions about the participants interpretations of the document, photograph, or video. One method that would support this sort of elicitation is to ask the participants to provide images from their everyday words. For example, asking students to document the literacy examples in their homes (i.e., pictures of calendars, bookshelves etc.).  With the availability of one-to-one tech, and iPads, participant documentation is easier.

There are many more methods of data collection also, as well as many variations of the methods described above. The goal for you is to find the data collection methods that are going to give you the best data to answer your research question. If you are unsure, there is nothing wrong with collecting more data than you need to make sure you use effective methods – the only thing you have to lose is time!

Use of Case Studies

Case studies are a popular way for studying phenomena in settings using qualitative methodology. Case studies typically encompass qualitative studies which look closely at what happens when researchers collect data, analyze the data, and present the results. Case studies can focus on a single case or examine a phenomenon across multiple cases. Case studies frame research in a way that allows for rich description of data and depth of analysis.

An advantage of using case study design is that the reader often identifies with the case or phenomena, as well as the participants in the study. Yin (2003) describes case study methodology as inquiry that investigates a contemporary phenomenon within its authentic context. Case studies are particularly appropriate when the boundaries and relationship between the phenomenon and the context are not clear. Case studies relate well with the processes involved in action research. Critics of action research case studies sometimes criticize the inevitable subjectivity, just like general criticisms of action research. Case studies provide researchers opportunities to explore both the how and the why of phenomena in context, while being both exploratory and descriptive.

We want to clarify the differences between methodologies and methods of research. There are methodologies of research, like case study and action research, and methods of data collection. Methodologies like ethnography, narrative inquiry, and case study draw from some similar methods of data collecting that include interviews, collection of artifacts (writings, drawings, images), and observations. The differences between the methodologies include the time-frame for research; the boundaries of the research; and the epistemology.

Triangulation of Data

Triangulation is a method used by qualitative researchers to check and establish trustworthiness in their studies by using and analyzing multiple (three or more) data collection methods to address a research question and develop a consistency of evidence from data sources or approaches. Thus, triangulation facilitates trustworthiness of data through cross verification of evidence, to support claims, from more than two data collection sources. Triangulation also tests the consistency of findings obtained through different data sources and instruments, while minimizing bias in the researcher’s interpretations of the data.

If we think about the example of studying the use of math journals in an elementary classroom, the researcher would want to collect at least three sources of data – the journal prompts, assessment scores, and interviews. When the researcher is analyzing the data, they will want to find themes or evidence across all three data sources to address their research question. In a very basic analysis, if the students demonstrated a deeper level of reflection about math in the journals, their assessment scores improved, and their interviews demonstrated they had more confidence in their number sense and math abilities – then, the researcher could conclude, on a very general level, that math journals improved their students’ math skills, confidence, or abilities. Ideally, the study would examine specific aspects of math to enable deeper analysis of math journals, but this example demonstrates the basic idea of triangulation. In this example, all of the data provided evidence that the intervention of a math journal improved students’ understanding of math, and the three data sources provided trustworthiness for this claim.